Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions fern/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,11 @@ navigation:
contents:
- page: How Vellum Drafts 50%+ of Docs with Promptless
path: ./docs/customer-stories/vellum.mdx
- section: Life at Promptless
icon: fa-regular fa-lightbulb
contents:
- page: Running a Lightweight Research Paper Club
path: ./docs/life-at-promptless/paper-club.mdx
- tab: changelog
layout:
- section: Changelogs
Expand Down
43 changes: 43 additions & 0 deletions fern/docs/life-at-promptless/paper-club.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
---
title: Running a Lightweight Research Paper Club
description: How we run a weekly paper club at Promptless that's surprisingly useful, surprisingly easy, and has led to tangible product improvements.
---

We've been running a weekly paper club at Promptless for the last 4 months. Every Friday, we pick a research paper and read it together in-person. It's been surprisingly useful, so we wanted to share how we set it up in case it's helpful to anyone else.

We're a small team (two technical founders and two engineers), and even though we're not doing research ourselves, we're trying to extract as much intelligence and capabilities from LLMs as we build AI agents. While reading research papers isn't in the critical path for our product development, following along with research is important to us.

## How We Run It

Operationally, this kind of paper club is super easy to run, but a few details made a ton of difference:

- **Print the papers ahead of time.** Get a $100 laser printer if you don't have one in the office. Print one-sided, so that people can always read adjacent pages side-by-side.
- **Bring pens** for people to mark up and take notes.
- **Leave the office,** or at least leave the desks in the office. Two reasons: (1) makes it harder for people to get distracted with computers, and (2) changing locations can help facilitate better brainstorming. Try to pick somewhere indoors (wind + paper = bad), not too loud, and well lit.
- **Just have one person responsible for picking a paper.** At first we did round robin paper selection, but not everyone is usually equally plugged into research. Of course, anyone can suggest or nominate a paper if they want.
- **Spend the first ~45 mins in silent reading,** before diving into open-discussion. There was no expectation for anyone to have read the paper ahead of time.

## Why This Has Been Helpful

Here are a few tangible improvements to our agent that came directly or indirectly from paper club discussions. Interestingly, many good ideas came up during discussion even when they weren't directly related to the paper we were reading:

- **Fine-tuned visual grounding models** can be great at guessing pixels and bounding boxes for elements in product screenshots (very relevant for how Promptless automatically updates screenshots in product documentation)
- **Reflecting on past experiences** and extracting learnings from the user's feedback helps with verbal reinforcement learning. We let Promptless maintain its own learnings as it gathers feedback from users and evolves its own knowledge about the customer's workflows and preferences.
- **The original LoRA paper** inspired us to fine tune customer-specific models to rewrite Promptless documentation suggestions in each customer's unique voice and style. This ended up removing AI-slop completely, making Promptless's writing much more concise and human-sounding.

<Tip>
Some of our best product ideas came from tangents during paper club—not from the papers themselves. Having a dedicated forum to talk about and debate approaches to agent design has been just as valuable as the papers we read.
</Tip>

## Papers We've Read

We've covered everything from foundational ML papers to very specific papers related to our domain:

- [LoRA](https://arxiv.org/abs/2106.09685)
- [Reflexion](https://arxiv.org/pdf/2303.11366)
- [Visual grounding](https://arxiv.org/pdf/2410.05243)
- [Agent early experience](https://arxiv.org/pdf/2510.08558)
- [WritingBench](https://arxiv.org/pdf/2503.05244)
- [On-policy distillation](https://thinkingmachines.ai/blog/on-policy-distillation/)

It might seem daunting to pick out a new paper each week, but we wouldn't over-index too heavily on the specific paper you choose. One of the biggest upsides of our weekly readings has simply been having a dedicated forum to talk about and debate approaches to agent design.