wdpkr: parallel pecking in a distributed forest

After two months of heavy use across our team, wdpkr is proving to be useful. Codebase exploration time has been noticeably reduced and the added dependency graph for indexed chunks of code has helped our AI agents identify and build fullstack features without exceeding the all-important context window. By saving so much session context, I have been fixated on optimizing the context we actually use.

Our team consists of many humans driving AI coding agents to plan and build features for a greenfield product at a rapid pace. Sometimes it feels like we're building a plane while it's taxiing to the runway. We were fortunate to have started a strong and scalable base for our mono-repo. But that base has grown and we're getting into the smaller, complicated bits that need to work together.

Sometimes, a snapshot of the codebase, captured at a point in time only tells a small part of the story that is our product. It does not describe the evolution of UX, the MVP feature that we continued to build up with functionality, the coding patterns that lost us hours having to debug, rebuild, and re-test in CI. The humans have always held that context and it is on us to pass that along to the coding tools to ensure they build in the ways we know best.

This is the context I want to expose in wdpkr. The decisions, the plans, the designs that shaped the very snapshot of code that my Claude is contributing to today. This was the motivation for the latest addition to wdpkr and our current development process in use today.

We run a version of "spec-driven" development which runs through a flow that gives us the most confidence in the final code change. This flow guides agent and human through a series of searching, scoping, planning, and refining steps to produce specification documents. We do this early on, with a high tier model to make the hard decisions first, consider tradeoffs, and evaluate alternative approaches to a solution before a single line of code is produced.

Specification documents hold all of the context for what feature we are adding and why it is being added in a specific way. It holds the same level of granularity that a Design Doc, an RFC, a PRD, a Spike, a you-name-it document would give. Decisions about a codebase written in English to source feedback and drive collaboration within our team.

With a finalized document, we kick off parallel sub-agents running on a smaller/cheaper model to implement the spec. I want these documents to live in wdpkr to add an extra dimension to future search calls. Through this, wdpkr can tell coding agents where the code lives, its dependencies throughout the codebase, and why it was implemented the way it is.

We added a Notion tap to wdpkr recently to achieve this. It follows the same steps that we use for code indexing. We have found that our flow for indexing code works equally well for prose by treating documents/sections in Notion like files/functions in code. Given a document wdpkr will:

1. Chunk the contents of that document into each of its sections
2. Embed the document into vectors using the same embedding model configured
3. Store those embedded vectors into the datastore. Notion documents are identified by a path prefix notion://<id> which an Agent can follow to fetch the full document if its search distance is close enough

But specification documents are not the same as code. We know the codebase is the source of truth as it is constantly updated to meet the needs of our users. The document is intended to be short lived. It will quickly become irrelevant to us and we don't want old documents cluttering search results. So we introduced the concept of decay in wdpkr.

Decay can be configured per-tap, meaning every data source indexed through that tap will have its relevance score slowly drop over time. This means our specification from 3 months ago will be much less relevant to us than a specification added two days ago. Decay logic is built on a set "half-life" duration and a "floor" rate (the lowest allowed relevance multiplier).

notion_score = score * max(floor, 0.5 ^ (age_days / half_life_days))

If a document is used and is still relevant, the caller can reinforce it through wdpkr as well. This resets the age stored in the index, keeping relevance up the more it is used.

Decay was a feature added for this specific use case but I see this mattering more for wdpkr long term. This adds a new dimension to the data we index and expose through the CLI. In a search tool like this, relevance is not static. The tool accidentally acquired the ability to not only show us where the context is, but also control when the context is relevant. The Notion tap tells the tool why the context exists, and the coding agent hopefully turns all of this into a how the feature is added. The only thing we don't know is what is next.