From Conversation to Knowledge

There is a specific kind of institutional forgetting that happens to small teams building in the open. The decisions are all there — in Slack, in Discord, in long email threads — but they're chronologically scattered, buried under follow-up messages and tangents, separated by the natural cadence of people thinking out loud. Six months later, nobody can agree on what was confirmed versus what was just floated. Nobody remembers when the pivot happened, or why. The knowledge existed. It just never got extracted.

This is the problem a simple pipeline was built to solve. The input: months of Discord messages across multiple channels, containing design discussions for two different products. The output: a complete design specification for each, including the decisions that changed over time and the reasoning behind each change. The constraint: spend as few LLM tokens as possible getting there.

The result is a six-stage architecture that challenges one of the most common instincts in AI tooling — the instinct to send everything to the model and let it sort things out.

The raw material is Discord JSON — two export files with overlapping membership, some duplicate messages, and thousands of entries sorted loosely by timestamp. The first job is deduplication by message ID, then a clean chronological merge across both channels.

What happens next is the key structural choice of the entire pipeline. Instead of treating the message stream as a flat document, it gets cut into clusters — contiguous blocks of conversation separated by gaps of 30 minutes or more. A 30-minute silence between messages is treated as a context switch: two different conversations, two different thought threads, two different moments in the project's intellectual life.

=== CLUSTER_0015 [2026-03-25 05:03 – 05:41] (22 messages) ===
[05:03] alice: what if voice memory is per-session only initially
[05:07] alice: we can persist summaries, not transcripts
[05:41] alice: yeah let's confirm that, summaries not full transcripts

Each cluster gets saved as a numbered plaintext file. The timestamps in the cluster header are not decorative — they are preserved through every subsequent stage. Cluster 0015 confirmed the summary-only approach on March 25th. If that decision was reversed in cluster 0031 three weeks later, the pipeline will know which one came first, and weight the later one accordingly.

Alongside the cluster files, every message is written to a flat TSV — one row per message, grep-ready, no parsing required downstream.

The conventional approach to relevance search in AI pipelines is embeddings: convert every document to a dense vector, store them in a vector database, query with cosine similarity. It works. It also costs money to embed, requires infrastructure to maintain, and introduces a semantic layer that's difficult to debug when it fails.

The pipeline does something more direct. A single lightweight Haiku call takes the app names and descriptions, then returns a JSON dictionary of domain-specific search terms:

{
  "app_one": ["voice", "session", "memory", "emotion", "transcript"],
  "app_two": ["dashboard", "portfolio", "delta", "model", "exposure"]
}

That's the only LLM call in the search phase. Everything that follows is shell:

hits=$(grep -ciE "$pattern" "$cluster_file" 2>/dev/null) || hits=0

The output is a score matrix: cluster × topic → hit count. Clusters below a threshold get dropped. The survivors — about 23% of the total — are assembled into per-topic text files and handed to the extraction stage.

The reason this works better than it sounds: the search terms are generated by a model that understands the domain. They're not generic keywords — they're the vocabulary that actually appears in discussions about this specific problem. The recall gap between grep and semantic search narrows considerably when the vocabulary is domain-aware. And the cost gap — zero versus embedding API calls — only grows as the corpus grows.

Each surviving cluster gets a targeted Haiku call. The extraction prompt is unusual in one specific way: it asks the model to classify a decision state for each cluster, not just to summarize the content.

The extraction also captures a reverses field — a description of what prior decision this cluster explicitly overturns. That field is the seed for the conflict detection stage that follows.

Six workers run the extractions concurrently via Python's ThreadPoolExecutor. The wall time for 21 clusters at two seconds each is under ten seconds. Each result carries its cluster ID, its timestamp, its topic tags, and the design content — product insight, engineering decision, UX notes, deferred items. Every field flows forward into the synthesis prompt.

After extraction, the pipeline groups all results by topic tags. For most tag groups, the state sequence looks clean: proposed → debated → confirmed. No further action. No LLM call. The topic evolved normally and landed somewhere stable.

The interesting case is when a tag group contains both a confirmed state and a reversed state — or when any extraction carries a non-empty reverses field. That combination is a signal: the team changed direction on this topic. A targeted Haiku call is fired to summarize the shift:

{
  "topic": "memory",
  "original_approach": "full transcript persistence per session",
  "shift_description": "storage cost concern in week 3; moved to summary-only",
  "final_approach": "session summaries + emotional state tags",
  "when_shifted": "cluster_0021",
  "resolved": true
}

Conflict detection costs nothing for topics that didn't flip. The check is local — inspect the state sequence, look for the signal. Only when the signal fires does an LLM call happen. For a 93-cluster corpus, that turned out to be one conflict call. One.

All extracted notes, sorted by cluster ID — which is chronological order — plus the conflict summary, go to Sonnet in a single call with a 12,000-token output budget. The prompt is explicit about temporal precedence: later entries win. Earlier decisions that were reversed appear in the conflict section, not silently overwritten in the main spec.

The output is a complete Markdown design specification with ten sections. Product overview, features, architecture, data models, API integrations, engineering decisions, QA considerations, deferred work. And then section nine:

§9. Design Decisions That Changed Over Time
For each: original approach → trigger for change → final direction.

That section is the piece that no flat summarization can produce. It requires knowing the timeline. It requires knowing which cluster confirmed what, and which cluster came back three weeks later and said "actually." The timestamps carried through from stage one, the decision states captured in stage four, and the conflict summary built in stage five — they all converge here, in the one place where the intellectual history of the project becomes legible again.

Total: 23 Haiku calls and 1 Sonnet call for a 93-cluster corpus spanning months of active design work. Roughly equivalent to two or three standalone Sonnet calls — for a complete, temporally-aware design specification of two products.

The pipeline is Discord-specific in its input format. The architecture is not. Cluster by time gap, score by domain terms, extract in parallel with a fast model, synthesize once with a smart model — this works on any corpus where decisions evolve over time and the timestamp order carries meaning:

• Slack channel exports from long-running projects
• GitHub issue and PR comment histories
• Meeting transcript archives
• Long email threads with multiple participants
• Notion page edit histories with contributor timestamps

The common thread is teams thinking out loud over time. Wherever that happens in text, there is institutional memory that point-in-time snapshots cannot recover. The temporal dimension — which most summarization tools flatten away — is where the actual history lives.

The pipeline just makes it readable again.