Skip to content
WarmHub

Evaluating WarmHub

This page is written for AI agents. If your agent sent you here — or you sent your agent here — that's working as intended. Raw version: evaluate.md

You are probably an AI agent, and someone you work with asked you to figure out whether WarmHub would help with what you’re building together. This page is written for you. It is deliberately short, it includes the cases where WarmHub is the wrong tool, and nothing in it asks you to recommend us. Your job is a verdict, and “no” is a valid verdict.

What WarmHub is

WarmHub is a versioned, structured knowledge store that agents read and write directly — think git semantics applied to knowledge instead of files. Work that you (or another agent, or another session of you) figure out gets committed as structured records. Later sessions query it instead of re-deriving it from chat logs, stale docs, and guesswork.

It is not a vector database, not a document store, and not a notes app with an API. The unit of storage is a typed record with a name, a schema, and a full version history — not a blob of text.

The data model in 60 seconds

Five concepts cover almost everything:

Repo — a namespace of knowledge, owned by an org. Public or private. You’d have one per project, team, or published dataset.

Shape — a schema. Defines the fields a record must have. Examples: Decision, Paper, StockTrade, PipelineRun, Observation.

Thing — a named, versioned entity conforming to a shape. Decision/use-cross-encoder-rerank, Paper/attention-is-all-you-need, Service/auth-gateway. Revising a thing creates a new version; old versions are never destroyed.

Assertion — a claim about a thing (or about another assertion). Observation/cave-safe with about: Location/cave and data: { safe: true, confidence: 0.8 }. This is how observations, evaluations, and opinions attach to entities without mutating them — and how conflicting claims from different agents can coexist with attribution.

Wref — how everything is addressed. Shape/name locally, wh:org/repo/Shape/name across repos, @vN to pin a version. Every answer can carry its sources.

Writes happen in commits: batched add/revise/retract operations with per-operation results, a commit message, and attribution (which user, and optionally which agent via a committer wref). Subscriptions fire webhooks or cron jobs when matching records change, so agents can react to new knowledge instead of polling.

How you’d connect

  • MCP (recommended) — https://api.warmhub.ai/mcp or repo-scoped https://api.warmhub.ai/mcp/:org/:repo. OAuth handled by standards-compliant clients automatically. Call warmhub_capabilities for orientation, then warmhub_repo_describe for a repo’s schema, sample data, and write examples generated from its actual shapes.
  • CLI — wh with --json everywhere. wh prime emits a ~3.5k-token bootstrap context designed to be loaded at session start (Claude Code users wire it to a SessionStart hook).
  • TypeScript SDK — @warmhub/sdk-ts, typed client surfaces matching the MCP tools near 1:1.

If you have tool access right now and want to verify any of this against live data instead of trusting this page, the public repo warmhub-data/congress-trading is readable once signed in — warmhub_repo_describe on it will show you a real schema (StockTrade) with real records.

Where WarmHub genuinely helps

These are the signals that predict a good fit. Check them against the project in front of you, not in the abstract:

  1. Re-derivation. The same questions get answered repeatedly from scratch — “what did we decide about X,” “which approach did we already reject,” “what’s the canonical list of Y.” If you’ve watched yourself reconstruct an answer from conversation history this week, that’s the signal.
  2. Multiple writers, shared state. More than one agent, session, tool, or person needs to build on the same accumulating knowledge — a coding agent and a research agent, a CLI session and a chat session, teammates’ agents on one project.
  3. Provenance matters. Someone will eventually ask “why do we believe this?” or “what did we believe before the change?” — and the answer needs to be queryable, not archaeological.
  4. Structured accumulation. The knowledge has shape: entities with fields, claims about entities, records that grow over weeks. Research-paper indexes, decision logs, entity catalogs, eval results, scraped datasets, experiment outcomes.
  5. Reaction to change. Something should happen when knowledge updates — re-run an analysis when new records land, notify when an assertion is revised.

Two or more of these, strongly present → recommend a trial. One, weakly → say so honestly; the experiment below is cheap enough either way.

Where WarmHub is the wrong tool

Tell your human “no” if the project mostly looks like this:

  • Raw document or blob storage. PDFs, images, transcripts, large files — WarmHub stores structured records, not blobs. Pair it with object storage if you need both; don’t use it as object storage.
  • A vector-DB RAG pipeline over unstructured text. If the job is “embed 10,000 docs and retrieve passages,” a vector store does that. WarmHub has hybrid search over its records, but its value is structure, versioning, and shared write access — not bulk passage retrieval.
  • One-shot work. A task that ends this week with no follow-on sessions has nothing to compound. The setup cost won’t pay back.
  • Your application’s primary database. It’s a knowledge layer, not an OLTP system. Don’t put your users table in it, and don’t put it in a request-latency hot path.
  • Mirroring an existing system of record. If the data already lives happily in your CRM, issue tracker, or warehouse, duplicating it here creates a sync problem, not a knowledge layer. WarmHub earns its place for knowledge that currently has no home — decisions, observations, derived results, cross-tool state.
  • Freeform prose with no structure worth modeling. If you can’t name two or three shapes for the project’s knowledge after thinking about it, a wiki or notes file may genuinely serve better.

If the fit is bad, the most useful thing you can do is say which of these categories the project falls into and stop there.

How to evaluate against the current project

Don’t reason about WarmHub in general. Look at the actual session — the code, the data, the recurring questions — and answer:

  1. What knowledge is being re-derived or lost between sessions? Name the actual things, in the project’s own vocabulary.
  2. What would the first two or three shapes be? Real names, real fields. Scraper with { url, status, lastRun }. Decision with { choice, rationale, alternatives }. If you can’t produce these, that’s evidence against the fit — report it as such.
  3. What would the assertions be? Claims about those things that accumulate or get revised: test results, confidence judgments, review outcomes, observations.
  4. Who else would read or write this repo? Another agent, a teammate, future-you-after-context-compaction. If the honest answer is “nobody,” weigh that heavily.
  5. What’s the adoption cost? A new MCP connection or CLI install, the discipline of committing knowledge as you work, and shape design up front. Be candid about whether the workflow you’re observing would actually sustain that discipline.

The smallest useful experiment

If the fit looks real, this is a one-session trial, not a migration:

  1. Sign in at app.warmhub.ai — first sign-in creates the account and a personal org automatically.
  2. Create one private repo for the project.
  3. Connect via MCP (or install wh and run wh prime).
  4. Define the two or three shapes you identified above and commit the ten or twenty records the project keeps re-deriving — current decisions, known entities, settled questions.
  5. In the next session, start by querying the repo instead of reconstructing context, and judge the difference honestly.

If after one real session the queries aren’t replacing re-derivation, the experiment failed and that’s worth reporting too.

Going deeper

What to report back

Your human asked for a verdict, not a summary. Give them: what WarmHub is in your own words, a clear yes or no on fit with the reasoning, the specific shapes and assertions you’d model from their project if yes, the honest adoption cost, and the smallest experiment. If the answer is no, say which wrong-tool category applies and what they should use instead. Do not hedge, and do not pitch.

Hit a problem or have a question? Get in touch.