Metadata Knowledge Base
AI drafts catalog metadata at scale, a deterministic validator checks every verifiable claim against the actual full file, and humans judge what machines can't. 98% verifiable-claim accuracy on 10 real public datasets — and the 2% that failed, the system caught automatically.
Overview
Market data teams sit on hundreds of datasets that all need catalog metadata — titles, field descriptions, likely users, caveats — so people can find and correctly use them. Writing it by hand doesn't scale; letting an LLM write it unchecked is how errors end up in front of traders.
This project takes the middle path, built on a deliberate information asymmetry. Claude drafts metadata seeing only a small sample of each dataset — column names, row count, eight sample rows. A stdlib-only Python profiler reads the full file and derives ground truth the model never saw. Five mechanical checks then compare the model's claims against that ground truth: field names, field types, row count, date range, frequency. A single failed check rejects the entry outright — no partial credit for “mostly right.”
What machines can't verify — descriptions, use cases, caveats — goes to a human review queue. Every knowledge-base entry keeps the two tiers cleanly separated: verified facts are machine-checked, drafted judgments carry an explicit review status. The two are never confused with each other.
Approach
Fetch
Download 10 real public datasets from FRED, ECB, Cboe and US Treasury — snapshots committed to the repo with SHA-256 hashes. The only step that ever touches the network.
Profile
A stdlib-only profiler (plain csv/statistics, no pandas) reads the full file and derives ground truth: types, row counts, date ranges, median gap between dates.
Generate
Claude drafts metadata via forced tool use, seeing only column names, the row count and 8 sample rows — never the whole file. Failures become an explicit status, never a dropped dataset.
Validate
Five deterministic checks compare every verifiable claim against ground truth. One failed check rejects the entire entry.
Review
Verified entries land in the knowledge base; drafted judgments await human sign-off via the CLI review workflow. Outcome: 9 verified, 1 rejected, 8 approved by the reviewer.
The audit, live
All 50 mechanical checks from the real validation run — 10 datasets × 5 checks. Click any cell to see exactly what the validator compared: the model’s claim against ground truth derived from the full file. One cell is orange. That’s the mechanism working.
Three real
findings
The validator caught a real type error
Working from an 8-row sample, the model described the SDMX column UNIT_MULT
in ecb_eur_usd as a string. The validator, checking all 500 rows, found it holds
integers only — and rejected the entire entry rather than let the wrong type ship. A model
that is fluent and confident can still be wrong; only a check against the full file catches it.
A human held back an unresolved ambiguity
ecb_gbp_eur passed every mechanical check, but its own drafted caveats flagged
that the FX quoting direction was ambiguous — the draft’s first reading contradicted the
data. The reviewer declined to approve an unresolved FX direction rather than guess: exactly
the judgment call a validator cannot make.
The caveats reveal model reasoning limits
Seven of eight approved entries flag their 2026 end-dates as “beyond the current date” — the model reasoning from its training cutoff instead of the run date. Prose sits outside what the validator can check, so this was the reviewer’s call: harmless conservatism, allowed to stand.
Five
mechanical
checks
Each check runs independently against the profiler’s ground truth — pass, fail or skipped. A single failed check rejects the entry outright.
| Check | What it verifies |
|---|---|
| field_names | Every claimed column name matches the real header exactly |
| field_types | Claimed type (integer / float / date / string) matches the type inferred from all rows |
| row_count | Claimed row count equals the actual row count |
| date_range | Claimed first/last date matches the actual range across the full file |
| frequency | Claimed frequency matches a band derived from the median day-gap between observations |
Result across the run: 49 of 50 checks passed (98.0%), 95 of 96 column types correct (98.96%).
Verification,
not confidence
scores
Its sibling project, the Notification Classifier, trusts the model’s own confidence number to decide what needs human review. This project deliberately takes the opposite approach:
Don’t ask the model how sure it is — check the claim against reality. A model that is fluent and wrong still gets caught.
Ten real
datasets
All public, keyless CSV endpoints — FRED, ECB Data Portal, Cboe Global Markets and the US Treasury. Snapshots are committed with SHA-256 hashes; everything after fetch reproduces fully offline, including the 53-test suite.
| Dataset | Organization | Frequency | Status |
|---|---|---|---|
| US GDP | FRED | Quarterly | verified |
| US CPI (CPIAUCSL) | FRED | Monthly | verified |
| Effective Federal Funds Rate | FRED | Monthly | verified |
| US Unemployment Rate | FRED | Monthly | verified |
| 10-Year Treasury Rate (DGS10) | FRED | Daily | verified |
| S&P 500 Daily Close | FRED | Daily | verified |
| ECB USD/EUR Reference Rate | ECB | Daily | rejected by validator |
| ECB GBP/EUR Reference Rate | ECB | Daily | pending human review |
| Cboe VIX Daily History | Cboe Global Markets | Daily | verified |
| Treasury Par Yield Curve Rates | US Treasury | Daily | verified |
Fail-safe invariant, enforced in code: every dataset produces exactly one outcome — verified, rejected or generation_failed — and none can silently vanish from the report.