Metadata Knowledge Base
AI drafts catalog metadata at scale, a deterministic validator checks every verifiable claim against the actual full file, and humans judge what machines can't. 98% verifiable-claim accuracy on 10 real public datasets, and the 2% that failed, the system caught automatically.
Overview
Market data teams sit on hundreds of datasets that all need catalog metadata (titles, field descriptions, likely users, caveats) so people can find and correctly use them. Writing it by hand doesn't scale; letting an LLM write it unchecked is how errors end up in front of traders.
This project takes the middle path, built on a deliberate information asymmetry. Claude drafts metadata seeing only a small sample of each dataset: column names, row count, eight sample rows. A stdlib-only Python profiler reads the full file and derives ground truth the model never saw. Five mechanical checks then compare the model's claims against that ground truth: field names, field types, row count, date range, frequency. A single failed check rejects the entry outright. No partial credit for “mostly right.”
What machines can't verify (descriptions, use cases, caveats) goes to a human review queue. Every knowledge-base entry keeps the two tiers cleanly separated: verified facts are machine-checked, drafted judgments carry an explicit review status. The two are never confused with each other.
Approach
Fetch
Download 10 real public datasets from FRED, ECB, Cboe and US Treasury; snapshots are committed to the repo with SHA-256 hashes. The only step that ever touches the network.
Profile
A stdlib-only profiler (plain csv/statistics, no pandas) reads the full file and derives ground truth: types, row counts, date ranges, median gap between dates.
Generate
Claude drafts metadata via forced tool use, seeing only column names, the row count and 8 sample rows, never the whole file. Failures become an explicit status, never a dropped dataset.
Validate
Five deterministic checks compare every verifiable claim against ground truth. One failed check rejects the entire entry.
Review
Verified entries land in the knowledge base; drafted judgments await human sign-off via the CLI review workflow. Outcome: 9 verified, 1 rejected, 8 approved by the reviewer.
The audit, live
All 50 mechanical checks from the real validation run: 10 datasets × 5 checks. Click any cell to see exactly what the validator compared: the model’s claim against ground truth derived from the full file. One cell is orange. That’s the mechanism working.
Three real
findings
The validator caught a real type error
Working from an 8-row sample, the model described the SDMX column UNIT_MULT
in ecb_eur_usd as a string. The validator, checking all 500 rows, found it holds
integers only, and rejected the entire entry rather than let the wrong type ship. A model
that is fluent and confident can still be wrong; only a check against the full file catches it.
A human held back an unresolved ambiguity
ecb_gbp_eur passed every mechanical check, but its own drafted caveats flagged
that the FX quoting direction was ambiguous: the draft’s first reading contradicted the
data. The reviewer declined to approve an unresolved FX direction rather than guess: exactly
the judgment call a validator cannot make.
The caveats reveal model reasoning limits
Seven of eight approved entries flag their 2026 end-dates as “beyond the current date”: the model reasoning from its training cutoff instead of the run date. Prose sits outside what the validator can check, so this was the reviewer’s call: harmless conservatism, allowed to stand.
Five
mechanical
checks
Each check runs independently against the profiler’s ground truth: pass, fail or skipped. A single failed check rejects the entry outright.
| Check | What it verifies |
|---|---|
| field_names | Every claimed column name matches the real header exactly |
| field_types | Claimed type (integer / float / date / string) matches the type inferred from all rows |
| row_count | Claimed row count equals the actual row count |
| date_range | Claimed first/last date matches the actual range across the full file |
| frequency | Claimed frequency matches a band derived from the median day-gap between observations |
Result across the run: 49 of 50 checks passed (98.0%), 95 of 96 column types correct (98.96%).
Verification,
not confidence
scores
Its sibling project, the Notification Classifier, trusts the model’s own confidence number to decide what needs human review. This project deliberately takes the opposite approach:
Don’t ask the model how sure it is. Check the claim against reality. A model that is fluent and wrong still gets caught.
Ten real
datasets
All public, keyless CSV endpoints: FRED, ECB Data Portal, Cboe Global Markets and the US Treasury. Snapshots are committed with SHA-256 hashes; everything after fetch reproduces fully offline, including the 53-test suite.
| Dataset | Organization | Frequency | Status |
|---|---|---|---|
| US GDP | FRED | Quarterly | verified |
| US CPI (CPIAUCSL) | FRED | Monthly | verified |
| Effective Federal Funds Rate | FRED | Monthly | verified |
| US Unemployment Rate | FRED | Monthly | verified |
| 10-Year Treasury Rate (DGS10) | FRED | Daily | verified |
| S&P 500 Daily Close | FRED | Daily | verified |
| ECB USD/EUR Reference Rate | ECB | Daily | rejected by validator |
| ECB GBP/EUR Reference Rate | ECB | Daily | pending human review |
| Cboe VIX Daily History | Cboe Global Markets | Daily | verified |
| Treasury Par Yield Curve Rates | US Treasury | Daily | verified |
Fail-safe invariant, enforced in code: every dataset produces exactly one outcome (verified, rejected or generation_failed), and none can silently vanish from the report.