QG Lakehouse is where the chapters stop being separate ideas and become a single run. The executable story is:
cargo run -- qglake-storyThe default output is a readable briefing. The full machine report is:
cargo run -- qglake-story --jsonThe story asks a mission-shaped question:
Where do fiscal capacity, energy burden, mobility disruption, and climate-health risk overlap, and what can a supervisor responsibly know without violating restricted-data boundaries?
The answer is not a single omniscient model response. It is a governed multi-agent run over Sail, Grust, Semantic Croissant, CDIF, OSI, DIDs, ODRL, TypeSec, OpenLineage, and optional Ollama inference.
Read this chapter as a worked example, not merely a demo. Each step removes work from the model and moves it into a more reliable layer. Loading removes file ambiguity. Croissant removes schema ambiguity. CDIF removes publication ambiguity. OSI removes domain ambiguity. Grust removes route ambiguity. DID, ODRL, and TypeSec remove authority ambiguity. OpenLineage removes historical ambiguity. The final model call, if used, is smaller because the system has already done the disciplined work.
The first step is ingestion. lakehouse.rs downloads the
default Dataverse and CODATA corpus, normalizes parseable assets, infers
strong column types, writes typed tables, and records a manifest. This
is where Rust earns its keep: every file has to become either a typed
table, a cataloged non-tabular asset, or an explicitly reported
unavailable/restricted asset.
From a textbook perspective, ingestion is not plumbing. It is the first act of responsibility. If the system cannot say what it loaded, what it skipped, what it typed, and how many rows it verified, later AI claims have no foundation. Local ingestion also means the organization can inspect the data path without trusting a remote indexing service.
The lakehouse does not hide partial success. A LiDAR asset can be cataloged even when it is not a table. A restricted survey can contribute metadata while raw rows remain unavailable. The navigator can reason over both facts.
cargo run -- lakehouse-load --root .querygraph/lakehouse --schema qg_lakehouse
cargo run -- lakehouse-verify --report .querygraph/lakehouse/manifest/load-report.jsonFor each loaded dataset, Querygraph writes a
semantic/croissant.json sidecar. This sidecar names files,
record sets, and fields so agents can inspect the data before requesting
access. It is the catalog entry an agent can actually understand.
This step turns raw storage into a reusable context cache. The model does not need to rediscover schema. The policy layer can target fields. The graph can connect metadata to concepts. A future run can compare its expected fields against the sidecar before touching the data.
In the story, FinanceAgent does not receive a vague instruction to “look at finance data.” It receives a semantic projection of the government-finance tables it is allowed to summarize. The projection tells the agent what tables and fields exist, and the policy layer tells it what action is allowed.
Next, Querygraph writes semantic/cdif.json. The CDIF
sidecar takes the same dataset and expresses it through interoperable
profiles: discovery, manifest, data access, access rights, controlled
vocabulary, integration, universals, and provenance.
This is how QG Lakehouse avoids becoming a private demo format. The local Sail schema can be inspected by Spark, the sidecars can be shared with FAIR data tools, and the graph can connect local variables to broader community semantics.
This step matters because local-first should not mean isolated. Querygraph can keep computation local while making metadata interoperable. That is the combination serious scientific and enterprise systems need.
The OSI layer turns dataset metadata into business concepts. In the example, the user asks about fiscal capacity, energy burden, mobility disruption, and climate-health risk. Those are not merely table names. They are concepts that must be connected to metrics, dimensions, variables, fields, and policies.
osi.rs can synthesize a model from available metadata
for the demo. In a real deployment, this is where domain experts make
the navigator precise: they define the terms the organization actually
uses and connect them to the data that can support those terms.
This is the step that changes retrieval from lexical matching to domain navigation. The navigator can ask which fields express fiscal capacity or energy burden because those concepts have been modeled before inference.
Grust turns semantic metadata into navigable relationships. The graph can say that a dataset has files, a file has record sets, a field maps to a concept, a concept appears in a policy, an agent has a role, and a run produced an answer.
The graph does not replace Sail. It makes Sail usable by agents. Sail answers table questions. Grust answers route questions.
This step is the difference between retrieval and navigation. Retrieval says “here are possible matches.” Navigation says “this is the route from question to concept, dataset, field, policy, run, and answer.” Routes can be audited.
Every actor in the story has an identity: SupervisorAgent, FinanceAgent, EnergyAgent, MobilityAgent, ClimateHealthAgent, ReferenceAgent, RestrictedDataBroker, and SynthesisAgent. Those identities are represented as DIDs and carried through TypeDID envelopes.
The supervisor is powerful, but not magical. Its DID allows orchestration. It does not automatically grant raw access to every compartment. That is the central discipline of the platform.
This step prevents agent hierarchies from becoming privilege laundries. A supervisor can coordinate work, but each specialist still acts under its own identity and scoped authority.
The ODRL layer evaluates what each agent may do against a semantic target. FinanceAgent can read or summarize finance assets. EnergyAgent can derive approved energy summaries. RestrictedDataBroker can inspect restricted metadata but cannot reveal raw restricted health records.
| Agent | Compartment | Allowed action | Explicit boundary |
|---|---|---|---|
| FinanceAgent | compartment:finance |
read, summarize |
No energy or health raw data. |
| EnergyAgent | compartment:energy |
summarize, derive |
No respondent-level restricted data. |
| MobilityAgent | compartment:mobility |
summarize, derive |
No finance-table mutation. |
| ClimateHealthAgent | compartment:climate-health |
summarize, derive |
No restricted health rows. |
| ReferenceAgent | compartment:reference |
normalize |
Can normalize units, not expand access. |
| RestrictedDataBroker | compartment:restricted |
metadata-only receipt | Raw access denied. |
| SynthesisAgent | compartment:synthesis |
aggregate signed summaries | Does not inherit raw specialist permissions. |
The signed denial is as important as the signed summary. It prevents a supervisor from silently assuming evidence was considered when it was not.
This step is where focused computation becomes responsible computation. The system does less work because it excludes forbidden data early, and it becomes safer because the exclusion is explicit.
Once a policy decision is made, TypeSec turns it into typed capability evidence. The capability is scoped to an action, resource, principal, and envelope. This is the difference between “the process is authenticated” and “this agent may perform this operation on this semantic asset for this run.”
TypeDID envelopes carry the request and response. They bind the agent identity, resource, action, payload hash, and signature. The result is an agent protocol that can be logged, replayed, and audited.
This step turns a policy decision into a software object. That object can move through Python, Rust, LangChain, Ollama, and audit tables without losing its meaning.
The supervisor delegates instead of centralizing all data. FinanceAgent, EnergyAgent, MobilityAgent, ClimateHealthAgent, ReferenceAgent, and RestrictedDataBroker work inside their compartments. They produce signed summaries, normalization notes, and denial receipts.
This is the human organizational model reflected in software. A supervisor can coordinate experts without becoming every expert and without inheriting every restricted permission.
This step is also a cost-control pattern. Specialists receive small contexts and produce small signed summaries. The synthesis agent aggregates summaries rather than raw datasets.
When the run uses a local model, Querygraph calls Ollama only after TypeSec has verified the governed prompt. The model receives a bounded question with approved context. It does not receive the lakehouse. It does not receive restricted rows merely because a prompt asked nicely.
In the JSON report, this appears under the Ollama TypeDID path. The important thing is not Ollama specifically. It is the pattern: any model runtime should be downstream of identity, semantics, policy, and lineage.
This step is the local-first model story. Querygraph can use an open-source local model when generation is useful, but the model call is not the system’s source of truth. It is a bounded operation over a prepared context.
The synthesis agent receives signed summaries and hashes. It aggregates them into a resilience briefing:
Priority areas are those where weak fiscal capacity, energy burden, mobility fragility, and climate-health exposure overlap. Restricted health data contributed only a signed metadata/denial receipt, so the briefing uses approved compartment summaries rather than raw restricted rows.
This is the product experience Querygraph is aiming for: a useful answer that also tells the truth about its limits.
This step proves that aggregation does not require universal access. A system can combine evidence without flattening compartments into one privileged prompt.
The run emits a COMPLETE OpenLineage event. Inputs
include each Sail scope. The output is the briefing. The job name, run
id, producer, event time, and facets become queryable audit data in
qg_audit.
OpenLineage in Sail means operators can ask ordinary lakehouse questions about AI behavior. Which datasets were used in this briefing? Which model path was called? Which runs touched energy survey data? Which answers included a restricted-data denial?
This step keeps AI operations inside the data platform. Audit is not a PDF appendix or a vendor dashboard. It is queryable data.
Finally, Querygraph signs a compact attestation root. The full event remains in Sail. The DID ledger carries the issuer, subject, Merkle root, signature, and payload hash. This gives the platform a verifiable memory without turning the ledger into a dumping ground for operational data.
This step separates evidence from proof. Sail stores the full queryable event. The DID attestation stores the compact proof that the event existed in this form. That keeps the ledger small and the audit trail useful.
