4 Part II: QueryGraph, the Semantic Layer

The layer that makes the substrate one system. Chapters 8–16 follow the Rust reference implementation; Chapter 17 covers the Python mirror. Everything in this part is exercised by the cross-language contract (Chapter 22).

4.1 Chapter 8. Semantic Croissant: What the Data Is

MLCommons Croissant is the ML community’s standard for dataset metadata: a JSON-LD document describing a dataset’s identity, license, files (distribution), record sets, and fields, with sameAs links from fields to shared semantic types. It is how Hugging Face, Kaggle, and Google Dataset Search describe ML-ready data — which makes it QueryGraph’s front door: a Croissant document is both what QueryGraph emits for everything it loads and what it accepts from outside (LakeCat bundles, public dataset exports).

QueryGraph generates a Croissant sidecar per dataset, then treats its fields as the atoms of meaning: they become OSI column expressions (Chapter 12), ontology terms, and Grust graph nodes. From a real bundle:

{
  "@type": "cr:Dataset",
  "@id": "https://querygraph.ai/datasets/hazards/#dataset",
  "name": "Hazard vocabulary",
  "license": "https://creativecommons.org/licenses/by/4.0/",
  "distribution": [{
    "@type": "cr:FileObject",
    "contentUrl": "https://querygraph.ai/datasets/hazards.csv",
    "encodingFormat": "application/octet-stream"
  }],
  "recordSet": [{
    "@type": "cr:RecordSet",
    "field": [{
      "@type": "cr:Field",
      "name": "subject",
      "dataType": "sc:Text",
      "sameAs": "https://schema.org/about"
    }, "…"]
  }]
}

The sameAs link is the semantic hook: it says this column means something shared, not just something typed.

4.2 Chapter 9. CDIF: How the Data Is Found and Accessed

CODATA’s Cross-Domain Interoperability Framework covers the FAIR-metadata axes Croissant leaves out: discovery, access, rights, units, vocabularies, and profile declarations, expressed over the DCAT and Dublin Core vocabularies that data catalogs and national research infrastructures speak.

QueryGraph projects every dataset into a CDIF resource that declares which CDIF profiles it satisfies and how to reach the data — and carries the ODRL policy reference, so discovery metadata and rights metadata never separate:

{
  "@type": "dcat:Dataset",
  "dct:title": "Hazard vocabulary",
  "cdif:profile": [
    "https://cdif.codata.org/profile/discovery",
    "https://cdif.codata.org/profile/manifest",
    "https://cdif.codata.org/profile/data-description",
    "https://cdif.codata.org/profile/data-access", "…"
  ],
  "dcat:landingPage": "https://querygraph.ai/datasets/hazards",
  "cdif:dataElement": ["…field descriptors…"],
  "dct:accessRights": {"odrl:hasPolicy": "…/policy/default"}
}

Croissant answers what is in the data; CDIF answers how a stranger finds, evaluates, and accesses it. QueryGraph derives the CDIF projection from the Croissant document (CdifResource::from_croissant), so the two can never disagree about the dataset they describe.

4.3 Chapter 10. DID: Who Is Acting

W3C Decentralized Identifiers give agents, services, and datasets stable identity without a registry. QueryGraph uses two DID methods, each earning its place:

did:oyd for deterministic agent and dataset identity: the DID is derived from a seed by hashing, so the same input always yields the same identifier — in Rust and Python, byte for byte (a fixture in the equivalence suite pins did:oyd:zQmciWcCbpq… for a known seed). Identity becomes reproducible: re-run the pipeline, get the same actors.
did:key for verification: Ed25519 public keys published as self-certifying identifiers (did:key:z6Mk…) in envelope verification_method fields, resolvable by anyone with no infrastructure.

A generated DID document, from a real bundle:

{
  "@context": ["https://www.w3.org/ns/did/v1",
               "https://w3id.org/security/suites/ed25519-2020/v1"],
  "id": "did:oyd:zQmciWcCbpqbsYcNPVzdQ4YqznbAK9kRsnNRDdXg5Z73qCe",
  "controller": "AI Navigator",
  "public_key_multibase": "zFNru6TqKpt4ymk5pbCM5Bq4beqJ9nEDgLucfyv9nr98e",
  "service_endpoint": "https://querygraph.ai/datasets/hazards"
}

Textbook rule: DIDs say who is acting; ODRL says what is allowed; TypeSec turns the decision into a proof the code can carry.

4.4 Chapter 11. ODRL: What Is Allowed

The Open Digital Rights Language expresses machine-actionable policy: permissions and prohibitions binding an action, an assignee, and a target, each optionally constrained. QueryGraph’s profile uses odrl:use, odrl:read, odrl:derive, and two namespaced actions — querygraph:translate and querygraph:index. The evaluation rule is strict: an action is allowed only if a permission matches the assignee and action and no prohibition does.

A generated policy — note the shape of the default: the public may read with attribution, the navigator may index, and nobody may derive (train on) without a separate agreement:

{
  "@type": "odrl:Policy",
  "odrl:target": "https://querygraph.ai/datasets/hazards/#dataset",
  "odrl:assigner": "did:oyd:zQmciWcCbpq…",
  "odrl:permission": [
    {"odrl:action": "odrl:read", "odrl:assignee": "public",
     "odrl:constraint": "attribution required"},
    {"odrl:action": "querygraph:index",
     "odrl:assignee": "did:oyd:zQmciWcCbpq…",
     "odrl:constraint": "local semantic indexing for AI Navigator"}
  ],
  "odrl:prohibition": [
    {"odrl:action": "odrl:derive", "odrl:assignee": "public",
     "odrl:constraint": "no model training without separate agreement"}
  ]
}

The four layers ship together as the Navigator bundle — one JSON-LD document with @context for all four vocabularies — built by the CLI (navigator), the /v1 API, and the MCP tools, identically in both languages (the equivalence suite asserts byte equality modulo timestamps):

querygraph navigator \
  --dataset-name "Hazard vocabulary" \
  --description "Controlled vocabulary with multilingual technical terms" \
  --landing-page "https://querygraph.ai/datasets/hazards" \
  --data-url "https://querygraph.ai/datasets/hazards.csv"
# → {"@type": "querygraph:AiNavigatorSemanticBundle",
#    "layers": {"semanticCroissant": …, "cdif": …, "did": …, "odrl": …}}

4.4.1 API reference (the four projections, both languages)

Layer	Rust	Python
Croissant	`CroissantDataset { files, record_sets, … }`, `Field::new(…).semantic_type(iri)`, `.to_json_ld()`	`croissant.CroissantDataset`, `Field(name, dtype, desc).semantic_type(iri)`, `.to_json_ld()`
CDIF	`CdifResource::from_croissant(dataset, landing, data_url)`, `.with_odrl_policy(id, json)`, `.to_json_ld()`	`cdif.CdifResource` — same constructors
DID	`DidDocument::new_oyd(seed, controller)`, `.with_service_endpoint(url)`, `.to_json()`	`did.DidDocument.new_oyd(seed, controller)` — byte-identical output
ODRL	`Policy { permissions, prohibitions }`, `Rule { action, assignee, constraint }`, `Action::{Use, Read, Derive, Translate, Index}`, `.allows(assignee, action)`, `.to_json_ld()`	`odrl.Policy` / `Rule` / `Action` — same evaluation rule
Bundle	`AiNavigator::build(NavigatorInput) -> NavigatorOutput { croissant, cdif, did, odrl, bundle }`	`AiNavigator().build(NavigatorInput(...))`

4.5 Chapter 12. OSI: Business Semantics

The Open Semantic Interchange model is where business language meets governed columns. An OSI document carries one semantic model:

datasets, each with a source (a governed Sail table like sail.qg_lakehouse.government_finance__countydata), primary/unique keys, and fields;
metrics, each with an expression per SQL dialect — QueryGraph supports ANSI_SQL, SAIL_SQL, SNOWFLAKE, MDX, TABLEAU, DATABRICKS, and MAQL, resolving a requested dialect with fallback to ANSI_SQL then SAIL_SQL;
relationships between datasets (join keys);
ontology terms linking fields to shared vocabularies;
ai_context on every level: instructions, synonyms, and examples written for the LLM — the model’s way of telling an agent how to use it.

Croissant documents project into OSI mechanically: every recordSet field becomes a governed SAIL_SQL column expression, sameAs types become ontology terms, and a row_count metric is attached. The projection is implemented identically in Rust (OsiDocument::from_croissant_json, behind POST /v1/models/import/croissant) and Python (OsiDocument.from_croissant).

4.5.1 Worked example: Croissant becomes OSI, identically in both languages

let croissant = serde_json::json!({
    "name": "Energy Burden",
    "description": "Demo energy fields",
    "recordSet": [{"field": [{
        "name": "monthly_cost",
        "description": "Monthly household energy cost",
        "sameAs": "https://querygraph.ai/ontology/monthlyEnergyCost",
    }]}],
});
let osi = OsiDocument::from_croissant_json(&croissant, "qg_lakehouse")?;
assert_eq!(osi.semantic_model.name, "energy_burden_semantic_model");
assert_eq!(osi.semantic_model.datasets[0].source,
           "sail.qg_lakehouse.energy_burden");

from querygraph.osi import OsiDocument

osi = OsiDocument.from_croissant(dataset)          # same projection rules
osi.semantic_model.resolve_metric("row_count")     # 'COUNT(*)' via SAIL_SQL
osi.semantic_model.find_by_synonym("energy")       # datasets/fields/metrics

4.5.2 API reference

Surface	Essentials
`OsiDocument`	`from_mapping` (accepts upstream `semantic_model: []` lists), `from_yaml_file`, `to_json`; Rust adds `from_croissant_json(&value, schema)` and `for_dataverse(&datasets)`; Python adds `from_croissant(dataset)`
`OsiSemanticModel`	`datasets`, `metrics`, `relationships`, `ontology_terms`, `ai_context`; Python: `resolve_metric(name, dialect="SAIL_SQL")` (fallback `ANSI_SQL` → `SAIL_SQL`), `find_by_synonym(term)`
`OsiAiContext`	`instructions`, `synonyms`, `examples`; bare strings coerce to `instructions`
Dialects	`ANSI_SQL`, `SAIL_SQL`, `SNOWFLAKE`, `MDX`, `TABLEAU`, `DATABRICKS`, `MAQL`

4.6 Chapter 13. Governance: The Dual Gate

QueryGraph’s access decision is deliberately redundant: RBAC and ODRL must both allow.

The RBAC layer is role-based: grants bind principals to roles; permissions bind roles to (resource, action) pairs.
The ODRL layer is policy-based: a permission rule must match the assignee and action, and no prohibition may match.

The two layers fail differently — a fat-fingered role grant does not bypass a prohibition, and a permissive policy does not bypass a missing role — and the combined decision is issued as an AccessReceipt: principal, resource, action IRI, the boolean, the reason, and the policy id. A denial is a receipt, not an error: it flows through the same channels as an allow, gets signed into the same envelopes, and appears in the same lineage.

from querygraph.odrl_rights import OdrlRightsLayer

decision = rights.decide(principal, "sail.qg_lakehouse.finance", Action.READ)
decision.allowed        # rbac_allowed AND odrl_allowed
decision.receipt.reason # "RBAC and ODRL permitted action" / "… denied …"

A real denial receipt, captured from a navigator-loop run (Chapter 21):

{
  "principal": "did:example:qg-navigator",
  "resource": "sail.qg_lakehouse.haalsi_baseline__restricted_raw",
  "action": "odrl:read",
  "allowed": false,
  "reason": "RBAC or ODRL denied action",
  "policy_id": "urn:querygraph:policy:navigator-demo"
}

4.6.1 API reference

Surface	Essentials
`RbacPolicy`	`grants: [RoleGrant{principal, role}]`, `permissions: [RolePermission{role, resource, action}]`, `allows`, `roles_for`; Rust adds `decide -> RbacDecision{matched_roles, …}`
`Policy` (ODRL)	`allows(assignee, action)` — permission must match, no prohibition may
`OdrlRightsLayer`	`decide(principal, resource, action) -> OdrlDecision{rbac_allowed, odrl_allowed, allowed, receipt}`
`AccessReceipt`	`principal`, `resource`, `action` (IRI), `allowed`, `reason`, `policy_id`, `issued_at`
MCP	`check_access(principal, resource, action)` returns the full decision; denials are results

4.7 Chapter 14. Lineage and Attestations

Every governed run emits an OpenLineage event — the open standard for data lineage that Marquez and the OpenLineage ecosystem consume. QueryGraph’s events are COMPLETE run events carrying a custom facet (queryGraph_typeDid) that binds the run to the envelope that authorized it: protocol, conversation id, payload hash, signature.

Conformance is proven, not asserted. The official 2-0-2 JSON Schema is vendored in qg-python and validated with format checking on — a discipline that immediately caught a real bug: the spec requires run.runId to be a UUID, and both implementations were emitting prefixed hashes. Both now derive run ids as deterministic UUIDv5 values under a shared namespace (uuid5(NAMESPACE_URL, "https://querygraph.ai/openlineage")), from the same seeds as before; a fixture pins the cross-language derivation, and the equivalence suite schema-validates the events both CLIs emit.

Above the events sit attestations: the event’s canonical hash is signed (Ed25519, did:key verification method) into a LineageAttestation with a Merkle root, so an auditor can verify that the lineage record is exactly the one the issuer anchored. Events and attestations land in JSONL sinks and in Sail tables (qg_audit) next to the data they describe.

# Python signs an envelope and writes it out…
uv run python - <<'PY'
import json
from querygraph.typedid import TypeDidAgent
env = TypeDidAgent.new("SupervisorAgent").request(
    TypeDidAgent.new("FinanceAgent"),
    action="summarize", resource="compartment:finance",
    payload={"question": "Where is fiscal stress highest?"})
open("envelope.json", "w").write(env.model_dump_json())
PY

# …and the Rust CLI verifies it with no shared state:
cargo run -- verify-envelope --file envelope.json

{
  "payload_hash_valid": true,
  "signed": true,
  "signature_valid": true,
  "verification_method": "did:key:z6MkrdhpoFnCtEGK3RhXqryfjxVpy…",
  "scheme": "ed25519"
}

Rust resolves the did:key, reconstructs the documented signing payload (querygraph-typedid-signing-v1), and recomputes Python’s canonical JSON byte-for-byte — sort_keys, compact separators, ensure_ascii escapes and all. Change one byte anywhere and signature_valid flips to false with a non-zero exit. The same check is served at POST /v1/audit/verify-envelope and as the verify_envelope MCP tool in both languages.

4.7.2 API reference

Surface	Essentials
`OpenLineageRunEvent`	`for_agent_run(request=envelope, job_name, inputs, outputs)` (Python) / `for_dataverse_agent_run(…)` (Rust); `event_hash()` — canonical sha256
`run_id_for(seed)`	deterministic UUIDv5 under `uuid5(NAMESPACE_URL, "https://querygraph.ai/openlineage")` — identical in both languages
`LineageAttestation`	`from_event(issuer, subject, event_hash, signer=…)`, `verify()`, `signing_payload()` (`querygraph-lineage-attestation-v1`)
Validation (Python)	`validation.validate_openlineage_schema(event)` — vendored official 2-0-2 schema, format-checked; `validate_croissant`, `validate_cdif`, `validate_openlineage` shape checks
Sinks	`append_jsonl(path, event)`; Sail `qg_audit` tables; Rust `--openlineage-{file,url,sail}`

4.8 Chapter 15. The Lakehouse Path

The data side of the reference implementation:

Loaders stream CSV/TSV/XLSX into typed Parquet tables in Sail, registering views and writing a load manifest. Types are inferred, columns normalized to safe SQL names.
The Dataverse end-to-end path (dataverse-e2e) takes live Dataverse datasets through the whole chain — fetch, stage into Sail, project Croissant/CDIF/OSI, gate with RBAC+ODRL, wrap the question in a TypeDID envelope, optionally call Ollama through a DID-encrypted prompt-bound gateway (--call-ollama), and emit lineage. The --live-sail flag runs it against a real Sail server.
The demonstration corpus both books use: county government finance, household energy access, dockless-transportation trips, climate-health pathways, CODATA physical constants, and a restricted health baseline that exists to be denied.
From Python, querygraph lakehouse-register and audit-register attach the loaded tables and the audit trail to a PySpark session over Spark Connect.

spark.sql("SELECT COUNT(*) FROM global_temp.government_finance__countydata")
spark.sql("SELECT event_hash, job_name FROM global_temp.openlineage_events")

4.9 Chapter 16. The QGLake Story: The Resilience Desk

The stack’s canonical governed multi-agent narrative, implemented deterministically in both languages — the golden baseline the live loop is measured against:

A supervisor takes the question (“Where do fiscal capacity, energy burden, mobility disruption, and climate-health risk overlap?”) and delegates to six compartmentalized specialists — finance, energy, mobility, climate-health, reference data, and a restricted-data broker — each holding capabilities scoped to its own compartment (CanReadDatasetCompartment, CanDeriveRedactedSummary, …).
Specialists never share raw rows. Each returns a signed summary in a TypeDID envelope.
The restricted-data broker is denied: it returns a signed metadata-only denial receipt instead of restricted health rows — the pattern that makes denial a first-class result.
A synthesis agent sees only the signed summaries (capability: CanAggregateSummaries), never the compartments.
The run emits an OpenLineage event over all consulted (and denied) sources, anchored by an Ed25519 attestation.

Run it: cargo run -- qglake-story --json or python -m querygraph qglake-story --pretty. The equivalence suite asserts the two implementations agree on the governance semantics: same roster, the broker (and only it) denied, attestation schemas field-for-field identical.

4.9.1 API reference

Surface	Essentials
Rust	`run_qglake_story() -> QgLakeStoryReport` (title, question, supervisor, specialists, synthesis, rbac, policies, semantic_catalog, typesec, open_lineage, did_attestation); `render_qglake_story(&report)` for the readable briefing
Python	`qglake.build_python_qglake_story() -> dict` (prompt, agents, responses, synthesis, openlineage, attestation)
CLI	`qglake-story --json` (Rust) · `qglake-story --pretty` (Python); also `GET /v1/qglake/story` and the `run_qglake_story` MCP tool

4.10 Chapter 17. qg-python: The Pythonic Mirror

qg-python mirrors the same layers Pydantic-v2-first — every model above is a BaseModel with the same field names — and adds the ecosystem surfaces Python is for:

Real crypto (crypto extra): Ed25519 signing with seed-derived keys, did:key verification methods, envelope and attestation verification. Without the extra, digests are labeled unsigned:sha256: — never mistakable for signatures.
The governed navigator loop (Chapter 21) and api_auth (Chapter 18’s client side).
MCP server (mcp extra) and A2A card (Chapters 19–20).
Framework adapters: LangChain StructuredTool (sync and async), vendor-neutral to_tool_schema().
Lakehouse helpers: PySpark against Sail’s Spark Connect endpoint.

Install: pip install querygraph (core is pure Pydantic), with extras crypto, mcp, agents, validation, lakehouse, or all. The package ships py.typed; the CLI mirrors the Rust commands.

The division of labor: Rust loads and verifies the warehouse and serves the platform; Python gives notebooks, PySpark users, and agent frameworks a typed interop layer — and the two are held identical where they overlap (Chapter 22).

4.10.1 API reference (package map)

querygraph.crypto — Ed25519Signer constructors and signing methods; verify; public_key_from_did_key; unsigned_digest; CRYPTO_AVAILABLE.
querygraph.typedid — TypeDidAgent construction, request, answer, signer, DID key, and tool-schema methods; TypeDidEnvelope creation and verification; AccessReceipt; GovernedPrompt; AgentResponse; signing_payload_v1.
querygraph.navigator_loop — GovernedNavigatorLoop; answer; demo; openai_compatible_llm.
querygraph.api_auth — mint_envelope_header; governed_post.
querygraph.mcp_server — create_server; serve; demo_rights_layer; load_rights_layer; parse_action.
querygraph.a2a — build_agent_card; SKILLS; A2A_PROTOCOL_VERSION.
querygraph.agents — TypeDidLangChainToolAdapter sync and async invocation/tool methods; to_tool_schema; deterministic_specialist; TypeDidAgentRun.aggregate.
Layer mirrors — querygraph.lakehouse, lineage, osi, croissant, cdif, did, odrl, odrl_rights, rbac, and validation mirror the Rust layers from Chapters 8–15.

Extras: crypto, mcp, agents, validation, lakehouse, all; the package ships py.typed.