6 Part IV: Integration in Practice

6.1 Chapter 23. Assembling the Lakehouse in Ten Steps

The dedicated book walks this assembly in depth; here is the whole system in one sitting — the sequence the dataverse-e2e and lakehouse commands automate, each step naming the component that owns it:

1. Load the lakehouse (Sail). Stream CSV/TSV/XLSX — or live Dataverse datasets — into typed Parquet tables; register views; write the load manifest.
2. Materialize Semantic Croissant (Chapter 8). A JSON-LD sidecar per dataset: files, record sets, fields, semantic types.
3. Project CDIF (Chapter 9). Discovery, access, and profile metadata derived from the Croissant document — never hand-maintained.
4. Build the OSI semantic model (Chapter 12). Fields become governed SAIL_SQL expressions; metrics get per-dialect SQL; ontology terms link shared meaning.
5. Load the Grust graph (Chapters 1–2). Datasets, fields, terms, agents, and policies become nodes and edges — queryable by Cypher, including from Spark via the Sail extension.
6. Identify agents with DIDs (Chapter 10). Deterministic did:oyd documents from seeds; the same seed yields the same identity in both languages.
7. Apply ODRL rights (Chapter 11). Permissions and prohibitions per dataset; the read/index/derive defaults that make training-by-default impossible.
8. Mint TypeSec capabilities (Chapter 3). CanReadDatasetCompartment, CanDeriveRedactedSummary, CanAggregateSummaries, … — proofs, not flags.
9. Route to compartmentalized agents (Chapter 16). Supervisor → specialists → restricted broker → synthesis; raw rows never cross compartments; denials are signed receipts.
10. Call the model through TypeDID (Chapters 4, 21). The prompt travels DID-encrypted and prompt-bound to Ollama (or any OpenAI-compatible endpoint via the loop); the reply comes back in a signed envelope; the run lands in OpenLineage with an Ed25519 attestation.

# The compressed version of all ten:
cargo run -- dataverse-e2e --live-sail --call-ollama \
  --question "Which governed datasets are relevant?"

6.2 Chapter 24. End to End: Catalog to Governed Answer

The full path, using only released commands:

# 1. Start the platform.
cargo run -- serve --port 8080 --require-auth

# 2. Import semantics — from a Croissant document (or LakeCat's
#    bootstrap projection, Chapter 6, or an OSI YAML).
python - <<'PY'
from querygraph.api_auth import governed_post
from querygraph.typedid import TypeDidAgent
agent = TypeDidAgent.new("ApiClient")
croissant = {
    "name": "Energy Burden",
    "recordSet": [{"field": [
        {"name": "monthly_cost", "description": "Monthly energy cost"}]}],
}
print(governed_post("http://127.0.0.1:8080",
                    "/v1/models/import/croissant", croissant, agent))
# {'imported': 'energy_burden_semantic_model', 'datasets': 1, …}
PY

# 3. Search it — open route, no auth needed for reads.
curl -s 'http://127.0.0.1:8080/v1/search?q=monthly'
# {"matches": [{"kind": "field", "name": "monthly_cost", …}]}

# 4. Ask — governed route, signed envelope required.
python - <<'PY'
from querygraph.api_auth import governed_post
from querygraph.typedid import TypeDidAgent
result = governed_post("http://127.0.0.1:8080", "/v1/answer",
    {"question": "What drives monthly energy burden?"},
    TypeDidAgent.new("ApiClient"))
print(result["answer"])          # planned over sail.qg_lakehouse.energy_burden
print(result["envelope"]["payload_sha256"][:16])  # …and it's signed
PY

# 5. Verify the answer's envelope — anyone can, with no shared state.
curl -s -X POST http://127.0.0.1:8080/v1/audit/verify-envelope \
  -H 'content-type: application/json' -d @answer-envelope.json

Every hop leaves evidence: the import is authorized by a path-bound envelope, the answer carries its plans and receipts, and the verification endpoint lets a third party check the signature.

6.3 Chapter 25. Live Dataverse to Governed Answer

The second integration walkthrough runs the whole chain over live research data: Harvard Dataverse, the largest public Dataverse instance. One command — every step of Chapter 23, against data QueryGraph has never seen:

cargo run -- dataverse-e2e \
  --dataverse-url https://dataverse.harvard.edu \
  --query "municipal finance" --limit 2 \
  --question "Which governed datasets describe municipal fiscal capacity?" \
  --openlineage-file lineage.jsonl --did-ledger-file did-ledger.jsonl

What happened, from a real run:

1. Live search and staging. The Dataverse Search API returned two real datasets — Graduate School Rankings for Public Administration Programs (doi:10.7910/DVN/YTAB7V) and Data Files for Criminal Municipal Courts (doi:10.7910/DVN/AXNUVP) — whose metadata and file listings were staged into Sail as typed views (dataverse_7424459_metadata, dataverse_7424459_files, …).

2. Semantics, derived not hand-written. Each dataset projected to Croissant; an OSI model (querygraph_dataverse_navigator) was built over both, with ontology terms from their subjects and keywords; the four-layer bundle was generated for the first dataset.

3. The dual gate, with a receipt. The agent’s answer action passed RBAC (role navigator matched) and ODRL, and the receipt says so:

{
  "action": "answer",
  "allowed": true,
  "principal": "did:oyd:zQmX7Cm5eikMzwMavBT4zoTUAsWxK3zGwqSbFHu729gb41r",
  "rbac": {"allowed": true, "matched_roles": ["navigator"]},
  "odrl_allowed": true,
  "reason": "TypeSec capabilities, RBAC role assignment, and ODRL
             policy allow the answer path"
}

4. The signed request. The governed prompt — question plus the staged metadata, nothing else — traveled in a typedid/a2a envelope (signature: a16f2cba…), ready for --call-ollama to carry it, DID-encrypted and prompt-bound, to a local model.

5. Lineage over DOIs. The OpenLineage event lists the DOIs as inputs (doi:10.7910/DVN/YTAB7V, doi:10.7910/DVN/AXNUVP), carries the semantic bundle hash as a job facet, gets a spec-conformant run id (f0539cd9-853f-589b-a70f-a9371f532c20 — UUIDv5, derived from the envelope signature), lands in lineage.jsonl, and is anchored by an Ed25519 attestation whose issuer is a resolvable did:key.

Add --live-sail --sail-endpoint http://127.0.0.1:50051 to execute against a running Sail server (views become queryable from PySpark), --call-ollama --ollama-model llama3.2 for live synthesis through the TypeDID gateway, --anchor-codata to anchor the bundle DID with the CODATA ODRL service, and --openlineage-url http://localhost:5000 to emit straight into Marquez.

The point of this walkthrough: nothing above was fixture data. The evidence chain — receipts, envelopes, DOI-level lineage, attestation — assembled itself around unfamiliar, live research data with one command.

6.4 Chapter 26. Plugging In Agent Frameworks

Claude Code / Desktop (MCP, stdio). Point the client at either server:

{"mcpServers": {"querygraph": {
    "command": "querygraph", "args": ["mcp-serve", "--osi", "model.yaml"]}}}

LangChain. Either consume the MCP server via langchain-mcp-adapters, or wrap governed agents natively:

from querygraph.agents import TypeDidLangChainToolAdapter, deterministic_specialist
from querygraph.typedid import TypeDidAgent

finance = TypeDidAgent.new("FinanceAgent")
adapter = TypeDidLangChainToolAdapter(
    finance, deterministic_specialist(finance, summary="Fiscal summary…"))
tool = adapter.as_async_tool()          # a StructuredTool; results carry envelopes

PydanticAI / OpenAI Agents SDK / LlamaIndex / CrewAI. All speak MCP — querygraph mcp-serve is the one integration. For direct function-calling, export schemas with to_tool_schema() (Chapter 20).

Local and hosted LLMs. The navigator loop binds any OpenAI-compatible endpoint (openai_compatible_llm(base_url, model)); the Rust Ollama path wraps calls in DID-encrypted, prompt-bound envelopes so even the inference call is attributable.

Your own governance. Both the MCP server and the loop accept your OSI model (--osi) and your RBAC+ODRL policy (--rights governance.json — {"rbac": {...}, "odrl": {...}}); the demo policies are defaults, not assumptions.

6.5 Chapter 27. Operating and Releasing

Build and test.

cd qg-rust && cargo test          # 40 tests; clippy -D warnings in CI
cd qg-python && uv sync --extra all && uv run pytest   # 49 tests,
                                  # including the live cross-language contract

Run. serve (with --require-auth in anything shared), mcp-serve, qglake-story, answer, navigator, verify-envelope, lakecat-import, dataverse-e2e --live-sail --call-ollama for the full kitchen sink.

CI. GitHub Actions in every repo; qg-rust’s workflow assembles the sibling grust/lakecat layout to satisfy the path dependencies; qg-python builds the wheel and twine checks it on a 3.11/3.13 matrix.

Release discipline. SemVer with codename pools per repo (birds of prey, Venetian landmarks, wild cats); coordinated stack waves; CHANGELOGs and RELEASES logs; versioned book artifacts (stem (version-hash).{epub,pdf}) published per release; GitHub releases with attached wheels.

Documentation. Two books in qg-rust/docs — the dedicated QueryGraph book (book/) and this guide (guide/) — plus the review deck (slides/) and the one-pager in HTML, typst, and troff (onepager/), all rebuilt at each release hash.

6.5.1 API reference (the CLIs)

qg-rust (cargo run -- <command> or the querygraph binary):

qg-python (querygraph <command>): navigator, anchor-url, qglake-story, lakehouse-register, audit-register, pyspark-examples, answer (--osi, --rights, --llm-base-url, --llm-model), agent-card, mcp-serve (--osi, --rights, --transport).