14 Python Interop

Rust is the control plane for Querygraph, but Python is the working surface for many of the people who will use it. Data scientists live in notebooks. Spark users expect PySpark. AI engineers assemble agents with Python libraries. Analysts want to inspect a Sail warehouse without learning the Rust internals. The sister project qg-python exists for that world.

The first principle is that responsible AI must meet practitioners where they work without abandoning the system’s guarantees. If the Rust implementation is precise but the Python notebook path is loose, the platform fails. If the notebook can bypass ODRL, TypeDID, or lineage, the local-first story collapses. The Python ecosystem therefore mirrors the same concepts with Python-native tools rather than inventing a separate, weaker layer.

The Python implementation is not a toy wrapper around a command line. It is a Python-native ecosystem over the same concepts:

• croissant.py and cdif.py build Semantic Croissant and CDIF JSON-LD.
• osi.py projects Semantic Croissant fields into OSI datasets, metrics, expressions, and ontology terms.
• typedid.py models TypeDID envelopes, governed prompts, access receipts, and signed agent responses with Pydantic.
• agents.py exposes compartment agents and an optional LangChain StructuredTool adapter.
• lakehouse.py registers Sail warehouse Parquet tables in a PySpark/Spark Connect session.
• lineage.py emits OpenLineage run events and DID-style attestations.
• odrl_rights.py and rbac.py give Python agents the same rights checks used in the Rust story.
• dataverse.py projects Dataverse native API payloads into Semantic Croissant.
• qglake.py runs the Python version of the supervised QG Lakehouse story.

This division is deliberate. Rust is where Querygraph wants tight control: ingestion, typing, hashing, policy boundaries, graph staging, and reproducible CLI workflows. Python is where Querygraph wants fluent exploration: notebooks, PySpark queries, LangChain tool composition, and domain-agent iteration.

Textbook rule: Python is the laboratory; Rust is the contract. Both must speak the same semantic, policy, identity, and lineage language.

14.1 Pydantic

Pydantic is the Python side’s type boundary. A TypeDID envelope should not be a loose dictionary passed from one agent to another. A governed prompt should have a question, semantic context, allowed sources, denied sources, and access receipts. An agent response should have a status, summary, evidence, redactions, and an envelope hash. Pydantic makes those shapes explicit while remaining natural for Python users.

In qg-python, TypeDidEnvelope validates the request/reply structure and recomputes payload hashes. GovernedPrompt carries the semantic context that came from Croissant, CDIF, OSI, and Sail. AgentResponse preserves the signed summary or denial. The synthesis agent receives those models, not ad hoc JSON.

The point is not type ceremony. The point is that Python agents can be creative without being unbounded. Pydantic gives the agent framework a shape that can be validated, logged, hashed, and compared to the Rust implementation.

14.2 LangChain

LangChain fits as an adapter layer, not as the source of authority. Querygraph does not ask LangChain to decide whether a model may see restricted data. That decision belongs to DID identity, ODRL policy, TypeSec capability checks, and the semantic target in Croissant/CDIF/OSI. LangChain receives a governed tool only after those boundaries exist.

The Python adapter TypeDidLangChainToolAdapter turns a TypeDID agent into a LangChain StructuredTool. The tool returns the same signed response or denial that a non-LangChain caller would receive. This keeps LangChain useful while preventing it from becoming a policy bypass.

Example: a LangChain planner may choose FinanceAgent to summarize fiscal capacity. The actual tool invocation still goes through a TypeDID request, policy receipt, payload hash, and signed response. If the planner asks the restricted broker for raw health rows, the tool returns a signed denial.

14.3 PySpark

PySpark is the inspection and analysis surface for Sail. The Rust loader materializes the warehouse. Python registers the generated Parquet tables into a Spark Connect session and lets analysts query them with familiar Spark SQL. That is how a user can inspect qg_lakehouse without leaving the Python world.

The current Python helper can register the loaded data tables and the audit tables:

uv run querygraph lakehouse-register \
  --manifest ../qg-rust/.querygraph/lakehouse/manifest/load-report.json \
  --warehouse ../qg-rust/spark-warehouse

uv run querygraph audit-register --warehouse ../qg-rust/spark-warehouse

Then a notebook or shell can ask:

spark.sql("SELECT COUNT(*) FROM global_temp.government_finance__countydata").show()
spark.sql("SELECT quantity, value, unit FROM global_temp.codata_constants_2022__codata_constants_2022 LIMIT 5").show(truncate=False)
spark.sql("SELECT event_hash, event_type, job_name FROM global_temp.openlineage_events LIMIT 10").show(truncate=False)

That is the interop story in one loop: Rust loads, Sail stores, Python queries, Pydantic agents reason, LangChain orchestrates when useful, and OpenLineage records the trail.