DuckDB as the Agent Query Engine Over Parquet Datasets

An AI agent querying a database is a different category of risk than a human analyst running SQL, because an agent operates at machine speed, holds persistent credentials, and carries no built-in sense of scope or consequence. A human analyst who writes a bad query notices the row count look wrong, pauses, and checks the WHERE clause. An agent with broad database access and a flawed instruction just runs it, over and over, as fast as the connection allows. Think of it as an over-eager intern handed administrator credentials and no manager in the room: the intern means well, but nothing stops the mistake before it happens.

OWASP's threat model for agentic systems names this failure mode directly. The fix isn't a smarter prompt; it's removing the agent's ability to cause that kind of damage. Agents that wander outside their intended scope also create exposure under GDPR, where least privilege supports the data minimization principle in Article 5(1)(c). That's a real compliance exposure, but the more immediate issue is operational: what happens the first time an agent gets it wrong.

Handing an agent a connection string to a production Postgres database is an incident waiting for a trigger. Removing the agent's ability to write, delete, or touch anything beyond what it was explicitly given to read is the practical answer. That means rethinking the data layer an agent touches before rethinking the agent itself, and it starts with the format that data lives in.

Parquet as a data format for agent consumption

Parquet's structure does something a live database connection cannot: it bounds what an agent can reach by what has been physically written into the file. Parquet stores data by column rather than by row, compresses it, and splits it into chunks that a query engine can read selectively. That columnar layout means an agent pulling revenue figures never touches the row that happens to also store a customer's name or email. Access is limited by which columns exist in the file, not by a permission check that has to run correctly every single time.

Parquet files are also immutable once written. An agent pointed at Parquet can't corrupt it, no matter how it's prompted or how its logic goes wrong. The write risk disappears at the format level, before any application code or access policy even gets a chance to fail.

Supabase has already built this pattern into a production product. Supabase's recommended setup keeps recent transactional data in Postgres and archives the rest to Analytics Buckets, so agents query the archive and never reach the live transactional system.

The deeper implication is about when governance happens. Materializing a governed Parquet dataset means the data an agent can reach gets defined at modeling time: a team decides which columns, which tables, which rows get exported, long before any agent sends a query. Scope is set upstream by the people building the data layer, not downstream by whatever an agent decides to ask for in the moment.

DuckDB's in-process architecture as the right engine for agents

Parquet solves where the data lives and what it exposes. DuckDB solves how an agent reads it, and it does so by running inside the same process as the agent itself rather than as a separate service the agent has to reach over a network. There's no server to provision, no cluster to keep healthy, no authentication layer sitting between the agent and the query engine. DuckDB loads in as a library, and the query runs where the agent runs.

That matters because of how DuckDB executes queries. DuckDB's SQL dialect adds PIVOT, ASOF JOIN, and GROUP BY ALL, extensions built for ad hoc analytical questions.

Two mechanisms do most of the work in making this fast. The first is predicate and projection pushdown: when an agent asks for revenue by region for Q3, DuckDB reads only the revenue and region columns, and only the row groups that match the date filter. Vectorized execution on top of that keeps memory use tight, which matters when the agent is running inside an API route or a serverless function with a hard memory ceiling.

For teams that need to scale this past a single machine, a separate distributed layer extends the same idea to distributed workloads: it reads Parquet natively from S3, GCS, or local filesystems using glob patterns, and supports distributed JOIN operations and window functions for the kind of large-scale analytics that a single DuckDB instance eventually outgrows.

The safety controls DuckDB exposes that make agent access trustworthy

Speed solves half the problem, and making sure the engine itself can't be talked into doing something destructive solves the other, with DuckDB enforcing that at the engine level rather than leaving it to application code that someone has to remember to write correctly. Read-only mode and SaaS mode are native flags, not add-on policies; they hold regardless of what SQL an agent generates or how it's prompted.

The --read-only flag turns on DuckDB's native read-only protection: no CREATE, no INSERT, no UPDATE, no DELETE. The weakest implementations skip all of that and let an agent run anything it generates. DuckDB's native flags make the stronger standard the easy path.

That's the real contrast with handing an agent a raw production database connection. An unrestricted connection makes the agent a liability by default, because every safeguard has to be configured correctly and stay that way. DuckDB over a governed Parquet dataset makes the constraint architectural: it doesn't depend on a setting nobody forgot to flip, because there's no path to a write. The performance cost is acceptable against non-encrypted, non-enclave execution, with the TPC-H SF30 benchmark running under 2x overhead. That's a demonstration that the same engine scales up to confidential computing requirements when the data behind it genuinely needs that level of protection, not just a baseline read-only flag.

The MCP server as a shared agent data layer over DuckDB-over-Parquet

A DuckDB instance running inside one API route is fast and safe, but it's also local to that one application. Putting DuckDB behind a standard protocol server changes that: it turns a local query engine into a shared, protocol-addressable data layer that any agent, any tool, and any human-facing surface can consume without custom integration work for each one.

That became easier to operate at scale once the MCP specification updated on July 28, 2026 to make the protocol stateless. A remote MCP server no longer needs sticky sessions, a shared session store, or deep packet inspection sitting at the gateway. It can run behind a plain round-robin load balancer, which strips out a meaningful chunk of the operational complexity that used to come with running MCP infrastructure. That same shift makes it simpler to instrument MCP servers for analytics, showing teams how their tools are actually being used and which tools are sitting unused, using the same infrastructure that already serves agent queries.

Tableau's announcement at Tableau Conference 2026 points to where this is heading industry-wide: open MCP servers built to deliver trusted analytics to Slack, Salesforce, Claude, ChatGPT, and any other MCP-compatible surface. Tableau put it directly: people shouldn't have to go to the dashboard for the truth, because the truth comes to them, wherever they're already working.

That reach comes with a caution that shouldn't get glossed over. The U.S. National Security Agency stated in a May 2026 report that MCP's rapid adoption has outpaced the development of its security model. That gap makes the governed Parquet layer underneath the MCP server more important rather than less, because it means agents only ever reach pre-approved, read-only data no matter what vulnerabilities exist at the protocol level above it. The protocol can mature over time; the data boundary underneath it has to hold regardless.

dbt v2 and DuckLake as the production default for this pairing

The dbt-duckdb adapter has existed since its first pull request landed on August 27, 2021, and it has built up 1.4k stars on GitHub in the years since. That's a community that found this pairing useful well before anyone made it official. dbt 2.0.0, released on September 14, 2026 and built on the new Rust-based Fusion engine, is the release that makes it official: it's the first dbt version to ship with a built-in DuckDB adapter, no separate Python package required, with the DuckDB driver downloaded and cached automatically the first time it runs.

Under the hood, dbt v2 connects adapters through ADBC drivers inside a Rust monorepo, a structural choice that embeds DuckDB more deeply into the transformation layer rather than treating it as an optional extra bolted on the side. DuckLake moves all catalog and table metadata into a SQL database while keeping the actual table data as Parquet files in object storage, with no manifest lists, no Hive Metastore, and no extra catalog API service to run and maintain.

dbt v2 even writes its own internal metadata as Parquet. On large projects, the old JSON manifest file could balloon to hundreds of megabytes; the Parquet version lets DuckDB read only the specific columns a query needs instead of loading the entire file into memory first.

The DuckDB-Iceberg extension has kept pace on the write side too. As of January 2026, it offers INSERT, UPDATE, and DELETE support for Iceberg v2 tables, with UPDATE and DELETE limited to tables that aren't partitioned or sorted. Taken together, these releases mark the point where DuckDB-over-Parquet moves from a clever workaround that a few teams discovered early to a supported default that dbt ships with out of the box.

Where DuckDB-over-Parquet has real limits

None of this makes DuckDB a general-purpose database, and it was never meant to be one. DuckDB isn't built for high-concurrency write workloads or systems with many simultaneous users all writing at once. An application that needs many concurrent writes per second, or that serves many dashboard users performing writes at the same time, still needs Postgres to handle transactions, with a separate layer handling analytics. The pattern described here is read-only analytical access, and stretching it past that boundary is where teams run into trouble.

The DuckDB MCP server is the lowest-friction way to hand an agent real analytical capability with no infrastructure to stand up or own. But local-first also means governance is only as strong as whatever the local environment happens to provide. It isn't a shared source of truth across a team unless someone deliberately builds a governed layer on top of it, with defined datasets and defined scope, rather than letting each agent point at whatever Parquet files happen to be sitting nearby.

Queries an agent generates still need a human to check them. An agent can inspect data, calculate metrics, and produce a report, but someone should confirm the calculations and the interpretation actually hold up. That's not specific to DuckDB; it applies to any workflow where an agent is doing the analytical work.

If the governance behind the Parquet layer is thin, DuckDB just becomes a faster way for an agent to reach ungoverned data. That's true, and the answer is that governance was never the query engine's job to begin with. It lives in the modeling step, in the decision about what gets materialized, which columns get included, and what schema gets exposed. DuckDB enforces the boundary once it's drawn, but somebody upstream has to draw it.

For teams that would rather not leave Postgres at all, pg_duckdb offers a Postgres-native alternative. It has reached a stable release and runs the DuckDB engine inside PostgreSQL as an extension, delivering real speedups on analytical queries with no code changes required. The constraint that matters here: enable it on a dedicated read replica, not the primary. Analytical queries are CPU-intensive, and isolating them onto a replica means complex analytical work never slows down write latency on the transactional node that the rest of the application depends on.

What this architecture means for AI-native data workflows

Adopting governed Parquet datasets queried through DuckDB via MCP is a decision about more than which query engine to run. It's a decision to give agents and humans the same pre-modeled, trustworthy version of the data, rather than letting each build its own path to a different version of the truth.

For a team already on Supabase, the stack falls into place cleanly: Postgres stays the transactional home for live application data, materialized Parquet datasets become the governed analytical layer, DuckDB serves as the query engine for both dashboards and agents, and MCP is the protocol that makes that data layer reachable from whatever surface a person or an agent happens to be working in.

That shift changes what the analyst role actually does. Once agents take over data retrieval, anomaly detection, and routine report generation, the people who define metrics, relationships, and business logic become the function that actually matters. Decisions about who gets to query the data give way to decisions about what agents are allowed to see and calculate.

Starting small serves teams better than starting comprehensive. A handful of decision-critical metrics, updated automatically and pushed to where leaders already work, beats a sprawling dashboard nobody opens. Readiness matters before any of this gets built: teams with a reasonable history of clean, consistent data get the most out of this pattern quickly, while teams whose data is scattered across tools with no consistent schema need to build the governed layer first, before pointing any agent at it.