MotherDuck CEO says single-node analytics can outperform distributed systems

I spoke with Jordan Tigani, CEO of MotherDuck, about a question analytics teams should ask before building a cluster: does this workload actually need one?

MotherDuck is a cloud analytics service built on DuckDB, an open-source SQL database for analytics; and before co-founding it, Tigani helped build Google BigQuery as an early engineer and led product at SingleStore.

We discussed when distributed systems are worth the complexity and how analytics teams can build and govern simpler alternatives.

Choose architecture based on what queries actually process, not just total data size

First I wanted to satisfy my curiosity about how BigQuery was built, so I asked Tigani what the starting point had been.

He said it was the need for distributed systems, which divide work across multiple machines. I then asked what had convinced him that many modern analytics workloads might no longer require that approach and he said:

When I was working on BigQuery I used to run a lot of analytics to understand our customers and what they were doing. I noticed that most of our customers didn’t have a lot of data. And even the ones who did have a lot of data, many of their queries only processed a small amount of data at once. We had some of the largest customers in the world, Walmart, HSBC, Equifax. They were all using a tool called “Big Query” but the queries they were running were not big at all.

Tigani also noted that he mostly just filed that information away as a curiosity. Later, while he was working at SingleStore to help build its SaaS database product, he began exploring the machines available on AWS. “I was shocked; you could get a machine with hundreds of cores and terabytes of RAM, and it wasn’t even expensive,” Jordan explained.

The third piece that tied it all together was when I saw how fast you could implement things in a single-node system. I had been following DuckDB for a little bit, and they added a join optimization that was similar to something we’d done in BigQuery years earlier.

That optimization took months to build in BigQuery because of all its moving parts. DuckDB built it in days and shipped it alongside other improvements. “That was shocking, just how fast you could move if you didn’t have to deal with all of the distributed systems machinery.”

So machines are getting bigger. But workloads are not. And distributed systems add a tax, both in the performance you can possibly expect but also the time it takes to build and improve things. I put those together and thought, “wow, if we were going to create BigQuery today, we would have done it very differently.”

Isolate workloads to avoid slowdowns and complex resource management

From there, I turned to query execution in MotherDuck: how it decides whether queries run locally, in the cloud (or both) and how developers can check.

“For context, MotherDuck has a feature we call “dual execution” which means we can run queries either locally or in the cloud,” Jordan explained.

He also explained that this lets MotherDuck integrate seamlessly with applications already using DuckDB, and also lets customers build highly performant applications where results get cached locally and follow-up queries can run out of that local cache:

However, we never run anything locally unless the user makes it clear that is what they want to do. For example, if they’re using a local duckdb database and then they join it against a MotherDuck database, we’ll run part of the query locally and part in the cloud.

He said there are also cases where users want to run entirely in the cloud and don’t want the dual-execution machinery, which includes a copy of DuckDB. For those cases, they can use MotherDuck’s PostgreSQL endpoint, which makes MotherDuck look just like a PostgreSQL server.

Speaking about the advantages and limitations of MotherDuck’s hypertenancy model compared with a traditional shared data warehouse, where multiple users or workloads draw on shared computing resources, Jordan noted that hypertenancy is a fancy way of saying “everybody gets a duck.” In other words, if you have 100 users, every one of those users gets their own individual DuckDB instance running in the cloud.

“We combine that with serverless execution, so we can start virtually instantaneously and shut down as soon as queries complete,” Tigani said and explained: 

The rationale from this was after seeing how complicated workload management was in other databases. If you have more than one user or workload accessing the same hardware, you have to figure out how to share resources. If you go from one user to 10 users querying concurrently, either that first user’s queries are going to slow down, the new users are going to run slowly, or you had vastly over-provisioned.

What I also learned from Jordan is that workload management tends to get really complicated, both for users and for people building the database. For example, Tigani said, configuring workload management in Teradata is so complicated that one of their most anticipated features was a mechanism for machine learning to be able to help you tune it correctly. In other words, it was too hard for humans to do it well:

For MotherDuck we thought, what if we made it so every workload was a different completely isolated database? Then workload management is simple. You don’t need to worry about one user slowing down another one, or complex priority schemes to decide who got what resources. You also don’t have to worry about over or under provisioning.

The other advantage of this, he added, is that it scales really nicely with how people actually use an analytical database. There are often a couple of users who have really heavy workloads and a lot of users who have much lighter workloads. So the heavy users scale up to big instances, the lighter users get tiny ones. And none of them slow each other down.

“If you’re building a SaaS application this ends up being even more useful, because you can give each one of your end users a different database. In other words, let’s say you’re building a stock portfolio app and you want to give your users the ability to analyze their holdings,” Jordan explained.

According to Tigani, each of those users gets a different DuckDB instance; these instances turn off when not being used, so it doesn’t matter how many users you have: you only pay for what’s actually running. And all of your users are nicely isolated from each other.

Storage formats and PostgreSQL compatibility

Our conversation also moved to DuckLake, Apache Iceberg and Delta Lake, technologies for organizing tables and managing their metadata in a data lake.

I asked Tigani what problem DuckLake was designed to solve that Iceberg and Delta Lake did not solve well enough. His detailed answer was:

Do you remember when everyone was putting things on the blockchain, and a handful of sane people said “wait, this is basically just an incredibly complicated database that can only do a couple of updates per second.” That’s kind of what Iceberg and Delta are; they’re a super complicated way of storing data on S3 and can only do one or two updates per second.

DuckLake he expleined, is a cleaner way of envisioning how a data lake can work by storing the metadata about how to find the active bits of the table in a database instead of on S3. This means it is a lot faster to find what you’re looking for as well as makes it much more reliable to make updates.

Jordan also mentioned that people say that a lot of the weaknesses of Iceberg are being solved in subsequent versions, but it is very hard to get around architectural limitations by adding complexity on top:

Of course, a lot of people use Iceberg, and there is a lot of momentum behind it. So while we believe that DuckLake is a better way to store your data, Iceberg isn’t likely going anywhere. So we have first class support for iceberg, including the various Iceberg catalogs, in MotherDuck.

MotherDuck’s support for the PostgreSQL wire protocol, the communication protocol used by PostgreSQL clients and servers, raised another practical topic. I asked where compatibility with Postgres ends and what differences developers should understand before using it in production.

Explaining this, Tigani said the primary reason for Postgres wire protocol support is that it creates virtually universal connectivity. A lot of people can connect to DuckDB, but virtually everyone knows how to connect to Postgres.

He also noted that the secondary reason is that a lot of people use Postgres for things that are really analytics and then start to hit the limits of what Postgres can do. “By switching to the postgres wire protocol, we can make it super easy to switch to MotherDuck; you don’t have to change your queries, your connector, or your code; it should just work,” Jordan explained and added:

That said, I should add all the usual caveats. While DuckDB SQL is mostly a superset of Postgres SQL, it is not a strict superset. That is, the vast majority of Postgres functions work in DuckDB, but there are some corner cases where things might be missing. There are also cases where some things may work subtly differently in DuckDB than in postgres. We’re hard at work on ensuring compatibility, but when you take a large system as complex as Postgres, there are going to be some differences.

Treat AI agents like any production tool: grant only the access they need

An important part of our conversation focused on AI agents querying production data through the Model Context Protocol, or MCP, a standard for connecting AI applications to external tools and data sources. I asked Jordan which security and governance controls teams should put in place before enabling that access. He explained:

There isn’t anything special about MCP that isn’t true with other AI tools, or with other tools in general. It is easy, and perhaps natural, for people to see tools that can produce magical results and to jump in and want to start using them in production. However, if set up poorly, these magical tools can cause embarrassing security holes. The principle of least privilege is a good one to follow.

He also urged teams to be careful about access permissions:

Be careful who can add new tools. Trim security groups. Watch out for confused deputies (things that run as privileged users that provide services to other users can accidentally give those other users access to things they shouldn’t be able to see or do).

Building on this theme, Tigani said that it is also easy to have a knee-jerk reaction to security and risk. However, it is important to think through the actual threat models. What is the actual harm that can be done? What things would happen in order for that harm to be achieved? And then what can you do to mitigate those situations?

Match review effort to the risk

Before the interview ended, I asked Jordan how engineering teams should test, review and monitor work produced by AI agents as they generate more SQL queries, data pipelines and analytics applications.

He answered that at MotherDuck, virtually everyone in the company creates Dives, which are their vibe-coded dashboards. According to Tigani, most of these people wouldn’t be comfortable using SQL, and yet they use these tools to help them in their jobs every day:

For most cases, the advantage of having a quick and easy answer is well worth the possibility that it will be wrong. And the benefit of having the line-of-business users actually generate the visualizations is that they are the ones who can detect when something goes wrong; they are going to be better than an analyst on a centralized data team to know that the revenue from a particular customer looks off.

Tigani also mentioned that the amount of scrutiny varies depending on how important it is that the answer is right. If you’re reporting numbers to a regulator, you probably need them to be right. But the vast majority of use cases are not actually like that.

“If you’re building dashboards to understand customers in your sales book, or trying to gauge the success of your marketing campaigns, rapid iteration is going to be more important than perfection,” Jordan said, adding:

Additionally, in my experience, the quality of AI written SQL is higher than human written SQL. You really don’t have to review it to figure if it is going to work. You don’t need to check to see if there is an off-by-one error in the bounds of the window function, or if it passed the arguments correctly. Important visualizations, important data pipelines, and context should be all treated like product code. Stored in source control. Changed via pull-requests with commit messages showing what was actually changed as well.

“Even if it is AI writing the code, you should make sure it is auditable by a human. But as AI gets better, humans are going to be doing less code review to check it to see if it is right and more auditing to see what happened when something goes wrong,” Jordan concluded.