> Five months ago, we rebuilt our warehouse on DuckDB instead of ClickHoue

They switched from ClickHouse to DuckDB five months ago, then switched from DuckDB to their own Trino derivative?

The article then ends before saying if the new setup solved their problems.

It’s great that we can now make these migrations with little downtime (I assume?) and try different things, but I wish we could have read more about what testing they did before jumping. The fact that they jumped to DuckDB without realizing it wasn’t going to work at their scale and spent only a few months with it before jumping again is not convincing.

I hoe there’s more to this story that just didn’t make it into the blog posts. As written it’s giving me flashbacks to some teams I worked with who changed tech stacks every six months because they liked migrating more than tuning and maintaining.

Things like this are a part of why I push back on people telling me we can do important things later. I don’t want to swap the engine out of a machine that’s moving that fast. There’s no time to learn about how it behaves with real versus synthetic traffic and what the failure modes are when you’re doing many thousands of transactions per second.

Why wouldn’t you want to sort stuff like that out when the problem is still relatively small?

The only other reasonable option is to wait until the problem gets so big that it splits into several domains. And then you can move one domain over, see how that goes, and move more over if it goes well.

If you can’t know for certain how the system will behave under organic load that may mean migrating traffic onto the new system and then back to the old one several times, which means you need migrations in both directions and each transition is an opportunity for human error to destroy data or take the service offline.

Founder here -- this is about our warehouse product which is separate from our main analytics DB. Our DuckDB warehouse was used by a handful of pilot customers. Our clickhouse database is serving all of our analytical queries.

What's wrong with ClickHouse again?

I think it was closer to what they need; it comes down to keeping TB of data on local SSD and having a query engine next to it. That takes persistent, shared CPU, caching, sharding, resource management, etc.

Duckdb is an interesting approach to resource management (all ephemeral), but TB's are TB's. I doubt it was finding data where it needed it.

Clickhouse seems slow at getting to the top end of scaling; it's stuck with the 99% of its userbase doing 1 GB queries. I'm hoping they put some of their money into it.

I don’t know? They have blog posts about it but I don’t know how much I’d trust them because their chosen replacement only lasted 5 months before the next replacement.

Unless something has changed in the last year, a lot of teams have serious problems with stability.

I am curious to learn more about those issues, anyone have shared their experience publicly? would love to read

That's where it started very long ago, but they are based out of the Netherlands mostly now.

https://clickhouse.com/blog/we-stand-with-ukraine

> We have no infrastructure, servers or people in Russia. The original engineering team successfully moved out of Russia and are now working in our Amsterdam office.

There are many good reasons why the industry largely moved away from having DBAs as a specialist position. But one downside I've seen is that it's increasingly common to have teams decide to just switch platforms without fully exploring whether that's really necessary.

I'm not really surprised, data storage and data access technology is a pretty large subject all by itself, and a "jack of all, master of none" developer might never develop the expertise needed to properly optimize a database.

Am I missing something here, or is the only technical detail in the post "a trillion rows is a lot of data, maybe trying trinio will work this time"?

It feels like a lot of churn and not a lot of measurements or engineering.

[Author] it was more like: we underestimated the problem and complexity of workloads our customers want to run. We did not rewrite anything but developing a new stack we can give to our customers doing warehousing. Agreed that this context is not really clear from the blog post.

I worked at so many companies that operated this way, the tool is new but the approach is not. Those failed startups you hear about usually did a couple of these before hitting the ground

I have lost count of the number of idiotic takes like that from M2s or M3s who have no idea how things work but instead focus on keywords they can latch onto that seems to be in the intersection of all issues.

This phenomenon is at the root of the hardest organizational & architectural challenges I’ve been involved in.

It really needs a name so it can be tagged as a leadership anti-pattern.

Good leaders find real connections to explore & strengthen.

Bad leaders reach conclusions at the fuzzy pattern matching stage.

To me to key “insight” is that if you are going to spin up single VMs (or even groups) per-customer and then route queries, you need to be able to estimate the query cost first.

I know snowflake does this, they do the query optimization before sending it for execution. So it knows about how many resources will be needed (it can obviously be wrong).

[Author] Basically yes, but we spun up pods on EKS. It was hard for the customer to estimate how much they might need and hard for us to know how much capacity to provide. Given our plans on pricing on CPUHours (or similar) the logic was kinda broken.

How does Snowflake use that information? You still need to pick the warehouse. It would be cool if there was an "elastic warehouse" where heavier queries are routed to bigger Warehouses, but that's not the case

[Author] Basically what we are doing with the help of Trino: a distributed query engine which can use as many resources as the query needs dynamically.

There is an elastic option, I think . Also, even with a specific warehouse, not every query uses every node in the warehouse.

Right, there is an option to scale horizontally, if a lot of queries are queued, Snowflake can spin up another node. But that doesn't help if you query is spilling to remote storage. You need a bigger node to speed up those queries.

Snowflake has an option to add more resources, which is supposed to be vertical scaling, but it doesn't work as well as increasing the size of the warehouse

Hi! Author here.

We engineered, then we measured and then we tried to make it work, probably for a too long of a time before we decided to completely scrap our current architecture and go back to the drawing board. Happy to answer more questions.

Why didn't you just use Iceberg and Trino from the get go (or Paimon if you are fancy)? This has been the accepted answer for quite some time now. The fast OLAP stores just don't scale as well as going object store to a proper query engine like Trino.

> Happy to answer more questions.

Ah yay, I appreciate it! I feel like its rare to get a look at how these decisions cone about.

What did you measure? How'd you realise there was a problem? And did you measure anything to give you confidence that Trino won't hit the same issues?

Author here. I admit I kinda left out the context which is important here. We did not replace Clickhouse. Instead we want to move our warehouse product from Clickhouse to another stack which suits the long modeling jobs and multi-tenancy better. We learned through all our early design partners running huge workloads what we need to change.

As much as I'd love to claim victory, we are not there yet :)

This sounds more like a 5 month POC that the engineer should have been able to prove sooner that DucksDB didn’t fit the use case.

Author here. We went past the POC stage to actually make real customer workloads work, which we did. Probably yes, we spent too much time not rethinking our approach but sunken cost fallacy so that's why.

> Engineers like to experiment

The job of management is to help manage that. Typically, the desire to experiment is constrained to lab environments and so on. Experimenting on your actual users carries a lot of risks - to product stability, customer relations, market perception, etc.

Author here. This is not about our internal warehouse but the warehouse product we offer to customers. This is still running on our centralized Clickhouse. Long-running data modelling queries and multi-tenancy is a challenge on Clickhouse.

I worked at a place with a couple of overly optimistic principals who were so happy about their computationally convoluted system responding to queries in 600 ms, but it practically meant we needed almost half a CPU per request, and then pre-autoscaling, a strange requirement to be able to failover from taking a region offline so we ran each at around 60% utilization, putting us at about 1 CPU per req/s. Which is just fucking embarrassing.

The first big system I worked on is the one where I learned not to generate expensive responses from a read mostly system, instead you should make writes expensive so reads are cheap. That project had an almost equally convoluted responses, and yet we were doing 30 req/s per instance on, IIRC, 4 core boxes. Adjusted for hot spare capacity, that ancient system was servicing 4x as many requests per core, 8x as configured.

A big part of the problem, besides selling a “premium” service to customers, was drinking the flavor of Knuth’s “premature optimization” misquoted koolaid that convinces people that all substantial speed improvements come from major architectural shifts. Which is how in the time I was on that team of 6-9 people, I delivered over half of the response time and cluster size improvements. If you count the work done to support upgrading the runtime, which brought perf improvements of its own, then more like 2/3.

Sounds a bit like this team might have the same disease.

The problem with internal service teams, of which this seems to be one, is that dealing with noisy neighbors is always a bigger problem than anyone is willing to tackle because it involves confrontation. Team X gets punished every time Team Y introduces a new feature that leans on the shared resource harder. It’s a multi tenant system without the fairness guarantees that we usually build into proper multi tenant systems.

Sometimes the easiest way to solve this is sharding. You end up deploying the same service many times and routing different classes of requests to different machines so if Team Y keeps trying to wreck the Commons they slow themselves down faster than they slow everyone else down. Those sped up consequences create a feedback loop that almost works instead of an amorphous one with consequences so far removed from actions as to become academic.

On that second team I mentioned we had an unfortunate situation of introducing things that took 100 ms off of response time and then three months later we were right back where we had been due to new features and creeping complexity of user content. It was demotivating for at least half of the team. My biggest contributions lowered the slope of that line, which allowed us to finally carry improvements over from year to year instead of being on the Red Queen problem.

I wonder if they have considered moving back to ClickHouse. ClickHouse Iceberg support has improved a lot over the past few months. You can easily run stateless ClickHouse on top of Iceberg. This solves the tenancy issue, and it can easily handle petabytes scale [1].

[1] https://clickhouse.com/blog/jump-trading-uses-clickhouse-with-iceberg

Author here. We didn't leave Clickhouse. It still drives most of what Posthog offers. I missed that bit of context in my post I admit. Instead we are looking into a new stack we can give customers so they can run their warehousing workloads on it instead of the centralized CH.

Stateless CH on Iceberg sounds interesting, we looked into it but Delta/Iceberg had other downsides for us.

But why even bother with ClickHouse? What's wrong with Trino?

I kind of liked the originality. It's what happens when you give software developers free reign on the marketing pages as well, I guess.

But it seems to have gotten a re-skin recently, and I preferred the previous more pixelated version. Also, there used to be a way to render a "normal" version of the website, but I can't find it anymore.

I actually like the menus. They work great on mobile too.

We built up all these UI conventions from decades of desktop development, then abandoned them for the web (and worse, electron). It's annoying learning a new convention for every website. We could do a lot worse than bringing back menus.

Not sure their singlenode customers will be happy with Trino

https://benchmark.clickhouse.com/#system=+Hoaus|ucqs|Tuts&type=-&machine=+gel&cluster_size=-&opensource=-&hardware=+c&tuned=+n&storage=-&metric=combined&queries=-

Author here. Good point. Trino is not made for fast response times. We are using it for long running data modelling jobs or nightly ETLs.

The key insight I am getting from this is that as migrations get easier to do with agents, there will be a lot more of them.

But more importantly a lot of them will happen because we will all get more trigger happy to start one. On the other hand I also don't know if at scales like PostHog, a successful migration can be tested without doing all of it.

Now they have their "own flavor" of Trino and a brand new open table format called "HogLake". This gives the impression of making excuses with little introspection.

It would be great to see their ducklake fork, their patches and github issues so the community can learn.

Author here. Our fork, like most of our work is public on Github: https://github.com/postHog/ducklake I think it does not include 100% of all changes since a few we upstreamed directly (to a few extensions related to Ducklake too) but if you are really interested, you can find them too.

The 2026 equivalent of 2016's "Why Uber Engineering Switched from Postgres to MySQL" ! https://www.uber.com/us/en/blog/postgres-to-mysql-migration/

Is Quack, the DuckDB client-server protocol, something that could have helped them with the scaling problems they encountered?

I'm surprised at this. That is a lot of engineering effort without revealing why they chose they own format and chose to query it via Trino.

Interestingly, it seems that a data lake is the hot new thing to build in house. Hell, we did it too. If you don't mind, I wrote up exactly what these new data lakes tend to look like: https://telemetrymachine.com/blog/the-new-state-of-the-art-in-observability

The git repo [0] has a lot more of the technical details.

[0] https://github.com/PostHog/hoglake

One thing I really appreciate about CloudFlare blog posts is that often they show their work, by showing what tickets they opened, what improvements they land upstream. With this march across clickhouse, duckdb, now trino, I'd love to know what if any places the team worked with upstream, what improvements got tried.

I'm pretty ok with the march across products myself. It felt like the big shift was from shared giant clickhouse to lots of duckdbs. I wonder what juggling many more clickhouses would have been like. Excited and hopeful to here more back on Trini, with hope!

It's wild how many really really good data stacks are out there. Apache has been collecting some absolutely stellar projects. And it feels like the rest of world meaning everyone but China is kind of sleeping on a lot of these options. Maybe it's just that these projects often go out of their way to have a place for emblems, for who is using the project, but a lot of these have pretty wide ranges of companies using them.

I do wish we were still in an age of company blogging, where people did talk more to the tech getting played with, talk to what's cooking in the shop. It's great to have these outreaches. There's so much tech and we are just so quiet about it! There's so much less signal to go off of, so much less shared awareness than the aughts and 10's.