2024 will be the year of cloud-native burnout
In 2024 developers' professional lives will not be as impacted by new tools and frameworks as by - mental health.
DHH and 37signals are saving $7 million by leaving the cloud. Should you do the same?
Self-hosting is not for everyone. Find out if it's the right fit for you.
Hardware is fun again, wrote DHH when he posted a photo of two pallets of R7625 Dell servers.
It’s been a long time since I last saw a physical piece of hardware used to run our services at 37signals.
David Heinemeier Hansson, or DHH, the cofounder of project management tool Basecamp and emailing product Hey (and the founder of Ruby on Rails), has been detailing their “exit from the cloud” for some time now. He announced 37signals were leaving the cloud in October 2022 and has been chronicling the journey in detail since, advocating for medium-sized companies to stop bleeding server money:
Renting computers is (mostly) a bad deal for medium-sized companies like ours with stable growth. The savings promised in reduced complexity never materialized.
David Heinemeier Hansson, 37signals
And the world paid attention. It seemed the time was high for CTOs or people responsible for infra costs to start voicing their frustration with their cloud bills.
We’ve talked to three people in charge of infra at medium-sized global companies with a significant amount of load, and it turns out DHH may even be late to the party!
Doing the math
At ReversingLabs, a software supply chain security company, on-prem data centers were their choice from the get-go. The main reason was cost, as they process a vast amount of data – even in 2011 when they started, it was a financially sound choice. Their VP of Technology Igor Lasic shares that they evaluate their strategy every two years, and the numbers are always in favor of self-hosting:
Hosting everything on AWS would cost us approximately 10 to 15 times more than self-hosting – even when you include renting servers and salaries for engineers managing data centers into the calculation.
Igor Lasic, ReversingLabs
37signal’s eye-watering cloud bill corroborates that. In 2022 they spent $3,201,564 on cloud services and almost a million on AWS! Since their project management application, Basecamp, is run almost entirely on their servers, this is just the bill for their emailing service, Hey, which was initially completely cloud-based.
The savings from the migration come to about $1,5 million a year. And we’ll have much faster hardware, many more cores, incredibly cheaper NVMe storage, and room to expand at a very low cost, he adds.
When SofaScore, a sports data company, did their math, the result was the same. Self-hosting just made more financial sense, says CTO Josip Stuhli:
Not an easy decision, but we’ve reached a stage where we processed so much data that we paid more for the traffic than for the servers. Thad led us to move just one layer of servers that generated most of the traffic out of the cloud to control the cost. When making that analysis, we accounted for a way oversized setup, and the cost was several times lower than our cloud spend at the time.
Josip Stuhli, SofaScore
The migration was smooth, with some minor iterations and optimizations. After a while, SofaScore moved the whole thing to Kubernetes to get some of the benefits of the cloud, like smart load balance and self-healing. SofaScore’s engineer even shared the technical details on their “descent to bare metal Kubernetes” at Shift Conference:
But what about operations and maintenance? SofaScore and 37signal have taken different approaches to handling this issue, but the cost-benefit ratio still favored self-hosting.
In his post, DHH said they would not have to add people to their ten-people operations team, as their servers will be delivered to the Deft colocation facility, where the on-site facility team sets them up. The 37signals employees never physically touch the actual servers but can manage them remotely.
SofaScore’s approach was more hands-on but is still not labor-intensive for the ops team:
I will be honest, for the first four years of our transition, no team was in charge of infrastructure; it was just me. It was really challenging, and it took a lot of experimentation with various tools and solutions. The magic of the cloud is that it offers many excellent ready-to-use tools, and we had to find alternatives. We’ve eventually formed a DevOps team, and now they spend a small portion of their time managing the infrastructure.
Cloud is still the future… For some
Hansson clearly states he wants to challenge the notion that “the cloud is the future” and that companies running their own data centers and owning, instead of renting, servers are somehow missing out or living in the past.
Any mid-sized SaaS business and above with stable workloads that do not benchmark their rental bill for servers in the cloud against buying their own boxes are committing financial malpractice at this point.
ReversingLabs Lasic tends to agree, noting that there has been a louder and louder resistance towards public cloud among engineers in the last couple of years:
It started when companies realized they never got to experience the benefits promised in the sales and marketing pitches. Even if you migrated all of your infra to the public cloud – which would cost you a lot, you’d still need in-house employees managing it.
However, there is some fine print in those statements. As DHH says, renting computers is a bad deal for medium-sized companies with stable growth. Note the details: a medium-sized company with steady growth. Setting up in the cloud is still a great way to go for a small business just starting or companies with highly irregular loads who can’t predict whether they would need ten servers or a hundred.
Stuhli notes that cloud also offers many fantastic tools that make setting up infrastructure straightforward and quick, and sometimes using those services makes sense:
I would 100% recommend the cloud for any project just starting. Cloud also makes sense for companies with a monthly or yearly subscription business model because they can model their system for a predictable number of users. In that case, the infrastructure cost is neglectable compared to the revenue brought by the user.
Setting up in the cloud is also a fantastic solution for companies that are just starting or are growing fast and need to scale quickly. At Infobip, a global omnichannel communication services provider, they prefer a hybrid multi-cloud strategy – out of their 40 international locations, some are smaller and set up in the public cloud, and some are bigger and have on-prem servers.
Oliver Lovakovic, Infobip’s Infrastructure Engineering Director, says hosting in the cloud is an excellent choice when you have to set up quickly to test a market:
It may cost more, but sometimes you don’t have the internal expertise or the time to build it. As the load starts growing at a location, it makes more sense to start building on-prem equipment and expertise. The increase in cloud cost is more or less linear, while on-prem investments balance out with scale.
Make it a matter of choice, not necessity
Whether you’re choosing cloud, self-hosting, or hybrid, it’s essential to keep your options open – circumstances, as well as business plans, can change. Having an experienced team prepared to switch between hosting setups is essential for risk mitigation, Lovakovic points out:
You want to be in a situation where you choose the cloud because it suits your needs and makes sense for you in terms of cost or flexibility. Not because you don’t have in-house experts and are locked in with a cloud provider with no way out in the short or mid-term.
Oliver Lovakovic, Infobip
Besides a lack of expertise in self-hosting, you should also be wary of “vendor lock-in,” warns Lasic from ReversingLabs. That could force you to make decisions not in your best interest:
If we hosted all of our data on one public cloud and wanted to switch to another public cloud or start our own setup, the cost of that change and getting everything out would be so significant that we would be forced to stay whether we wanted or not.
Their real challenge, however, is not deciding on one or the other; it’s defining the current and future needs of your product, Stuhli concludes:
Indeed there are companies like ours for which their own infrastructure makes more financial sense, even when you add the cost of engineers. Companies that would benefit the most from migrating to bare metal systems are those that generate and process a lot of data or offer a free app. Of course, there is no silver bullet when it comes to infra, and the real challenge is to know the product and the business model in-depth and then custom design the infrastructure to fit that particular product.
To spice the debate up towards the end, let’s also throw in some cultural context in the mix. Considering the number of layoffs at tech companies and startups needing to cut costs in the downturn, there is merit in examining whether the cloud is the best option. If not, follow DHH’s advice and cut cloud costs before cutting payroll.
It may be more complicated than laying people off, but it’s worth it.
