When I first encountered MongoDB, I tried to understand it through the database I knew best: PostgreSQL. My journey was a baptism by fire. I was managing a sharded cluster with 20 TB of data and as many as ~55k operations per second, including ~29k writes. The cluster had six shards, each with its own replica set, a config server replica set, and six routers.

To operate that cluster safely, I had to stop treating MongoDB like PostgreSQL and learn how it worked internally.

At one point, while tackling a flow control problem on the stated cluster, an AI model recommended increasing the portion of memory assigned to the WiredTiger cache. At the moment, the recommendation sounded genuine and correct, but I did not know what the model had based its conclusions on. To be completely honest, I didn’t even know how RAM is used by MongoDB. The nail in the coffin was the official docs saying don’t change WiredTiger cache to RAM ratio.

That experience clarified why I wanted to learn the internals. I want/need enough understanding to connect operational advice to a mechanism, identify the evidence it depends on, and decide. This applies whether the advice comes from AI, a colleague, documentation, or my own assumptions.

What happens after a MongoDB client sends a write?

How does that write pass through MongoDB and WiredTiger, and when does it become durable on disk?

And when someone recommends changing a WiredTiger setting, what would have to be true for that recommendation to make sense?

Following one write

When a client sends a write operation to a sharded cluster, the request first reaches a router called mongos. Routers cache data from the config server about chunk – shard placements. Using this data mongos determine which shard owns the relevant data and forward the operation to that shard’s primary node.

However, this post does not compare replica sets and sharded clusters. Once routing is complete, the same question applies inside every shard.

How does the primary mongod process request, persist it, and replicate it to secondary nodes?

Separation of concern

To answer my question(s), I firstly had to understand where mongod ends and the storage engine WiredTiger begins.

mongod handles client commands, authorization, query execution, transaction coordination, and replication. WiredTiger is the embedded storage engine that stores collection and index data. It provides local transactions, MVCC, caching, compression, journaling, checkpoints, and crash recovery.

MongoDB client
      │ write operation
      ▼
mongos query router
      │ routes to the owning shard
      ▼
shard primary (mongod)
      ├── authorization and command handling
      ├── query parsing and execution
      ├── transaction and session coordination
      ├── replication and oplog coordination
      └── storage-engine API
                  │
                  ▼
             WiredTiger
                  ├── local transactions and MVCC
                  ├── internal cache
                  ├── journal
                  ├── checkpoints
                  └── collection and index data in dbPath

Ok, but how does this come into play? Boundaries are not absolute.

The oplog belongs to MongoDB’s (logical) replication model, but WiredTiger stores it alongside other collection data.

One write, two logs

At this point, I had another question. If MongoDB already has a journal, why does it also need an oplog?

The WiredTiger journal is the closest equivalent to PostgreSQL’s WAL (Write Ahead Log). It records the changes needed to recover one MongoDB member (keep this thought) after a crash. Unlike PostgreSQL’s WAL, MongoDB does not use the journal as its replication stream.

The oplog has a different purpose. It is a capped collection stored in the local database as local.oplog.rs. Secondary members copy its entries and apply them to their own data.

This raises an important consistency question. What prevents MongoDB from committing a document change without its corresponding oplog entry?

Data folder

Let’s examine what happens on one node and then additionally complicate the story by adding secondaries.

To understand local durability, I first looked inside the dbPath of a MongoDB member. A simplified listing:

$ ls /data/mongodb/
collection-<ident>.wt
index-<ident>.wt
journal/
WiredTiger
WiredTigerHS.wt
WiredTiger.lock
WiredTiger.turtle
WiredTiger.wt

The exact files and their names depend on the MongoDB version and configuration, but these are the main WiredTiger files relevant to this discussion.

The collection-<ident>.wt and index-<ident>.wt files are WiredTiger tables that store collection records and index entries.

MongoDB’s durable catalog maps logical objects, such as a collection namespace, to these storage-engine identifiers. WiredTiger maintains its own metadata in WiredTiger.wt, including its tables, their configuration, and their latest checkpoints.

The small WiredTiger text file identifies the WiredTiger version used to create the database:

WiredTiger
WiredTiger 10.0.2: (November 30, 2021)

WiredTigerHS.wt is the history store. It keeps older committed versions of records needed by MVCC readers, snapshots, etc. WiredTiger removes versions once no active reader or snapshot can require them.

WiredTiger.turtle contains enough metadata to locate and open the latest checkpoint of WiredTiger.wt. It bootstraps the metadata table during startup and recovery.

WiredTiger version string
WiredTiger 10.0.2: (November 30, 2021)
WiredTiger version
major=10,minor=0,patch=2
file:WiredTiger.wt
...
checkpoint_lsn=(2248,16291200)
...

The checkpoint_lsn identifies a position in the WiredTiger journal using a log file number and byte offset. During recovery, WiredTiger opens the last consistent checkpoint and replays later journal records.

The journal/ directory contains those write-ahead log files. They protect changes made after the last checkpoint from being lost during an unexpected shutdown.

Finally, WiredTiger.lock is the file on which WiredTiger acquires a lock to prevent two processes from opening the same database directory simultaneously. The lock state matters, not merely the existence of the file.

The WiredTiger cache

WiredTiger does not modify the collection-*.wt and index-*.wt files directly for every client operation. It first reads the required pages into its internal cache and applies the change there as part of a storage transaction. Modified pages become dirty until WiredTiger reconciles and writes them to disk.

This internal cache keeps uncompressed collection data and is separate from the operating system’s filesystem cache where compressed data is stored.

MongoDB therefore benefits from both caches, WiredTiger and the OS cache.

Giving all available memory to the WiredTiger cache would leave too little memory for the filesystem cache and the rest of mongod and respecting the default, max((availRAM - 1024) x 0.5, 256 MB), is strongly advised by the official docs.

This is already enough to show why “increase the WiredTiger cache” is incomplete advice. A larger cache does not create memory. It transfers memory away from the filesystem cache and other allocations. Whether that trade is useful depends on the workload and the source of the observed pressure.

When the cache approaches its limits, WiredTiger evicts pages to make room. Clean pages can be discarded and read again later. Dirty pages must first be reconciled into an on-disk representation.

A committed change does not have to wait for its dirty page to reach the collection file. The journal supplies durability between checkpoints. This separation is the reason a write can be durable even though its final data page is still dirty in memory.

Recovering after a crash

In case of a crash, WiredTiger starts from the last complete checkpoint. The checkpoint metadata identifies a consistent view of the WiredTiger tables and the journal position associated with it.

WiredTiger then replays the journal records created after that checkpoint. At a high level, recovery combines two inputs:

 last complete checkpoint + later durable journal records = recovered local state  

Checkpoints and data files

WiredTiger normally creates a checkpoint every (configurable default) 60 seconds. A checkpoint can take longer when there is more dirty data or the storage device is under pressure.

During a checkpoint, WiredTiger writes a point-in-time view of dirty pages to the collection, index, history-store, and metadata files.

WiredTiger transaction
        │
        ├── dirty pages in cache ──checkpoint──▶ .wt data files
        │
        └── journal records ───────────────────▶ journal files

                         crash
                           │
                           ▼
       latest checkpoint + subsequent journal records
                           │
                           ▼
                    recovered state

How the primary records an operation

For an incoming query, mongod will, among other things, determine collection and index changes and construct the oplog logical entry.

It uses WiredTiger to start/use a transaction and either commits both parts or commits neither. This prevents a successful document change from existing on the primary without an operation (oplog entry) that secondaries can replicate.

The oplog entry contains an operation type, namespace, timestamp, and the data required to reproduce the change. Its timestamp provides an ordering position in the replica-set history. The entry describes a logical insert, update, delete, command, or transaction operation rather than the low-level page modifications stored in the WiredTiger journal.

rs_a [direct: primary] local> db.oplog.rs.findOne()
{
  ts: Timestamp(...),
  t: NumberLong(...),
  op: "u",
  ns: "foo.bar",
  o: { $v: 2, diff: { ... } },
  o2: { _id: 123 }
}

This is not a complete definition of how oplog and collection changes work, but the important invariant is: the primary commits the replicated data changes together with the oplog records which describe them.

Only the local commit is atomic. Sending or copying the oplog entry across the network is not part of that storage transaction.

How secondaries copy and apply oplog entries

Each secondary continuously selects a sync source and streams newer oplog entries from it. The sync source is often the primary.

The secondary writes fetched entries to its own oplog and then applies the represented operations to its local collections and indexes.

The secondary uses its own WiredTiger instance to apply changes. They modify its cache, produce local journal records, and later become part of its checkpoints.

This is the important distinction and the answer why we can’t just ship primaries *.wt binaries to the secondary.

On two MongoDB servers, these are different files.

Oplog versus journal

Takeaways

This gives a practical sequence for evaluating operational changes:

1. Identify which component owns the behavior
2. Describe the mechanism that is a cause of the problem
3. Find evidence supporting that mechanism
4. Clarify what the improvements should be
5. Measure
6. Iterate

The technical facts in this article support that way of reasoning:

• The WiredTiger journal and the oplog solve different problems. The journal recovers one member; the oplog replicates logical operations between members.
• MongoDB commits a replicated data change and its oplog entry in the same local storage transaction on the primary.
• A durable change can still exist as a dirty cache page because the journal protects it between checkpoints.
• Primaries and secondaries have separate WiredTiger databases, caches, journals, and data files.
• Memory assigned to WiredTiger is part of a larger allocation decision that includes the filesystem cache and other process memory.

Why I wanted to understand this

Learning how WiredTiger works does not mean that I can derive every production decision from first principles. It gives me a way to examine a recommendation instead of accepting it because the source sounds confident, and we all know AI can sound like that.

The quality of an (AI) recommendation depends on the question and the evidence supplied to it.

Further reading

• MongoDB manual: WiredTiger storage engine
• MongoDB manual: Journaling
• MongoDB manual: Replica set oplog
• MongoDB manual: Replica set synchronization
• MongoDB source documentation: Storage engine

A note on scope – This article records my current understanding of MongoDB and WiredTiger, built through hands-on operation and continued study. I have tried to keep the technical details accurate, but any errors or oversimplifications are mine. MongoDB official docs and WiredTiger documentation should always be consulted for correctness and clarity.

The post MongoDB: One write, two logs appeared first on ShiftMag.

This post describes a failure mode we addressed while operating Debezium with PostgreSQL 16 and Patroni 3.3.1.

After promoting a standby, Debezium needed a logical replication slot from which it could safely continue. Creating a new slot after promotion could start beyond Debezium’s last durable offset and introduce an event gap.

The central lesson was to treat PostgreSQL’s replication slot and Debezium’s Kafka Connect offset as parts of one distributed checkpoint.

The pipeline

I have worked with applications that use the transactional outbox pattern. A business change and the corresponding outbox record are committed in the same database transaction. Debezium reads changes to the outbox table from PostgreSQL’s write-ahead log, transforms them into the event format expected by consumers, and publishes them to Kafka.

This removes the unsafe gap between committing a database transaction and separately attempting to publish an event. If Debezium or Kafka is temporarily unavailable, the committed outbox row remains available for later processing.

The pipeline provides at-least-once delivery. A failure can occur after an event reaches Kafka but before the corresponding source offset is durably recorded. Debezium can then replay the event after restarting, so consumers must be idempotent.

WAL refresher

WAL and LSN

PostgreSQL reads the relevant data page into memory (shared buffers). The page is modified in memory, and a corresponding WAL (write-ahead log) record is created. The WAL record is placed in the WAL buffers, and the corresponding page is marked as dirty with a page LSN (log sequence number). The page LSN records which WAL entry corresponds to the page’s latest changes. Before that dirty page can be written to its table or index file, the corresponding WAL must be flushed to durable storage.

wal_level = logical

Changing wal_level from its default value of replica to logical is required to support logical decoding for logical replication. This change requires a full server restart. WAL includes additional information (catalog snapshots, changes, transaction boundaries, etc.) so that row changes can be reconstructed.

One checkpoint across two systems

PostgreSQL and Kafka Connect each retain part of the connector’s progress:

• PostgreSQL’s logical replication slot retains the WAL required by the connector.
• Kafka Connect stores Debezium’s most recently committed source offset in its offset topic.

Conceptually, progress moves through the system like this:

PostgreSQL generates WAL
      └─ the logical slot exposes changes
            └─ Debezium streams and decodes
                  ├─ advances slot's LSN → PG recycles older WAL
                  └─ commits source offset to Kafka topic

These positions do not remain identical at every instant. PostgreSQL, Debezium, and Kafka progress independently. The important invariant is:

After failover, PostgreSQL must retain every WAL record Debezium might request from its last durable Kafka Connect offset.

Starting from an older position may replay events. Starting from a newer position may skip them. Replay is recoverable with idempotent consumers; an LSN gap is not. The accompanying PostgreSQL 16 and Debezium playground can be used to inspect slots, publications, and connector offsets in a local cluster.

The PostgreSQL 16 failover problem

PostgreSQL 16 supports logical decoding and logical replication slots on a standby. However, it does not automatically synchronize the primary’s logical slot state with the standbys.

Before failover, Debezium consumes through a logical slot on the primary while Kafka Connect stores its durable source offset:

If the standby is promoted without a usable copy of that slot, Debezium cannot simply continue from the same checkpoint. Creating a replacement slot on the new primary is not equivalent. A new slot begins from a position available at creation time and cannot be moved backward to recover WAL that is no longer retained.

No events were lost in our case. We identified this as a failure mode that had to be eliminated before the standby could safely accept Debezium after promotion.

Patroni permanent logical slots

Alexander Kukushkin (the Patroni guy) explains how Patroni solved this problem with permanent replication slots in a podcast with Nikolay Samokhvalov .

As described in Patroni’s official documentation for permanent replication slots, logical slots are copied from the primary to standby nodes and their positions are advanced periodically. Patroni’s slot synchronization implementation creates missing logical slots on replicas by copying them from the primary and advances existing slots when their confirmed position falls behind.

Permanent logical slots require postgresql.use_slots to be enabled. Patroni also enforces the hot_standby_feedback setting on nodes that host permanent logical slots so the primary retains catalog rows needed for logical decoding.

Potential problems:

• The requested WAL segment pg_wal/XXX has been removed.
• The physical slot is behind the logical slot (unlikely to happen).
  • The physical slot did not reach the catalog_xmin transaction of the logical slot on the old primary. Patroni logs a warning that it might be unsafe to use.

Zeno’s paradox

Enabling permanent slots on an already busy cluster was more difficult than enabling them on a new cluster.

While a standby was restarting and catching up, the logical slot on the primary continued to advance. PostgreSQL could remove catalog tuples required for logical decoding before the copied slot became usable on the standby. The standby then invalidated (wal_status = lost) the slot because it conflicted with recovery:

This slot has been invalidated because it was conflicting with recovery.

This was not simply a matter of retaining more WAL. Logical decoding also depends on catalog visibility represented by catalog_xmin. Increasing wal_keep_size alone could not restore catalog rows that recovery and vacuum had already made unavailable. Increasing max_standby_streaming_delay also did not help while the standby still had to be rebooted.

The system was effectively chasing a moving checkpoint: by the time the standby came back and attempted to use the copied state, the required catalog horizon had already moved.

How we approached the transition

Instead of treating the operation as a reusable sequence of commands, we treated it as a checkpoint-alignment problem.

In our Patroni 3.3.1 rollout, allowing Patroni to introduce missing permanent logical slots required restarting the busy standbys so that the copied slot state could take effect. Those restarts created the race described above: by the time a standby returned, the required catalog horizon could already have moved.

The turning point was to create the corresponding logical slot directly on each PostgreSQL 16 standby before enabling Patroni’s permanent-slot configuration. PostgreSQL could create those slots without restarting the standbys. We aligned each slot only to a checkpoint that we had verified as safe, then reloaded the Patroni configuration. Because the slots already existed, Patroni adopted them and continued advancing their positions instead of entering the missing-slot copy path.

This is intentionally an analysis rather than a reusable runbook. The safe checkpoint depends on the primary, standby replay position, retained WAL, and Debezium’s durable offset; advancing a slot too far can create the event gap this process is meant to prevent.

The property we validated was:

Every candidate primary retained a usable logical slot covering every WAL record Debezium might request after promotion.

Subsequent production failovers completed without detected LSN gaps or skipped events.

What PostgreSQL 17 changes

PostgreSQL 17 introduced synchronized failover slots. A logical slot marked for failover can be synchronized from the primary to a standby using PostgreSQL’s native slot-synchronization facilities. The official PostgreSQL 17 logical replication failover documentation describes the configuration and readiness checks.

This reduces the need for PostgreSQL 16-era slot-copying approaches, but it does not remove the operational responsibility. Synchronization is asynchronous, and failover readiness must still be verified before promotion.

For PostgreSQL 17 and newer deployments, use the native failover-slot design and its readiness checks instead of copying this PostgreSQL 16 and Patroni 3.3.1 approach.

Takeaways

1. Treat PostgreSQL’s replication slot and Debezium’s offset as one distributed checkpoint.
2. Prefer replay with idempotent consumers over any possibility of an LSN gap.
3. Make logical-slot continuity part of the database failover design.
4. Distinguish a healthy connector from a proven gap-free recovery.
5. Test promotion with real writes and checkpoint comparisons, not only process-health checks.
6. Scope operational guidance to the PostgreSQL and Patroni versions for which it was validated.

This blog post was originally published on Frane Jelavic’s blog.

The post How to Survive Database Failover – Debezium and PostgreSQL in Production appeared first on ShiftMag.

The task at hand was to migrate from the AWS relational PostgreSQL database to our on-prem self-hosted solution. We assembled a mission-based team comprising a Developer, DevOps, and two DBAs.

The main driver of this migration was the steadily increasing monthly bill we got from AWS. If you don’t read all the fine print when navigating through AWS UI, you could be unpleasantly surprised by the amount of dollars you end up spending.

But, we chose AWS for a valid reason. We had a POC (Proof Of Concept) and wanted to quickly gather customer feedback.

We chose PostgreSQL at that time and opted for an “enhanced” Aurora version. Although we had an on-premise (in-house) solution, it was never battle-tested.

From there, our journey began…

Choose your technologies carefully

Our cluster was AWS Aurora PostgreSQL RDS with one writer and 4 standby servers. The servers were db.r5.8xlarge instances with 32 vCPU and 256 GB RAM with Aurora storage.

The storage solution provides almost instant (approx. 20 ms) replication, with 4 out of 6 commit quorum.

All servers use the same underlying storage, so commits from writers are propagated instantly to standbys, and asynchronous replication updates only the data in memory (shared buffers).

Reference from AWS PostgreSQL In-Depth Training.

So we had help from the beginning…

Replication was instant, so we hooked up the whole UI on our standby servers and got an immediate real-time response. Our solution was built on the assumption that changes from the writer server will be, nearly instantly, available to our standbys.

The writer server was offloaded by efficiently using the standbys. We used Enhanced monitoring, set up Cloud Watch dashboards, analyzed and tracked every query on Performance Insight, and created alarms for every anomaly spotted in the setup.

The level of observability and monitoring let us diagnose issues very quickly and correctly. Little did we know that that behavior differed widely from Vanilla Postgres. But we would find out soon enough.

Think about your data early

Our database DDL was structured to support the product launch and was never meant to be a long-term solution.

Data was added but never deleted. Tables were not partitioned, and indexes have metastasized with various workloads. This resulted in tables with terabytes and indexes with hundreds of gigabytes of size.

We had to restructure. Keep the traffic data together and distinguish it from read-only and configuration data.

Archiving was a very long and IO-intensive feature to develop, and it clashed with the migration efforts. During the product lifecycle, we had stale and test data we wanted to get rid of. We started data cleanup and removed gigabytes of data.

Again conflicting with the migration process.

AWS charges everything

We didn’t start with a huge cluster, but with time and feature development, user adoption, and traffic increase, we had proportional demands on our underlying DB hardware.

Fast forward to being approx. 4-5 years on the market, in production, we had huge I/O usages of our standbys. UI was sucking in two standbys, and their usage was through the roof in peak traffic hours, while the other standbys were battling the remaining read queries.

We also hooked up some analytic reports on one of the servers to provide better statistics for our users.

The monthly bill was rising steadily. Although we neglected costs apart from instance uptime, every I/O request had its price and so did our setup.

On-premise setup

Although our cluster was not tested with the same traffic AWS was handling at the moment, our setup was powerful.

We had complete data center redundancy, meaning if a whole DC crashes, we could failover to the other one. Each DC had servers spread across three availability zones, so redundancy on the network level was also covered. Cluster orchestration with Patroni allowed for smooth failovers/switchovers and performing upgrades without downtime or maintenance window scheduling.

Hardware-wise, we had Intel’s Platinum CPUs, a few generations better than we had provisioned on AWS. Storage was NVMe SSDs, so IO bottlenecks we had on AWS should be history.

Use your standbys with caution

PostgreSQL allows you to use standby servers in a variety of ways:

• High availability where the standbys’ primary purpose is not to fall back from the writer and to replay WAL logs
• Offloading big queries, AKA poor men’s data warehousing
• Horizontal scaling, performing select queries to off-load the writer database

Every setup comes with its pros and cons, leading to streaming replication conflicts. We needed to choose whether we preferred replication changes or serving the current queries running on the standby(s).

While the abilities of configuration are broad, they are not bulletproof. The development mindset had to change, since we were now dealing with streaming replication instead of the one provided by Aurora (which was almost completely synchronous without latency increase).

Applications using standbys need to cope with eventual consistency. UI architecture for displaying real-time data from standbys in the new setup became an oxymoron from a logical perspective.

Limitations of logical replication

Logical replication was used to transfer data from the current cluster to AWS. We had previous experience migrating data but on a completely different DDL layout (remember the portion of the text “Think about your data early“).

The replication process has two phases. Initial data sync, where all the data currently in the table/schema is migrated, and the “catch-up” phase where the cluster syncs the subsequent changes from WAL logs.

Since logical replication does not copy DDL changes to the target server, all changes were frozen for other developers working on the product.

Our efforts were primarily aimed at reducing the cost, but we’ve managed to give ourselves some extra work in order to bring more value and earnings to the product.

The migration started by introducing replication slots on the source (AWS cluster) and subscribers on the target (on-premise). The first strategy, empirically, was to define one replication slot per database schema, as we did in our previous migration(s). Smaller schemas, up to 50 GB in size, were synced extremely fast and we experienced no issues.

Incidents from database degradations

By the time we got to our main schema counting several TB of size, the first incident happened. During the initial sync phase, WAL logs with subsequent changes are kept on the primary server. Given the way PostgreSQL provides transaction isolation, initial sync is thought of as a long-running transaction on the primary instance.

The snapshot xmin value is held and with it the event horizon. When this value is locked all queries “see” the DB state from the moment the transaction with xmin value started. In layman’s terms, every query has to start the execution from the snapshot of the DB that it was in at the moment of the ongoing transaction and go through changes (tuple versions) applied in the meantime.

This results in prolonged query cost, inaccurate database statistics, and the most problematic one bloating of the database since the vacuum is postponed. Approximately 5-6 hours in the replication we experienced complete degradation. IOPS got exhausted and we hit AWS limits, for our database instance level. We had to stop the replication, lose all the data currently migrated, and start over.

On the second attempt, armed with the experience we had from the first execution, we first divided our replication slots inside the schemas. Tables small in size were merged into a single replication slot, and every other table got its dedicated replication slot. Additional monitoring and alarms were set up to check the bloat and queries that were first impacted in the previous execution. Bigger tables were stripped from their indexes, and we decided to add them after the complete sync was done. The only index kept was the primary key one, since logical replication needs at least one unique constraint upon applying subsequent changes from WAL logs.

The migration was still a struggle.

The whole team was checking dozens of metrics and also browsing through our UI solution examining delays upon usage. We had to stop the migration and start over more than once, but this time it affected only a single, problematic table. Once we got to the biggest table inside our database, we needed to remove the primary key as well and recreate it once the initial sync was replicated.

We managed to copy all of the data to our on-premise cluster and were in the sync phase. Features such as data removal were halted until we fully migrated and we were now maintaining two separate clusters.

Load tests with business-specific scenarios

Data was replicated, clusters were up and running, DDL was synced, applications were working without issues, and the overhead of logically replicating changes didn’t affect the AWS cluster. Now what?

We needed to put our HW to the test, and we needed a fallback if we ran into problems.

This is the moment where we regretted not putting that extra time into having a load test. Not any kind of load test – the one that specifically mimics our production behavior, a user on our solution, and the ability to multiply it by a custom number.

Luckily, the majority of our load was SELECTs. We used our load balancers to include standby servers from the on-prem solution together with production ones. This allowed us to easily fall back, and kick out of the balancer, in case of degradation.

The major downside of our approach was that every latency introduced in the system was immediately affecting the client. We have repeatable tests that monitor the solution and, in 99% of the cases, spot the degradation before the user does, but there is still that 1%. Apart from “testing in production”, another major downside is that there were too many unknowns inside the equation to have exact conclusions. There is always a probability something else is affecting the results when you are part of a live system.

During these tests, we saw that our backups were disturbing our workloads and that we would need to introduce a separate server for full and incremental backups, something we didn’t even think about while we were on AWS.

Amzon’s Performance Insight is worth every penny. Only when we stopped using it and lost the ability to check the lifecycle of every executed query and pinpoint the exact wait event causing the delay did we realize how powerful it was.

We had to implement our solution, and we did, but again, we learned that we took some things for granted when using AWS.

The patient recovered in no time

The final maintenance was scheduled for the complete migration of all services to the new on-prem solution. Here our preparation was on point. We’ve completely automated the process, we just needed to execute a few CLI commands to gracefully migrate the whole solution. Given the usage of our product, the only window we had, with minimal traffic and client impact, was from 0:00-3:00 UTC. Since we are in CET this meant a long night for us.

We were struggling not to fall asleep before the maintenance started and would probably struggle to sleep afterward. The migration went surgically precise, and our patient recovered in no time. The window was scheduled for a full three hours, and we managed to redeploy, clean up error logs, and validate in a little more than an hour.

Although falling asleep was hard because of the adrenaline spikes, the sense of accomplishment helped us rest our tired eyes.

Life after migration

Looking back at the journey of the migration, a pragmatic observer could spot the good and the bad of each setup we had. Product maturity, access to powerful on-prem hardware, gathered know-how inside the company and cost reduction made the decision of the migration a no-brainer.

However, there is a lot of work now on our hands that we just “threw money at” when using the cloud. Maybe the best thing to understand as an organization and as a developer is to recognize when you are ready to make this step.

Identify a step in your product lifecycle where such moves have to be taken. With increased usage of your solution, you should think about scalability and cost-efficiency not only on service level or feature bottlenecks but the ecosystem as a whole.

Definitely choose your technologies wisely in the beginning. When you introduce new tech (platform change,  database, framework, or even a library), build up expertise, check the knowledge inside the company, understand and take responsibility for bugs in the tech now being your bugs.

Don’t forget to think about scale. Premature optimization is the root of all evil, they say, but nevertheless, your design has to have a vision.

These decisions are not easy, but neither is our job. We are fortunate to grow as individuals, learning by doing and owning our decisions, making us better developers but also human beings.

The post Database migration: Developers’ open-heart surgery appeared first on ShiftMag.