It was a great vacation, packed with long walks, beach time, fun, and rest.

I came back to work rejuvenated, knowing I’d need a few days to get back into the working rhythm. Most of the day was spent catching up with colleagues and the latest news.

One was about a brand-new data center, ready to go live in just a few days.

“Production-ready” turned into “data-empty”

“It turns out delivering new data centers is becoming almost routine,” I thought to myself, feeling as light as a breeze.

After catching up with my colleagues, I was greeted by a mountain of unread emails. Honestly, I can’t tackle that for hours without feeling my will to live slowly evaporate. So, I decided to shift my focus and check out the new data center, just to see how everything was set up…

My fresh bronze tan didn’t help – I went pale as a ghost: one of the key databases in our shiny new “production-ready” data center was completely empty.

No data. Nothing. Niente. Nada.

The problem? We needed a full month of historical data available FROM DAY ONE. And all we got were… good vibes and positive energy.

Two days, no data, one hero

The clock was ticking. We had maybe two days (probably less) until go-live. Yes, it was an emergency, and no, there wasn’t a second to waste.

We turned to our trusty old Replay app, hoping it could save the day.

And once again – it delivered!

The shining star of our toolkit, Replay came through, regenerating over a month’s worth of data before the deadline.

In the end, the data was restored, the deadline met, and our sanity… mostly intact. But one thing was clear: returning from vacation had never felt more like a horror story. And you? Keep your tools close and your backups closer.

The post The Day Our Data Center Went Ghost appeared first on ShiftMag.

Picture this scenario: You’ve just joined a new tech company, excited to start onboarding and dive into your first projects. But soon, it hits you – the tension.

Developers gripe about sluggish and obstructive DBAs, while Database Administrators grumble and murmur about the constant stream of poorly optimized queries, chaotic code, and last-minute demands from developers.

Did you stumble upon a toxic workplace? Not exactly. It’s the age-old clash that’s been brewing for decades between two ‘tribes’: those who want to build quickly and those who safeguard stability.

So, do DBAs play the role of protectors, or are devs just a tad bit spoiled?

How did we get here?

The friction comes from a classic disconnect: speed vs. stability.

Developers are eager to ship features as fast as they can (preferably yesterday), often under pressure from clients, management, or some other deadline-driven force, while DBAs are focused on ensuring the database doesn’t crash tomorrow.

When these priorities clash, chaos is inevitable – and I know this firsthand because I’ve been on both sides.

Let’s take a closer look at it from a developer’s angle

• ‘We need to speed up our delivery!’

The main argument is that developers often work under tight deadlines. To them, a database is just a tool – the simpler, the better. Why involve DBAs early if it slows down prototyping?

There’s also a prevailing ‘good enough for now’ mentality, where getting a working prototype is the priority.

• ‘Setting up new databases takes ages.’

Waiting for approvals, scheme checks, or infrastructure setups feels like watching paint dry.

• ‘Getting DBAs involved can feel like pulling teeth.’

When DBAs push back on quick fixes or demand documentation, developers see red tape rather than guardianship.

Now, let’s dive into the DBA’s side of the story

• ‘We’re called in too late!’

DBAs often inherit poorly designed schemas or performance nightmares. Fixing a burning database at 2 a.m. is never fun, especially when you don’t know anything about it or where to begin.

• ‘Devs don’t listen.’

Recommendations about indexing, query optimization, or security often fall on deaf ears – until an outage happens. No matter how many times you share documentation, hold knowledge-sharing sessions, or discuss the topic, it sometimes feels like no one listens.

• ‘We’re the unsung janitors.’

While devs chase innovation, DBAs clean up the mess. Their priorities – backups, scalability, and compliance – may not be glamorous, but they’re non-negotiable.

If a company wants to make it in the worldwide market, it needs to be compliant with numerous regulations. And that is just the beginning – data needs to be secured, and data leaks and privacy concerns aren’t fun for anyone. We need backups and replication set up for disaster recovery, satisfy numerous architecture decision records, and all that complicates the process. It will never be as simple and fast as just deploying a DB locally in Docker.

So, what do devs and DBAs really want?

DBAs just want preventative care – they’re like doctors urging you to eat your veggies, not just treating heart attacks.

On the other hand, devs need speed without fallout. They want to innovate without waking up to a dumpster fire.

But do devs really want to own databases?

Sure, devs can deploy their own database, but maintaining performance, security, and costs? That’s a full-time job – one many devs aren’t signing up for.

It may sound great at first, but once you deploy it, the database works fine – until it doesn’t. It’s just a matter of time before the first issues arise. And surely, devs would rather spend their time programming than maintaining and troubleshooting a database.

Think it can’t get worse? Well, say hello to AI

Enter AI and vector databases – tools promising lightning-fast analytics.

The number of new vector databases is skyrocketing, and devs want them all, each with a specific feature they need. But for DBAs, this is a nightmare. Integrating a new database is a slow and painful process, as all checks for security, compliance, and requirements need to be tested and approved. New database types (e.g., vector DBs for ML) often lack mature tooling or expertise. DBAs scramble to secure and scale them, while devs resent the learning curve.

So, who is right?

They both are, and at the same time, they both aren’t – much like the relationship between devs and clients.

Devs often complain about clients’ broad demands, tight deadlines, and constant changes. Sound familiar? The Dev vs DBA battle is similar: devs forget that, as clients of DBAs, they’re the ones pushing for deadlines and changes, just like clients do to them.

So, devs must make compromises with clients because the company’s revenue depends on it. However, I think DBAs tend to be ‘harder’ on their clients since they don’t face the same pressure of potentially losing business.

Is there hope for peace?

There isn’t – at least not a simple one.

But we can all work together to reduce tension and aim for some kind of truce. We need to be more understanding.

Devs should recognize that DBAs aren’t just complicating things for no reason and should involve them in the service architecture process. On the other hand, DBAs should view devs as clients, understand the need for compromise, and simplify the process of providing databases by using automation or internal tools whenever possible.

The post Devs and DBAs can’t find peace, but could they call a truce? appeared first on ShiftMag.

Occasionally, everyone needs to handle data stored in files. CSV files seem like the way to go and are often the first choice – but I’d rather call them an anti-pattern.

Let’s say a customer requests an Excel report containing specific data. If this is a one-time thing, we can conveniently send it via email. In most cases, however, the data needs to be sent regularly. Not all clients use sophisticated software like Kafka; even when they do, we don’t always have access to it. So we need a solution that is both universally available and easily understood. SFTP and CSV files seemingly fit the bill.

This solution seems optimal for sending data to an external client. The problem arises when an internal application needs to integrate with ours, requires some data, and we decide to solve it by exporting CSVs. At first glance, this seems a low-hanging fruit.

Everyone can download a file.

Everyone can read a CSV.

It’s already developed.

While it may appear that simple, ultimately, it proves to be more costly than using a dedicated component for data exchange. I learned my lesson the hard way. I’ve worked on both applications that sent and received data via files. Here are some of the issues I discovered along the way.

 Data storage

The first thing we need is a place to store the data. Since our data is stored in files, it must be placed within a file system. The decision we face is whether to store the files on the producer machine, consumer machine, or an intermediary machine. Each option has its advantages and disadvantages.

Using the producer machine results in the consumer bearing the entire networking load. The consumer will need to periodically retrieve the data through a PULL operation. In the event of a network link failure, the producer will remain unaffected. 

However, this solution has a drawback – when consumers are redundant, and there are multiple instances, each instance may attempt to download the same file. Consequently, the same data will be processed multiple times. To address this, each instance should reserve the file before initiating the download. This can be accomplished by appending a suffix to the file name or moving the file to another folder. Once the file is reserved, the consumer application can proceed with the file download.

Using the consumer machine puts all the networking burden on the producer. The producer needs to PUSH the data to the consumer machine once the file is created. The consumer machine can watch the local file system for changes and react when a new file is created. As a result, the push-based approach can yield a faster response by the consumer and does not put any networking burden on the consumer. Since the file is sent directly to the consumer instance, this approach does not need file reservation. 

Pushing files to consumers also has a considerable downside – the producer must balance these files between the consumer instances. Implementing effective client-side balancing is a tough nut to crack. The producer would need to adapt its balancing strategy when a consumer instance gets offline or online and is added to or removed from the group.

Using an intermediary machine combines elements of both previously mentioned approaches. It eliminates the need for client-side balancing, but the consumer still needs to reserve the file before downloading it. The data storage is independent of both the consumer and producer, although both parties are still required to transfer files over the network.

Data retention

Regardless of the chosen method for storing data, it is crucial to ensure that the disk is not filled with files. Failing to do so will render the machines (and the services running on them) unusable. The primary line of defense is to store files on a separate section of the disk. 

Even if the partition gets filled with files, it will not impact other services. This advice applies even when files are stored on an intermediary machine, as some operating system functionalities may not work properly if the disk space is occupied.

Accumulating files can happen quite easily. For instance, the consumer application may be down (or not deployed) while the producer application continues to generate files. It can also occur if the producer generates a high volume of data the consumer cannot keep up with. 

In such cases, it is necessary to prevent the files from overflowing the disk. If the data represents events and skipping a few events in the event of a failure is acceptable, a retention policy should be implemented. 

For example, a script that periodically deletes the oldest files can be developed. This script adds another component to the architecture that needs to be created, maintained, and monitored. If the script malfunctions or contains a bug, the data retention may not work as intended.

Data processing

A file is removed from the disk when a consumer has read and processed the whole file content. The ‘whole file content’ is emphasized since a file represents a batch of data. Errors or failures can happen at any point between parsing and processing the data. If you delete the file without processing all the data contained within it, the data will be lost. On the other hand, if you do not delete the file but have already processed some of the data, you will end up processing the same data twice. This poses a data integrity concern that needs to be addressed. Both discarding data and processing data twice are undesirable and potentially unacceptable. That is why it is necessary to process a file in its entirety before deleting it.

This can be challenging, especially if the application involves multiple processing steps, including several database writes. Errors occurring in any of these phases can result in the entire batch being reversed. Additionally, processing a batch of data within a single transaction restricts the data processing to only one thread. File reading and data processing should be performed in separate threads to optimize performance.

One possible solution is to relax the data integrity requirement by implementing a queue where all the data is stored after parsing. This approach considers a file processed once all its records are inserted into the queue. Records can then be dequeued one by one, preventing the entire batch from being reverted. The issue with this approach is that the data in the queue will be lost in the event of an abrupt application failure. Since abrupt application failures are expected to be rare, this trade-off may be acceptable. If preserving data is not acceptable, a persistent queue for the data can be introduced (but we opted out of using new components in favor of using simple CSV reading).

It is also important to set a reasonable limit to the number of records in a file. A large file might present a huge load of data at once. So huge that it might crash the data if it does not have the memory to read the whole file.

Data Sharing

Once everything is set up and all the monitoring points are implemented (excluded from this article), we can enjoy our manually implemented file-based data exchange solution… until a new consumer is needed for the same data. 

Fortunately, we already have this functionality (file download+reservation, data retention, data queuing, monitoring, etc.) supported, and we can easily re-use it in any new consumer. Hopefully, the code has been written well enough to be copied and pasted into a library. Otherwise, there will be significant code duplication. Let’s not forget to create documentation for the module so that the solution can be integrated effortlessly.

Once we move this code to a module, our file-based data exchange solution will become a reusable component that any application can utilize (provided it is written in the same programming language). The only thing left is instructing the producer to export the files to a new location. There might be some duplicated data between the two locations, but let’s hope we will not add many of these consumers to the system.

Data storage already handles this – and it does it better

These are only some of the issues that we encountered. There are more of them, some not even mentioned here, and others are yet to be discovered. To resolve them, we had to invest time in development, refactoring, and as a result, we now have a code base to maintain. 

But these problems were not unique to us. They were common problems that most data storage software handles out-of-the-box and does it better!

The selection of software depends on the nature of the data. In our case, the data were events, and the producer application performed some batch processing on it. A more suitable storage option than the one we have been using would be a messaging system:

· Producers and consumers would write and read from a topic or a queue.

· Data would be stored and replicated to another machine, also enabling Geo-redundancy.

· Retention would be managed by the platform, eliminating the need for manual scripts.

· Messaging systems have a commit feature to indicate that a record (or batch) has been processed.

· Some messaging systems support consumer groups, allowing concurrent processing out-of-the-box.

· Different consumers (applications) can be subscribed to the same topic.

· Applications can be written in the most popular programming languages.

· Monitoring tools for popular systems are already available.

· Popular systems are usually well-documented.

For other types of data, the solution might be a more suitable data store like Redis or Postgres. It all depends on what the data is about and how it can be presented.

Exchanging data via files should be considered an anti-pattern and only be used when other approaches cannot be implemented (like when integrating with a legacy system). Each time some data is exchanged using files, a red flag should pop up, and you should ask yourself – what is a better way to do this? There probably is.

The post From file frustration to streamlined data exchange: Rethinking the approach appeared first on ShiftMag.