Working on it didn’t make me hate legacy projects any less; it did help me get a deeper understanding of the processes and practices we use today. 

I am proud that I am a part of a team that uses most of the best practices: 

• Writing clean code and automated tests 
• Participating in pull requests and tasks reviews 
• The app is in production within the same day of merging into the master branch 
• Agile is well-adopted 

 It’s not perfect. Pull requests sometimes contain trivial suggestions and discussions. The ops team usually messes something up (or at least that’s what we devs like to think). Our product owner occasionally wants us to fast-track some ‘easy’ features… again. But all in all, things are pretty good. 

A trip to the Ant museum 

Since our team was performing well, our velocity was borrowed to another product, managed by another part of our company. We weren’t thrilled about it since the project uses an older version of Java, and the code wasn’t written the way we would do it.  

The job was to add a few metrics – simple ones, like whether the app is running and for how long, is it processing data (fast enough). The project itself was in maintenance mode, and no new features had been added in a while. Since we are accustomed to adding metrics, this should have been a piece of cake. 

Rolling up our sleeves, we first noticed that the project had been built using something very old - Ant  build files. They are big XML files that contain every piece of information on how to build the project in question. For example, you need to specify that the project needs to be compiled, tested, and packaged. All details like source, target, and resource location must be explicitly configured within those files. This used to be pretty common for many programming languages. You wrote a build file once, copied it on each new project, and then changed it until it worked for the new project.  

Here’s what an Ant build file for a Hello World project would look like:

There must be a more convenient way to manage that, right? Well, that’s why the idea of convention over configuration came to life, suggesting that developers only need to specify unconventional aspects of the application. Modern build tools have adopted this paradigm by providing developers with defaults that can be overridden if needed. That’s why most java source files are in src/main/java source, while the compiled classes are in the target folder—no need to specify this over and over. 

That got us thinking about whether this approach would be helpful in our current projects. We had a huge application properties file, with most values having the same value (e.g., application port). Could we apply the same principle to our application properties? Most properties could have defaults, and our properties file wouldn’t be as large. 

Things we take for granted

Back to our legacy project – we managed to build and package our application! The tedious part was behind us, and we could start coding. But how do we embed our metrics component into the legacy code base? We took this for granted because our application framework usually handled it.  

Now, what would be the best way to inject our metric components into various parts of the legacy code? Singletons! Well, it seemed the easiest way, at least, but the community considers them anti-patterns. Why? Isn’t our beloved XY framework relying on singletons? If not, what does it use? What is dependency injection? How does it function under the hood? 

These questions got us thinking about basic concepts that we took for granted. Using singletons wasn’t exactly a bad idea in this case since most of the code wasn’t covered with unit tests. Nevertheless, we couldn’t have a good night’s sleep unless the code was immaculate. We tried a different approach and ended up with decent and simple code – no singletons, no new abstractions. 

The limited role of developers 

The only thing left was to deploy it so that we could test it. This was a problem, of course—in this case, we were not the ones deploying or testing it. Deploying had to be done by the ops team, and testing by the test team. Why couldn’t we, as developers, manage the whole feature from start to production instead of opening tickets and waiting for other teams before we could close our task?

Firstly, we couldn’t avoid manual testing since much of the code wasn’t covered with tests. We also couldn’t deploy the application ourselves since the infrastructure didn’t allow us to.  

This had us thinking about the reasons behind the separation of duties and how our current approach is better. Seeing that our lead and cycle time on this project was significantly higher than usual (it took us weeks to deliver something we usually deliver in several days), the evidence was strongly in favor of that.

Learn outdated practices to understand current ones

At the end of the day month, our metrics were up and running in production. My feelings about legacy projects haven’t changed – I still hate them, and I am not expecting you to hate them any less.  

We couldn’t change the project we were assigned to, and these were the cards that we were dealt. What we could change is our attitude towards the legacy project. Instead of feeling resigned, we saw it as a place to ask questions and learn.  

It taught us how things were done before and why they are done differently now. Instead of just knowing the best practices, we gained first-hand experience with the history behind them.  

Once you start having this deep-rooted knowledge, other developers will recognize this and trust your knowledge and expertise. If you want to be that person, you better be ready to dig into some legacy projects. 

The post You’re not a senior engineer until you’ve worked on a legacy project appeared first on ShiftMag.

My biggest dilemma when I started to program was this: How do I write things correctly?  

Should a controller directly access the repository or use a service method as an intermediary? Where should we validate objects – within the Controller or the Service layer? Should repositories be placed in a repository package or a feature package?

There are endless ways to write code, but there must be a right way. To settle the ongoing debates, I turned to established best practices. Best practices provide limitations to the endless landscape of programming options. It is a lot easier to answer the afore-posed questions when an authority suggests that:

• Controllers should typically avoid direct access to repositories, with a few exceptions.
• Object validation is often best placed within a Controller.
• Feature packages tend to be more practical than technical packages.

Embracing these best practices both streamlined my decision-making process and helped resolve disagreements with fellow developers. Why engage in heated debates about the superiority of feature packages when you can simply reference authoritative articles? When those references come from a reputable source like Wikipedia, the argument is immediately over, right? 

The problem arises when people take these practices for granted, and they turn into dogmas that never get questioned.

The problem with dogmas

Best practices in software development are not static and can evolve. 

Consider the evolution of database access patterns. When starting my development career, the best practice for accessing a repository (at least in Java) was through DAO objects, which is now considered outdated in favor of repositories. If we look at other data source architectural patterns, we could add many more of these patterns to the equation. How do we know which one of these best practices is the best? The key lies in developing a deep understanding of these practices.

Without comprehending why a particular practice is considered good, you risk making bad decisions. Such decisions can lead you to “hype-driven development,” where you adopt the latest trends without fully understanding their implications. This can result in complex, service-oriented, NoSQL, Big Data monsters that nobody knows how to maintain anymore.

Questioning everything, even the most widely accepted authorities, is a valuable approach. Instead of blindly following trends, consider the context and the trade-offs of each practice.

You need an Igor in your life

I had a colleague (let’s call him Igor), a skeptic by nature.

He was unimpressed by dogmas, both in his professional and private life. He was an old-school developer who clung to his tried-and-true methods, largely unfazed by new ideas and trends in our field. 

Convincing Igor to accept a new coding practice was not an easy thing to do. It would even get terribly frustrating if you didn’t fully understand the practice yourself. Suppose Igor is dissatisfied with grouping packages by feature rather than by technology. Here is how a conversation with Igor might unfold when we don’t understand why we do things the way we do them:

• I don’t know why repositories are suddenly placed in packages named by entities.
• Well, putting them in the repository package is not a good practice. It doesn’t foster Domain Driven Design.
• What is this Domain Driven Design?
• Well, it states that you should organize your code by domains, not technology. That domain thinking should drive how you design things.
• And what’s wrong with the way we’ve been coding so far?
• Well, the domain knowledge is not represented in the code.
• You want to say that you understand the domain behind this software we are writing.
• Not right now, but…
• The analytics department just hands us use cases, and we code what we are given. Nobody asks us anything.
• So you think you know better than all these people that have written this and have been adopting this?!
• I don’t care what some fool has written. Today, they promote this as the best thing, tomorrow, it will be something else.

There is no way to win the argument this way, really. It immediately steers off to terminologies rather than answering the question and uses authorities instead of providing explanations. There is no real understanding behind these terminologies.

What if we approached the same argument differently?

• I don’t know why repositories are suddenly placed in packages named by entities.
• Why would you rather place them in a repository package?
• Because you do not need to think much about where to put a class, it’s straightforward.
• Let me ask you a question: do you spend more time reading or writing code?
• Definitely reading, trying to understand what the heck is going on in there.
• How many classes do you need to understand a certain use case?
• Controller, service, repository, a couple of classes.
• When using your IDE, would you rather have these classes close to each other or scattered across the project?
• Close, I guess.
• This is what ‘Package by Feature’ is all about. The idea is that packages should contain related concepts that you would usually look at simultaneously. I’m pretty sure that when reading AccountService, the next class you are going to look at is AccountRepository and not CountryService, right?
• Yeah, but it’s harder to think now which class goes where
• Yeah, I agree it is harder, but remember you will read your code more than write it.

My encounters with Igor made me question, dissect, and refine my understanding of best practices. Each new concept had to withstand scrutiny and provide rock-solid answers to a stream of whys. Was it genuinely a best practice? Were there situations where it didn’t apply? This rigorous approach ultimately led me to a point where the questions became less necessary, and the knowledge became more ingrained.

There is only one real dogma

Money. 

It serves as the ultimate driver behind our decisions and actions. To understand and deconstruct any dogma, you can employ a series of questions until we arrive at an answer that inevitably involves creating more value (money).

Why should we write clean code – and what happens if we don’t?

• Bad code is harder to understand + harder to maintain = requires more money to change.
• Applications that are hard to change eventually get dropped in favor of new ones (more money).

Why should we write tests? What happens if we don’t?

• We need to employ testers (which costs money) who need not only to test the new feature but also to test against regression.
• Regression can be introduced that not even the tester can catch (which loses us money).
• The feedback cycle between devs and the quality of the solution is increased. The developer needs to wait for the testers’ approval (or disapproval) before merging the code. The feature is released later than sooner (losing more money).
• Non-functional code changes are discouraged since they require extensive regression testing (and, therefore, cost more in man-hours). This means the software will soon become ugly, inheriting the same drawbacks as not writing clean code from above.

Engaging in these mental exercises can greatly assist in organizing and solidifying your knowledge. Once mastered, these conclusions can be used as corollaries for more complex questions.

Why is the microservices architectural style popular?

Why are applications favoring embedded Tomcat over traditional Tomcat?

Why should product managers drive epics instead of developers? 

Context matters

In addressing complex problems, a thorough exploration often requires delving into the historical context of best practices and considering various aspects that influenced their inception.

• What specific problems did engineers face at that time?
• Were resources readily available or constrained?
• What tools were available at that time?
• How were organizations structured?
• Did developers take on testing and deployment?
• Did they write SQL code?
• Did they possess a deep understanding of the domain?

Take, for instance, the common practice of prefixing interface class names with the letter ‘I.’ Some consider this a best practice, but why?

It comes from a time when IDE-s were not that sophisticated, where developers needed to put prefixes in names to differentiate a regular class from an interface. There were also no ‘this’ pointers, so class members were prefixed with ‘m’, while method parameters were prefixed with ‘p’.

Good books will usually cover these topics, but it’s not something that an engineer will remember during their first reading. That’s why these practices need to be revisited until they click. You know it clicked if you could explain them to Igor.

There is no one right way

Do note that both you and Igor might be right at the same time!

In the conversation above, Igor stated that it’s harder to know where each class goes when packaging classes by feature. It requires people to define the boundaries of a feature.

What if the feature undergoes a significant transformation, as it often happens? Adapting the packages becomes necessary, implying an evolutionary architecture. Do your team members possess the skills for such adaptability? You might not even be on the same project to enforce these practices in a month.

Beyond understanding why something is good, researching why something is bad can be equally enlightening. A simple Google search for ‘why microservices suck’ reveals how even established authorities don’t consider microservices a silver bullet. For example, StackOverflow, a successful platform, operates as a monolithic application on on-premise servers, defying the cloud-microservices paradigm. The point is not that microservices are inherently flawed but that deeper comprehension will always outperform blind adherence to trends.

Building a deeper understanding can take various forms, but the core message I leave you with is simple. Never take things for granted and always ask:

The post Are your engineering “best practices” just developer dogmas? appeared first on ShiftMag.

Occasionally, everyone needs to handle data stored in files. CSV files seem like the way to go and are often the first choice – but I’d rather call them an anti-pattern.

Let’s say a customer requests an Excel report containing specific data. If this is a one-time thing, we can conveniently send it via email. In most cases, however, the data needs to be sent regularly. Not all clients use sophisticated software like Kafka; even when they do, we don’t always have access to it. So we need a solution that is both universally available and easily understood. SFTP and CSV files seemingly fit the bill.

This solution seems optimal for sending data to an external client. The problem arises when an internal application needs to integrate with ours, requires some data, and we decide to solve it by exporting CSVs. At first glance, this seems a low-hanging fruit.

Everyone can download a file.

Everyone can read a CSV.

It’s already developed.

While it may appear that simple, ultimately, it proves to be more costly than using a dedicated component for data exchange. I learned my lesson the hard way. I’ve worked on both applications that sent and received data via files. Here are some of the issues I discovered along the way.

 Data storage

The first thing we need is a place to store the data. Since our data is stored in files, it must be placed within a file system. The decision we face is whether to store the files on the producer machine, consumer machine, or an intermediary machine. Each option has its advantages and disadvantages.

Using the producer machine results in the consumer bearing the entire networking load. The consumer will need to periodically retrieve the data through a PULL operation. In the event of a network link failure, the producer will remain unaffected. 

However, this solution has a drawback – when consumers are redundant, and there are multiple instances, each instance may attempt to download the same file. Consequently, the same data will be processed multiple times. To address this, each instance should reserve the file before initiating the download. This can be accomplished by appending a suffix to the file name or moving the file to another folder. Once the file is reserved, the consumer application can proceed with the file download.

Using the consumer machine puts all the networking burden on the producer. The producer needs to PUSH the data to the consumer machine once the file is created. The consumer machine can watch the local file system for changes and react when a new file is created. As a result, the push-based approach can yield a faster response by the consumer and does not put any networking burden on the consumer. Since the file is sent directly to the consumer instance, this approach does not need file reservation. 

Pushing files to consumers also has a considerable downside – the producer must balance these files between the consumer instances. Implementing effective client-side balancing is a tough nut to crack. The producer would need to adapt its balancing strategy when a consumer instance gets offline or online and is added to or removed from the group.

Using an intermediary machine combines elements of both previously mentioned approaches. It eliminates the need for client-side balancing, but the consumer still needs to reserve the file before downloading it. The data storage is independent of both the consumer and producer, although both parties are still required to transfer files over the network.

Data retention

Regardless of the chosen method for storing data, it is crucial to ensure that the disk is not filled with files. Failing to do so will render the machines (and the services running on them) unusable. The primary line of defense is to store files on a separate section of the disk. 

Even if the partition gets filled with files, it will not impact other services. This advice applies even when files are stored on an intermediary machine, as some operating system functionalities may not work properly if the disk space is occupied.

Accumulating files can happen quite easily. For instance, the consumer application may be down (or not deployed) while the producer application continues to generate files. It can also occur if the producer generates a high volume of data the consumer cannot keep up with. 

In such cases, it is necessary to prevent the files from overflowing the disk. If the data represents events and skipping a few events in the event of a failure is acceptable, a retention policy should be implemented. 

For example, a script that periodically deletes the oldest files can be developed. This script adds another component to the architecture that needs to be created, maintained, and monitored. If the script malfunctions or contains a bug, the data retention may not work as intended.

Data processing

A file is removed from the disk when a consumer has read and processed the whole file content. The ‘whole file content’ is emphasized since a file represents a batch of data. Errors or failures can happen at any point between parsing and processing the data. If you delete the file without processing all the data contained within it, the data will be lost. On the other hand, if you do not delete the file but have already processed some of the data, you will end up processing the same data twice. This poses a data integrity concern that needs to be addressed. Both discarding data and processing data twice are undesirable and potentially unacceptable. That is why it is necessary to process a file in its entirety before deleting it.

This can be challenging, especially if the application involves multiple processing steps, including several database writes. Errors occurring in any of these phases can result in the entire batch being reversed. Additionally, processing a batch of data within a single transaction restricts the data processing to only one thread. File reading and data processing should be performed in separate threads to optimize performance.

One possible solution is to relax the data integrity requirement by implementing a queue where all the data is stored after parsing. This approach considers a file processed once all its records are inserted into the queue. Records can then be dequeued one by one, preventing the entire batch from being reverted. The issue with this approach is that the data in the queue will be lost in the event of an abrupt application failure. Since abrupt application failures are expected to be rare, this trade-off may be acceptable. If preserving data is not acceptable, a persistent queue for the data can be introduced (but we opted out of using new components in favor of using simple CSV reading).

It is also important to set a reasonable limit to the number of records in a file. A large file might present a huge load of data at once. So huge that it might crash the data if it does not have the memory to read the whole file.

Data Sharing

Once everything is set up and all the monitoring points are implemented (excluded from this article), we can enjoy our manually implemented file-based data exchange solution… until a new consumer is needed for the same data. 

Fortunately, we already have this functionality (file download+reservation, data retention, data queuing, monitoring, etc.) supported, and we can easily re-use it in any new consumer. Hopefully, the code has been written well enough to be copied and pasted into a library. Otherwise, there will be significant code duplication. Let’s not forget to create documentation for the module so that the solution can be integrated effortlessly.

Once we move this code to a module, our file-based data exchange solution will become a reusable component that any application can utilize (provided it is written in the same programming language). The only thing left is instructing the producer to export the files to a new location. There might be some duplicated data between the two locations, but let’s hope we will not add many of these consumers to the system.

Data storage already handles this – and it does it better

These are only some of the issues that we encountered. There are more of them, some not even mentioned here, and others are yet to be discovered. To resolve them, we had to invest time in development, refactoring, and as a result, we now have a code base to maintain. 

But these problems were not unique to us. They were common problems that most data storage software handles out-of-the-box and does it better!

The selection of software depends on the nature of the data. In our case, the data were events, and the producer application performed some batch processing on it. A more suitable storage option than the one we have been using would be a messaging system:

· Producers and consumers would write and read from a topic or a queue.

· Data would be stored and replicated to another machine, also enabling Geo-redundancy.

· Retention would be managed by the platform, eliminating the need for manual scripts.

· Messaging systems have a commit feature to indicate that a record (or batch) has been processed.

· Some messaging systems support consumer groups, allowing concurrent processing out-of-the-box.

· Different consumers (applications) can be subscribed to the same topic.

· Applications can be written in the most popular programming languages.

· Monitoring tools for popular systems are already available.

· Popular systems are usually well-documented.

For other types of data, the solution might be a more suitable data store like Redis or Postgres. It all depends on what the data is about and how it can be presented.

Exchanging data via files should be considered an anti-pattern and only be used when other approaches cannot be implemented (like when integrating with a legacy system). Each time some data is exchanged using files, a red flag should pop up, and you should ask yourself – what is a better way to do this? There probably is.

The post From file frustration to streamlined data exchange: Rethinking the approach appeared first on ShiftMag.