We built a Python framework bridging the gap between AI and streaming platforms

We are a small AI startup, and we’ve learned that it’s (relatively) easy to do pilot projects when you are small, as you can get away with your solutions not being on an enterprise level. But once you have to go to production, it’s a whole new ballgame. That’s how we got to create FastStream.

It all started with our work on pilot projects with several large enterprise clients. We quickly realized that the key to a successful transition to production was seamless integration between our predictive models and existing streaming platforms.

However, we faced a challenge.

Java is the dominant language in the streaming ecosystem, while Python is the king of AI. This made it difficult to bridge the gap between the two worlds.

We looked for an existing tool but couldn’t find anything that fit our needs besides Faust, which is deprecated and not something we can rely on, not when working with large companies.

That’s when we decided to create FastStream. We wanted to develop a framework that would make it easy to develop scalable and robust streaming microservices in Python, with the added benefit of automated code generation and predictive models.

From FastKafka to FastStream

We started by calling it FastKafka, thinking we’d only use it internally and just for Kafka, but then decided to release it as an open-source, purely as a tool that might help someone else. Well, soon, we realized that it needed to be more than just a Kafka wrapper. 

The response of the community took us by surprise. We quickly crossed 500 stars on GitHub, but more importantly, we received valuable input in terms of feature requests, suggestions, and feedback from seasoned enterprise developers.

One of our key collaborators was Nikita Pastukhov, the author of the Propan framework. Nikita contributed his expertise to merge the best features of Propan and FastKafka into a single, unified framework — FastStream.

This collaborative approach helped us to create a robust and fully extendable API that supports not only Kafka but also other key streaming protocols.

Generate an application from a short textual description

FastStream is not just about simplifying the codebase. Its architecture supports automatic code generation because we designed it with speed, scalability, and maintainability in mind. Namely, code generation is, in terms of time, a small part of the development; much more time is spent on the specification, creation of documentation, and testing. 

In our case, we have automated the whole process, whose result is a specified, documented, and tested microservice. In other words, our first prototype can generate an entire application — complete with unit and integration tests — from just a short textual description of the requirements but at the level of quality expected in large companies.

It is absolutely mindblowing that a task that would normally take several hours, sometimes even days, for a single developer now takes about one minute.

All this also enables seamless integration of our predictive models and future-proof microservices by allowing real-time data predictions.