These rules are not new, and they are not mine; I stole them from Andrew Ng and Benedict Evans, two men with a huge following.

AI’s Law of diminishing returns

To paraphrase Andrew’s words from Coursera’s Deep Learning Specialization course:

The effort to half an AI system’s error rate is similar, regardless of the starting error rate. 

This is not very intuitive. If an AI system passes 90% of test cases and errors on 10%, then you are 90% done, right? Fix the remaining 10% of errors, and you will have 100% accuracy? Absolutely not.

If it took you six months to halve the error rate from 20% to 10%, it will take you approximately another six months to halve 10% to 5%. And another six months to halve 5% to 2.5%. Ad infinitum. You will never achieve a 0% error rate on a real-world AI system. For an illustrative example, see this typical chart of error rate vs the number of training samples:

Credits: Zeljko Svedic

Notice that later in the training process, the training set size increases exponentially with each error rate halving, and the error rate never reaches zero. Sure, you will get more efficient with acquiring training data (e.g., by using low-quality sources or synthetic data). Still, it is hard to believe that acquiring 10X more data is going to be much easier than acquiring the initial set. 

This rule becomes more intuitive when dissecting what an AI system error rate represents: uncovered real-world special cases. There are an infinite number of them. For example, one of the easiest machine learning (ML) tasks is classifying images of dogs and cats. It is an introductory task with online tutorials that get 99% accuracy. But solving the last 1% is incredibly hard. For example, is the creature in this image a dog or a cat?

It is Atchoum, the cat, who rose to fame because half of the humans recognized him as a dog. The human accuracy on dog/cat classification within 30 seconds is 99.6%. A dog/cat classifier with less than a 0.4% error rate would be superhuman. But it is possible. A training set with hundreds of thousands of strange-looking dogs and cats would teach a neural network to focus just on details encoded in dog or cat chromosomes (e.g. cat eyes).

However, building such a dataset is orders of magnitude more complex than a tutorial with 99% accuracy. Other problems lurk in that 1% error rate: photos that are too dark, photos in low resolution, photo compression artifacts, photo post-processing by modern smartphones (adding of non-existing details), dogs and cats with medical conditions, etc. The problem space is infinite. This is still considered a solved ML problem, though, because a 1% error rate is low enough for all practical purposes. 

But for some problems, even a 0.01% error rate is not satisfactory, for example: full-self driving (FSD). Elon Musk said in a 2015 article with Forbes:

We’re going to end up with complete autonomy, and I think we will have complete autonomy in approximately two years.

Tesla was so confident in that prediction that they started selling a full self-driving add-on package in 2016, and they weren’t the only ones. Kyle Vogt, CEO of Cruise, wrote a piece called How we built the first real self-driving car (really) in 2017, in which he claimed:

“…the most critical requirement for deployment at scale is actually the ability to manufacture the cars that run that software”

So, the software and the working prototype are done; they just need to mass-produce “100,000 vehicles per year.” 

Fast forward to 2024. Elon Musk’s predictions for autonomous Tesla vehicles deserved a lengthy Wikipedia table, mostly in red: 

Credits: Wikipedia

What about Kyle Vogt? In October of 2023, Cruise’s car dragged a pedestrian for 20 feet, after which California’s DMV suspended Cruise’s self-driving taxi license. Kyle “resigned” as CEO in November 2023.  

Don’t misunderstand me—I believe autonomous cars will have a significant market share, probably in the next decade. The failed predictions above illustrate what happens when entrepreneurs don’t respect the AI law of diminishing returns.

Elon and Kyle probably saw a demo of a full-self-driving car that could drive on its own on a sunny day on a marked road. Sure, a safety driver needed to intervene sometimes, but that was only 1% of the drive time. It is easy to conclude that “autonomous driving is a solved problem,” as Elon said in 2016.

Notice how ML scientists and engineers didn’t make such bombastic claims. They were aware of many edge cases, some of which are described in crash reports. Edge cases include:

• A pedestrian crossing a two-lane avenue with a bicycle at night and without lights (2018 Uber crash).
• The lane’s white lines diverging before a barrier (2018, Tesla Model X crash).
• A white trailer truck against the white sky (2019 crash, Tesla Model 3 decapitated the driver and continued driving for 40 seconds).
• A vehicle kicking a pedestrian under an FSD car (2023 Cruise incident mentioned above).

Why so many companies promised a drastic reduction in self-driving error rates in such a short time without having a completely new ML architecture is an open question. Scaling laws for convolutional neural networks have been known for some time, and the new transformer architecture obeys a similar scaling law. 

AI’s Product vs Feature rule

When is an AI system a good stand-alone product, and when is it just a feature? In the words of Benedict Evans from The AI Summer podcast: “Is this feature or a product? Well, if you can’t guarantee it is right, it’s a feature. It needs to be wrapped in something that manages or controls expectations.” I love that statement. The “it is right” part can be broken down using the error rate:

If your AI system has a higher error rate than target users, you have an AI feature in an existing workflow, not a stand-alone AI product.

This rule is more intuitive than the law of diminishing returns. If target users are better at a task, they will not like stand-alone AI system results. They could still use AI to save them effort and time, but they will want to review and edit AI output. If AI completely fails at a task, humans will use the old workflow and the old software to finish the task.

Let’s take MidJourney, for example, which generates whole images based on a text prompt. When I used it for a hobby project last year, satisfying artistic images appeared instantly, like magic. But then I spent hours fixing those creepy hands:

Credits: CC0 via neural.love  

Each time MidJourney created a new image, one of the hands had strange artifacts. Finally, it generated an image with two normal hands—but then it destroyed the ears. The problem was less with wrong details and more with bad UI, which didn’t allow correction of the AI’s mistakes.

Adobe’s approach is different—it treats generative AI as just one feature in its product suite. You use an existing tool, select an area, and then do a generative fill:

Credits: Adobe press release

You can use it for the smallest of tasks, like removing cigarette butts from grass in a wedding photoshoot. If you dislike AI grass, no problem—revert to the old designer joy of manually cloning grass. Also, Adobe Illustrator has generative Vector AI that generates vector shapes you can edit to your liking.

MidJourney makes more impressive demos, but Adobe’s approach is more useful to professional designers. That doesn’t mean MidJourney doesn’t make sense as a product; its target users are the ones who don’t care about details. For example, last Christmas, I got the following greetings image over WhatsApp:

Credits: Zeljko Svedic

Did you notice baby Jesus’ hands and eyes? Take another look:

Credits: Zeljko Svedic

That would never pass with a designer, but that is not the point. There is a whole army of users who don’t care about image composition and details; they just want images that go with their content. In other words, MidJourney is not a replacement for Adobe’s Creative Suite—it is a replacement for stock photo libraries like Shutterstock and Getty Images. And judging by the recent popularity of AI-generated images on social media and the web, people like artsy MidJourney images more than stock photos.

Low-hanging fruit in stand-alone AI products are use cases where a high error rate doesn’t matter or is still better than the human error rate. An unfortunate example is guided missiles; in the Gulf War, the accuracy of Tomahawk missiles was less than 60%. But the army was happy to buy Tomahawks because they were still much more accurate than older alternatives, as fewer than 1 in 14 unguided bombs hit their targets.

Evaluating startups based on the above rules

The great thing is that error rates are measurable, so the above rules give a framework to judge an AI startup quickly. Below is a simple startup example.

Devin AI made quite a splash in March of 2024 with a video demo of developer AI that can create fully working software projects. From the announcement Devin was “evaluated on the SWE-Bench” (relevant benchmark), and “correctly resolves 13.86% of the issues unassisted, far exceeding the previous state-of-the-art model performance of 1.96% unassisted.” So, the current state-of-the-art (SOTA) has a 98% error rate, and they claim to have an 86% error rate. Even if that claim is valid (it wasn’t independently verified), why do their promo videos show success after success? It turns out that the video examples were cherry-picked, the task description was changed, and Devin took hours to complete.

In my opinion, Microsoft took the right approach with GitHub Copilot. Although LLMs work surprisingly well for coding, they still make a ton of mistakes and don’t make sense as a stand-alone product. Copilot is a feature integrated into popular IDEs that pops up with suggestions when they are likely to help. You can review, edit, or improve on each suggestion.  

Again, don’t get me wrong. I think coding SOTA will drastically improve over the next few years, and one day, AI will be able to solve 80% of GitHub issues. Devin AI is still far away from that day, although the company has a valuation of $2 billion in 2024.

More formally, the framework for evaluation is:

1. Find a relevant benchmark for a specific AI use case. 
2. Find the current state-of-the-art (SOTA) error rate and human error rate on that benchmark.
3. Is the SOTA better or comparable to the human error rate?
   1. If yes (unlikely): Great, the problem is solved, and you can create a stand-alone AI product by reproducing SOTA results.
   2. If no (likely): Check if there is a niche customer segment that is more tolerant of errors. If yes, you can still have a niche stand-alone product. If you can’t find such a niche, go to the next step.
4. You can’t release a stand-alone AI product. Wait for SOTA to get better, pour money into research, or go to the next step.
5. Think about how to integrate AI as a feature into the existing product. Make it easy for users to detect and correct AI’s mistakes. Then, measure AI’s return on investment:
   
   AI_ROI = Effort_saved_by_accurate_AI_responses / Effort_lost_on_checking_and_modifying_AI_responses
   
   If too much user time is spent checking and correcting AI errors (AI_ROI <= 1), you don’t even have a feature.
   

Or, to summarize everything discussed here in one sentence:

Credits: Zeljko Svedic

Every innovative AI use case will eventually become a feature or a product once the error rates allow it. If you want to make it happen faster, become a researcher.

OpenAI’s early employees spent seven years on AI research before overnight success with ChatGPT. Ilya Sutskever, OpenAI’s chief scientist, still didn’t want to release ChatGPT 3.5 because he was afraid it hallucinated too much. Science takes time.

If you found this article useful, please share.

The post Two rules of AI business and startups that ignore them appeared first on ShiftMag.

Two years of Covid-19 did more for remote work than decades of business conferences and all of Dilbert comics. Both lovers and haters of remote work were forced to close their offices and work from their pajamas. But, as lockdowns ended, haters were quick to demand going back. Offices are “more productive” and “build the team spirit,” or at least those are the claims. The same experience—working from home—was perceived quite differently by the two camps. 

And then, there is a third camp—the ones who didn’t notice the lockdown in their workday. The ones without a strong opinion because they accepted remote work, with its pros and cons, long ago. Current online debates make them smile and remember the younger days when they started their remote journey. 

Our pre-employment testing company, TestDome, is firmly in the third camp. We went from a startup of 4 to a company of 20 people, all working remotely. This year we met each other for the first time to celebrate 10 years of doing business. Before I share funny anecdotes from that meeting, let me explain how we got here and what we learned about remote work.

Our remote experience

We started remote work almost accidentally. The founding team was all from Croatia, but one developer wanted to work from a Croatian island, another from his home city, and I wanted to be flexible to travel around the world. So, in 2013 we decided to use Assembla for tickets and code hosting, Google Docs for writing documents, and email and Skype for communication. But the tools are less important than the mindset. We learned that: 

Remote work requires asynchronous communication, in contrast to the office, where colleagues being a few steps away facilitates synchronous communication.

Many companies use Slack or Discord chat rooms to simulate a feeling of constant online “togetherness.” But that never worked for us. You never know if the other party on chat will answer your question in 30 seconds or 30 minutes. We tried many tools, but:

We never found chat software that accurately shows if the other person is at their computer, away, or in “do not disturb” mode because of an ongoing call.

If you count on people to manually change their status every time they go to the toilet, good luck. And requiring people to be constantly attentive to every message in Slack, be it important or funny, is a mass killer of company productivity. Instead, we adopted one style of communication that works for chat, email, and ticket mentions:

Provide full context when asking or requesting something, and get back to that task once the other person answers. 

For example, when a coworker asked about his post idea over chat, he provided full context:

Written communication is the king of remote work. You need to write clearly and concisely in tickets, wikis, documents, emails, etc. Everything is written down because:

The mortal sin of remote work is blocking other colleagues. The easiest way to block others is not to write something important, and then they need to wait until you respond in your time zone.

That explains the disagreement at the beginning of the article. If you are a structured person who likes to work in the zone, without distractions, and then write things down, then you will probably like remote work. If, on the other hand, you like impromptu conversations, fluent structure, being able to walk to a colleague and ask them a question, or dislike writing—then you will probably hate remote work. In general, it seems that:

Remote work is more suited to introverts, while office work is more suited to extroverts.

With all that wisdom, we were able to grow from 4 people in the beginning to 20 people today. We have remote employees in Brazil, Costa Rica, Morocco, Portugal, Finland, Croatia, Poland, Azerbaijan, India, China, and Australia. We found most of them using a lesser-known job board called We Work Remotely. Only remote jobs are allowed there.

FUN FACT: We Work Remotely started as a side project of 37 signals and featured in their popular book REMOTE: Office not required. I highly recommend this book; it explains many of the ideas mentioned here.

The location of either companies or candidates is irrelevant on We Work Remotely, which led to one funny situation. A candidate called Bruno decided to try remote work. He had a bad experience working for local Croatian companies and decided to work for a foreign company. He applied via We Work Remotely and solved the pre-interview screening test on TestDome. When it was time for the interview, he was surprised that his interviewer spoke Croatian, not English. But he thought, “It is a big international company; they assigned someone from the region to interview me.” Only in the end, he realized that TestDome founders live in the same country he tried to avoid—Croatia!

Employees aside, there is another essential ingredient:

To be a remote-only company, you need to have remote-only customers.

If your customers demand physical meetings or want to come to your office, no remote book will help. Fortunately, TestDome is a prepaid SaaS solution, and most new customers come via SEO (Search Engine Optimization). Typically, our customer is a CTO or a project lead responsible for technical interviews. They get too many unqualified candidates; today, everybody claims to be a skilled programmer. When they google for e.g., “angular coding test,” TestDome’s free test shows up on top:

If they like our free questions, they can sign-up and make a purchase with a credit card; everything is self-service. We have large customers we have never talked with or exchanged emails with, and honestly, they probably don’t want to get any emails or calls from us. 

In other words, it took an effort to have remote-only customers. We give 30% of our questions for free to have attractive landing pages and demonstrate value. We give developer certificates for free. Onboarding, global CDN, and responsive pages are all there for a smooth customer experience. If your product requires on-site support, remote is hard.  

It is not all milk and honey in the land of remote work. The most significant drawbacks we noticed are:

• The lack of team spirit.
• General loneliness you can feel if you don’t leave your house for a day.
• Sometimes you just want to make small talk.

We tried to address that with the following:

• Daily team standup calls—There are separate calls for content and product teams, and they are in the middle of the workday to be convenient for all time zones.
• Paid coworking membership—Choose any local coworking location you like, and we will pay for it. I worked from coworking places in Oxford, Berlin, Thailand, and Croatia.
• Virtual team buildings—2-3 times a year, we have a virtual team building where the entire company plays silly team games.

Daily calls show what others in the team are working on. A coworking place surrounds you with people in a similar situation where you can hang out for coffee or lunch. Games on virtual team building are needed for people to relax and stop talking about business. Classics we played over the years are:

• Two truths and a lie—each employee tells three personal things, and others need to guess which one is a lie. As it is often in business, the best liar wins.    
• Geoguessr—pin the exact location on a world map based on just the Google Street view. Quizmaster usually takes one location from each city our employees live in, so everybody has a fair chance, and we get to see what our home cities look like.
• Company silly events quiz—e.g., who did a code commit at 3AM, who replied to a customer with an unchanged template email, who crashed the server on Friday afternoon. Use events that happened some time ago; otherwise, not everybody will laugh.

Still, building team spirit and getting to know your colleagues is a challenge for a remote company. That leads us to the next topic, our first in-person team building.

First in-person meeting

The first question was, where in the world to meet? Since our employees live from Brazil to Australia, going near a big international airport made sense. Rome won over Istanbul and Barcelona, both because of better flight connections (all flights lead to Rome) and because everyone wanted to visit the The Eternal City. 

In retrospect, we should’ve checked with EU visa officers. It turns out that, for some nationalities, getting EU business visas is quite hard. Applicants that are young, unmarried, without real estate, and without children (“strong ties” criteria) are flatly rejected, regardless of the company guarantee. We later learned that the unwritten rule of international team building is:

If US-based, organize international team buildings in Mexico. If EU-based, organize in Turkey. Getting US or EU visas for international employees is a royal pain.

Unfortunately, our employees from Morocco and India were denied visas and couldn’t come. But that highlighted one enormous benefit of remote work:

Remote companies get access to an enormous talent pool, and remote workers get opportunities they couldn’t get otherwise.

It is easy to forget how divided the world is if you have a Western passport. Our employees both have education and years of experience in IT, but couldn’t get an EU business visitor visa. Ridiculous.

We booked a hotel in Rome that has a meeting space and is close to all attractions. Visas were stamped, plane tickets received, and we flew across eight time zones to meet for four days, starting May 22nd.

Needless to say, everybody was quite excited.

When everybody showed up on Monday, the first surprise came. International employees only saw the founders (Mario, Josip, and me) in a video call, where our heads occupied the same number of pixels. In reality, let’s just say we don’t occupy the same volume:

During the next four team-building days, another small surprise was that most of us were quiet introverts. I said “small surprise” because we kind of expected that—IT and remote work seem to select quiet introverts, and our company was both. 

Anticipating that people may be shy because they are meeting for the first time, we packed the next three days with social activities: Rome treasure hunt, pasta cooking class, walking tour, and a TestDome history presentation with a selection of funny events that happened in the last 10 years. While we were doing that, new funny anecdotes kept happening.

For example, I was able, on my first try, to find the worst pizza restaurant in Rome. Pizza Romana is supposed to have a stiffer dough, but this one was both hard as a rock and small—a perfect frisbee!

We learned that a senior team member has been rebuilding his house for 10 years now and still has problems with construction licenses and building statics. Coincidentally, he was also the first TestDome software architect.

A senior on the content team impressed everybody by delivering an engaging one-hour presentation with just five slides. His strategy was simple—he mentioned how ChatGPT would change our questions in the future, and suddenly everybody was participating in the heated discussion. 

We joked that a coworker from Brazil liked the team building so much that he decided to stay in Europe—and yes, he did use the travel opportunity of the team building to move from Brazil to Portugal. We are afraid to organize the next team building too soon.

Office not found: what next?

We planned to have in-person team buildings on anniversaries like 10, 15, or 20 years of TestDome. Now everybody wants to have it every two or three years. Now people find it easier to ask or request something from a colleague after they have spent a few days with them. All communication flows easier, although it is still over emails and mentions. I will give it to the haters of remote work; there is something to the in-person connection. 

But I don’t see how we could transition to the physical office. Our employees are spread around the world, I don’t know how I could ever attract such a team in the small city of Zagreb. Search for talent was the initial reason we became a remote company. We became a local poster company for remote work by accident. But even if we could attract talent locally, requiring everyone to crawl through a rush hour to be in one office from 9 to 5 now seems like an inefficient use of people’s time. Remote work, with occasional in-person team building, is still our preferred mode of operation. 

The post After 10 years, our remote-only company had its first in-person meeting appeared first on ShiftMag.