The Challenge

While applying deep learning to quality assurance tasks in manufacturing at wattx we faced the problem that some use cases required a lot of time to prepare datasets for training purposes — even simple bounding box annotations required hours of “boring” work. We were searching for alternative tools to automate this work, but none had the desired performance or were too expensive. First, we tried to extend existing tools using APIs, but we only got integration problems. Then we tried to hire working students or use crowdsourced platforms, which also took a few days to label a dataset — with suboptimal accuracy. Thus, we decided to build a tool, that fits our requirements.

So when the time of our company-wide hackathon came, my colleague decided to create a prototype that would learn from you on the go, and with the help of machine learning, you would be able to do it faster. After the hackathon, we wrote a blog post, which received a lot of attention, and since wattx is a company builder, we decided to give this project a chance and put together a small team to work on an MVP.

Our high level goals for MVP were to:

• Create a minimum set of manual labeling features
• Create a set of AI assistants to speed up the workflow
• Test out the speed of labeling using Hasty vs it's competitors

My first product research and design roadmap looked like this:

Research & MVP

Our early research was mainly focused on Automation Level 1 (please see this article to read more about automation levels). We hypothesized that we had to provide our users with the best possible manual labeling tools so that they could label the first set of images in a speedy manner but not sacrificing the quality. This was confirmed through the first set of interviews and tests. ‍

The defined problem statement was

""

My main goal as a Product Designer was to come up with a smart and familiar way for the users to label the first 10-15 images. As an inspiration, I referred to the tools like Adobe Photoshop and Illustrator.

Here is an overview of the first set of manual tools that we came up with:

• Contour looks at outlines based on contrasts - great for objects on a contrasting background
• GrabCut extracts the background from the foreground for a predefined region great for objects on a monochromatic background
• Magic wand selects an area by finding similar pixels near the selected pixel for a given range - great for monochromatic (or close to) objects

magic wand

After the first phase was finished, we focused on AI tools implementation:

• Class predictor - a couple of classification networks, that need only a few labels per class, to build
  a model that can set class labels for the annotator. The model is continuously updated and
  great for projects with many classes (such as FMCG).
• DEXTR uses a model trained on a large generic dataset to attempt to identify the outline of an object within a defined region - great for dynamic objects on dynamic backgrounds
• Object detector - a network detects the location of each object automatically and applies a
  bounded box. The user will only need to adjust the predicted labels at the earlier stages.
• Object segmentation - this network identifies the location of each pixel of each object on the
  image and applies a pixel-perfect mask. After dozens of labelled images, the remaining ones will be
  annotated completely automatically - this is the goal of the tool, automated labeling

Manual vs Automated Labeling Case Study

We made Hasty Freemium so that users can do a pilot or proof of concept for free and only need to start paying when taking a project into production. Further, we gave users full access to all the AI assistants before having to pay. Our research distinctly showed that this alone could be a great differentiating factor against our competitors.

This meant that my main focus as a designer was to bring first-time users through a funnel of manual labeling to automated labeling as quick as possible because that's where the fun happens!

My initial designs failed miserably after the first (qualitative) user test. The problem was that most of the people understood the concept of tools like Photoshop and Illustrator, but couldn't understand what and how long they needed to go from manual to automated labeling because the set of tools wasn't available for them yet. I solved it firstly by visually drawing a clear distinction between manual and automated sets of tools, and secondly, by providing context menus and notifications that informed the user about the availability of the automated tools.

Before:

After: