Challenges
Over the last few years, the field of machine learning has had major gains and computer vision has gotten VERY good. Built on deep learning, a cutting edge form of machine learning, the font identification neural net can identify most fonts in its database with high accuracy, and it can do so very quickly and without needing human intervention. It’s so good that it turned out one of the main challenges with the design of this app was actually to intentionally slow things down, due to limitations on the number of API requests we could reasonably make at one time.
My original concept for the UX was something like a two step process where the user would upload a photo, and the app would tell them what fonts were used in it, just like the way you’d be able to ask your smart designer friend to identify fonts for you. Sounds straightforward, right? Well, in practice, it turned out that that intended user flow was a little too good to be true.
In testing, we discovered that although it was technically possible for the font identification neural net to identify every piece of text in a complex image at once, doing so would cause too many API requests to the server which would slow the whole app down pretty badly. This technical limitation ended up affecting the final design of the app. My challenge was to create a user experience with just the right amount of friction, to balance technical needs with user needs while still keeping the experience as smooth as possible.
Computers don’t see text the same way we do. While a human may look at this image and intuitively know that each word in it is the same font, a computer looks at it and assumes that each word could be a different font and each one needs to be identified separately.
So in the case of a complex image, like a page from a book, the server would get bogged down as the system tried to individually identify each piece of text spotted by the computer vision API.
My solution for this was to have the user select a single piece of text they wanted to identify. I was originally concerned that the user would want to select all the text at once, but in usability testing I found that they easily adjusted to this workflow. We helped them out by pre-selecting one of the bits of text that we identified automatically. This worked quite well in practice and users were able to easily tap through to their font results.
At the end of the day, as user experience designers, it’s not enough to understand how the user would want to use the product, we also need to understand the benefits and limitations of the technologies we work with. I was lucky enough to work with a really engaged and passionate team who were excited to help me understand how the new technology they had built worked, and it was through this that I was able to come up with design solutions to meet the technical challenges.
