Google’s Mood Board Search is a simple web-based Computer Vision tool that lets you teach your computer to recognize visual concepts using mood boards and Machine Learning. Mood Board Search was designed with a primary focus on ‘Visual Feels,’ bringing an abstract and subjective angle to an AI-based visual platform. This platform lets you express your mood and thoughts in a visual format, especially for abstract concepts like ‘Atmosphere,’ ‘Duality,’ ‘Fracture,’ or any particular idea or state of mind.
The thought behind Mood Board Search was to create an approachable and flexible interface for people without ML expertise, enabling them to train a model and recognize a visual concept. With each mood board, you can train a machine learning model using a technique called Concept Activation Vectors or simply CAV.
What is a CAV?
Before understanding how Google combines the ML and Mood Boards to bring out interesting visual concepts to life, we need to understand what CAV is. Put simply, CAVs are a creative approach to understanding how machines think.
For humans, CAVs are simply a way of seeing things. You can think of it as a pair of glasses that lets you see the world in a specific way. For machines, CAVs are like a direction in a higher-dimensional space. CAVs are great at spotting visual threads across different images, spotting similarities in color, shape, pattern, composition, and even texture.
CAVs are also very tiny compared to a traditional Deep Learning model. For example, a trained object detection network might be hundreds of megabytes large – where as a trained CAV file might be just a few hundred kilobytes.
How does it work?
You need to select a concept you want to express in images. You can pick anything, from well-defined objects like ‘cars’ to something as abstract as ‘spirituality.
The next step requires you to gather and upload images to the platform. For this, you only need to pick 50 images that best represent your concept.
Select an image set to search and click on ‘Learn Concept’ to start training the model. Train your CAV and explore the results to see how well the model understands your concept. You can also inspect faults and retrain your models to get better results every time. Once you are satisfied with your CAV, you can download it and use it for other projects.
Uses and Applications
You can use your downloaded CAV file and integrate it into your existing projects, websites, applications, Python libraries, and above all, share your perspective with others.
This entire project is a team-up between Google Research and Nord Projects. Nord Projects have also developed a CAV Camera that lets you take your CAV and run it in real life.
Further, you can also run inferencing programs in Python to derive the model’s accuracy and verify image classification.
If you are an explorer, AI enthusiast or an artist, this platform is the right place for you. Speaking of artists and designers, Mood Board search is the perfect digital blank canvas which can be used to align your vision with the machine’s.
Further, for artists and designers creating dozens of mood boards daily, this tool is a perfect way to experience and source new visuals. You can train the model with your images and expect to get a surprising new result every time, something you can use later for your presentations.
Capabilities and Our Experiments
We tried to compare Mood Board Search with traditional ML models used in the industry for Computer Vision applications, and the results shocked us. Even with a fundamental and lightweight framework, the accuracy of this model for certain concepts astonished us.
We trained this model for simple concepts like staircases, cars, and other everyday objects with well-defined edges, along with things that have confusing patterns and random edges like galaxies and clouds.
We experimented with an image-set of galaxies to train a model and then tested it against a random galaxy image. To our surprise, the model showed an accuracy up to 94%.
We then tried the classification of different objects through python inference and found that the model was unable to distinguish between basic geometrical shapes like squares and rectangles.
Mood Board Search is a playful way to explore and analyze image collections. A perfect tool for people with no Machine Learning experience to make the most out of different visual concepts. This platform uses existing pre-trained computer vision models like GoogLeNet and MobileNet and focuses on visual experience and abstract prompts rather than objective classification and distinction like traditional models.
Created for an artistic experience, the model’s accuracy was something that blew us away. The ease of access and usage is commendable, and the platform only takes 50 images to train a model and deliver results. But the best part of this model remains the subjective approach, a quotient that has been missing from AI-based applications. Google Mood Board Search is all about capturing your thoughts and state of mind to represent them visually.