Watch what you eat with Clarifai!
Learn how to use Clarifai’s food model and the Wolfram Alpha API to recognize what you’re eating & get a nutritional breakdown. Full source code available on Github.
Whether you’re a college student, an athlete, or an office worker, food is essential to your daily survival. Even so, we aren’t instinctively able to keep track of the nutrition of what we’re eating. Even while our smartphones are able to send emails, take phone calls, play videos, and complete thousands of other tasks, it’s still difficult to figure out and keep track of the nutrition values of the things we’re putting into our body. Using this simple tool, you can instantly check the nutrition facts of anything that you’re eating. Say hello to a healthier you!
Clarifai’s Food model offers an incredible tool for developers to use to build apps using machine learning and image recognition. It recognizes over 1,000 food items down to the ingredients, from commercially available foods to home-cooked meals. Recently, a team at Electric City Hacks implemented a JavaScript-based Ionic project that automatically checks an image taken of food and checks it for nutritional information. We were inspired by what they built and wanted to show you how to recreate a part of the app that allows you to find the nutritional information for your food and display it in your browser.
If you want to take a look at the completed project, feel free to view the GitHub Repo here. You can refer to the readme for details on how to deploy the app and start testing out your photos. The code is also commented, so you can follow along and modify the codebase if you’d like to change any of the functionalities.
Getting Started
First, we’ll need to set up a Clarifai account. You can follow the direction here to create a new application. Select ‘Food’ as your basic workflow.
Next, you’ll want to click on ‘API Keys’. You’ll find your information in the dropdown below. Make sure to take down your API Key, and don’t share this string with anyone else! It will be used to verify that you’re the developer accessing the API.
We’ll also be using the Wolfram Alpha API to access nutrition information about our food items, so we’ll also need to set up a developer account here. Click on ‘Get an AppID’ to generate an ID for your application.
We’ll also need to set up a markup for the project. You can create a file called index.html and copy the code below. This ensures that you have jQuery and the Clarifai JavaScript client.
Diving in
First, we’ll need to fill in our API credentials and initialize a new application with the Clarifai JavaScript Client.
Next, we’ll include some boilerplate code to process an image file from an input and convert it to a Base64-encoded string. This will be the method used to send the information from the image to Clarifai in order to run it through the Food model.
We’ll want some way to visually indicate to the user that their photo has been accepted. To do this, we’ll show a preview of the photo below the upload section. We can also do some CSS styling on the page to make it more visually appealing.
Now for the fun part — making our prediction calls to Clarifai! Here is a code snippet that predicts the food found in the photo. We’ll explain a little bit of what this does later in the tutorial.
We first feed in the function an image in Base64 and then use app.models.predict() to tell Clarifai which model we’re querying against (Food) along with an image string.
The system then goes through the photo and outputs tag responses based on the highest probability food that the photo matches from its own trained database. Finally, it sends the response to the Wolfram API in the form of a POST request and retrieves the output XML as a string. We can then parse this string for the nutrition data in the form of a generated .gif file.
We created a #contents div earlier, but left it empty. Now, we’ll populate it with our image preview and the .gif containing the nutrition information, as well as the highest probability food item that was identified.
The Final Steps
You can now pull up index.html in a web browser and see your very own nutrition analyzer! You can use this as a lightweight means to track the health and nutritional value of food you’ve recently eaten, or to just check on your daily percentages if you’re ever curious.
Next try changing the values you’re parsing from the Food model, testing out the app with different photos, or even integrating another API to add different functionalities!
Download the full source code on Github.
—
If you thought this project was cool, try attending an MLH hackathon and building a project of your own with Clarifai! We’re always looking for cool projects to feature on the blog, so make sure to share yours with @MLHacks and @Clarifai for a chance to be featured.
