Artificial Intelligence 5: Software Options for your AI Model

Software Options for your AI Model

Artificial Intelligence 5

This lesson will help you earn points in explaining your AI project in the Demo video part of the rubric

In this lesson, you will…

  • Learn how to integrate your AI model into a project
  • Start to code your AI project

Key Terms and Concepts

  • Software - term for programs or applications that run on a computer or device

Integrating your Platform

By now, your team should have trained its AI model using a healthy dataset. You’ve tested the model with new and diverse data to make sure it performs fairly and accurately. But you can’t just leave it there! You want your model to take action based on the prediction it makes. 

You will need to implement your model into some platform or software system to make it useful and meaningful, and to be able to solve your chosen problem.

Most of the platforms discussed in Artificial Intelligence 4 will allow you to use your model directly within the platform to create an app or program that can help to achieve your goals. The one platform that just allows you to create the model is Teachable Machine. You will need to use an alternate platform, like Scratch versions from the MIT Media Lab or Stretch3, to use the model to perform some action. 

Check out your chosen platform below to learn how to integrate your trained model.

Stretch 3 (integrated with Teachable Machine)

stretch3.github.io

Stretch3 is a version of Scratch that easily incorporates machine learning models created with Teachable Machine. After you train your model in Teachable Machine, click the Export Model button.

This uploads your model to the Teachable Machine server and gives you a URL you can use in Stretch3. Copy the URL.

Open a new project in Stretch3: https://stretch3.github.io.

Click on the button to add an extension.

Add the TM2Scratch extension by clicking on it.

You will now have access to the TM2 blocks. Drag out the classification model URL block. In this example, we used a sound model, but you could also have an image or pose classification model.

Paste the copied URL from your Teachable Machine model into the blank field for URL.

To activate your model, click on the green classification model URL block to run it. It will be highlighted in yellow. When the highlight disappears, your model has been loaded, and your different classes will be accessible. 

For example, you can use a when received sound label block. This event handler will be triggered when the program detects, in this case, one of the sounds the model was trained to detect. The dropdown for label should have all the different classes you trained. In this example, it will detect snaps, claps, whistles, and background noise.

You can then code your project to take action when a label is classified. You could have a sprite do different actions depending on which sound it hears. Tailor your code to whatever your particular solution is!

Here is a video showing how to incorporate the Snap, Clap, Whistle model into Stretch3.

Machine Learning for Kids
MIT App Inventor
mblock

Activity: Coding your AI Model

What you will do:

1. Review what is needed for your particular platform to add your trained AI model to a project.

2. Create a new project in your selected platform and add your model. How you do it will depend on the process for your particular platform.

3. Take action! Add components and code to the platform so your project takes an action based on the prediction from your model.

Reflection

You’ve started to code the guts of the action part of your AI project! Here is where you can see the results of your model and what it can really do!

Now is a good time to check in with your Project Plan/Canvas to see how your project is progressing. You might want to adjust timelines and tasks based on the platform you are using and the steps you will need to achieve the goals for your project.

Once you get your code working, it’s also time to check back in with your users. Find some people to test out your project and provide feedback.

Additional Resources: Advanced Integrations

More tutorials for coding with platforms that can use an AI model can be found below.

 

MIT Media Lab Scratch version that can use a Teachable Machine model

 

Want to use some hardware like micro:bit? 

Artificial Intelligence 4: Train your AI Model

Train your AI Model

Artificial Intelligence 4

This lesson will help you earn points in explaining your AI model in the Demo video part of the rubric.

In this lesson, you will…

  • Decide on a platform to train your AI model
  • Gather your dataset and train your AI model for your project

Key Terms and Concepts

  • Platform -software or website that allows its users to perform a task or use a tool
  • Extension - external software that be easily added to an existing program or platform to gain special features or capabilities
  • Classification - Machine learning model used to identify or categorize different data

By now you should have gathered your data for your dataset. As a reminder, there are three ways of collecting data:

  1. Collect training data from your community
  2. Invention gathers data with sensors or user input
  3. Use data from public datasets for training data

You also should have decided on what type of data you want to use in your dataset - images, text, sound, or poses. 

Let’s take a final check that your dataset meets the following criteria:

  • The right kind of data
  • Lots of examples
  • Varied examples - diverse and representative (unbiased)

Choosing a Platform

There are many platforms available online to train an AI model. Most of the platforms work in a similar way. You input your data, either through upload or using the webcam for images, then you train the model, and finally test it to make sure it is accurate. Choosing which platform to use will depend on your project and what you want to do with your model once it is trained. They all allow you to create AI models that will classify or categorize things, so they are called classification systems.

 

The AI model training platforms we recommend for ease of use for novices and integration with software to allow your model to take action are:

Thunkable does have AI capability but does not give you the ability to train your own model and use it in your app. You can use Microsoft’s Azure Image Recognition, a pre-built AI model, as part of the Camera component in Thunkable. You can also use the Web API component and external Machine Learning platforms to integrate your own AI model into a Thunkable project.

In this video from IJCAI 2021, Pratham Goradia from Maker Bay in Hong Kong shows how to make an app to identify coral species. Pratham uses Ximilar to train the AI model and then uses the Web API component in Thunkable to integrate his model  into his project.

There are other more advanced AI tools available as well. Some are listed in our Additional Resources section. They may require using advanced languages like Java, Python, or Swift and will most likely entail using APIs (Application Programming Interface).

Below is a little more information about each of the platforms suggested above. Read through to determine which platform will work best for your project. It may depend on the type of data in your dataset, as not every platform can train every type of data. If you are intending to make a mobile app, App Inventor or Machine Learning for Kids are options. If you hope to integrate some sort of hardware device such as micro:bit or Raspberry Pi, you should investigate Stretch3 and mblock. 

Try out one or more of the tutorials from one or two platforms to see which platform will help you achieve the goals for your project.

Teachable Machine

teachablemachine.withgoogle.com

Google's Teachable Machine lets you easily train AI models that can be used with other platforms. In this video, learn a little about Teachable Machine and training an AI model. Teachable Machine can be used with several other websites and tools. The most straightforward integration with a blocks-based language is Stretch3, a version of Scratch. If you are intending to use any external hardware devices such as micro:bit, Arduino, or Raspberry Pi, Stretch3 in combination with Teachable Machine is a good option because it has many extensions built into it to integrate those devices.

 

Here are three tutorials to try out Teachable Machine using different data types.

Pose classification to detect head tilt

Machine Learning for Kids
MIT App Inventor
mblock

Activity: Training your AI Model

What you will do:

1. Review the different platforms above and choose one or two tutorials to test out the platform. The platform should:

    • Be able to train the type of dataset you are using (images, sounds, text, etc)
    • Allow you to use the model to integrate into your proposed AI solution. Some possibilities are:
      • Mobile app
      • Invention with sensors
      • Physical system with hardware devices

2. After you have experimented with one or two platforms/tools, decide which one your team will use for your Technovation AI Project.

3. Train your AI model with your dataset. If you haven’t completely gathered all the data examples for your dataset, you can start to add examples and add more as you collect more data. Don’t forget to save your project/model so you can return to it later!

4. Once you have trained your AI model, test it with some additional examples. These test examples should be different from the training examples you used.

5. If your AI model is not performing well, go back and add more training examples. Did you add enough examples? Were the examples diverse and representative?

Reflection

Hopefully you have chosen the platform you will use to train your AI model and eventually use to build your AI project. Consider the following questions. Answering these will help you make your case for why you chose AI and why you chose this particular technology for your Technovation project.

  • What type of data is in your dataset?
  • What aspects of your chosen platform made it a good choice for this type of data?
  • What do you hope to achieve with your AI model now that you have built it?
  • Are you confident the platform you chose will help you achieve your goals?

Additional Resources: Advanced Integrations

Remember to go over Coding 13: Cloud Storage and APIs to learn more about integrating outside services to your app. Some of these are only compatible with more advanced coding languages (such as Java or Swift) but they’re definitely worth a look regardless if you intend to use it in your app.

  • Dialogflow
    • Great for creating conversational AI apps
  • TensorFlow
    • Lots of tools like transcribing handwritten numbers, pose guessing, and more!
  • Google

VianAI

Learn from VianAI engineers how to code in Python with Google’s Colab platform to build an AI tool to help people find search results that help people achieve their dreams.

 

Also check out DialogFlow with Google Engineers:

Artificial Intelligence 3: Datasets

Datasets

Artificial Intelligence 3

This lesson will help you earn points in the selected technology section of the Demo video of the rubric.

In this lesson, you will…

  • Learn about different types of datasets
  • Start to plan the dataset for your project

Key Terms and Concepts

  • Datasets - large sets of data that are used to teach AI to recognize patterns and predict something
  • Microcontroller - small computer on a single integrated chip, used in larger computers and other systems such as appliances, vehicles, and robots
  • Sensor - a device that detects changes in the environment and is used to monitor that information within an electronic system
  • Class - a label that is provided to an AI model so it learns how to classify inputs by its class

The first step in creating an AI model that can classify something is to plan the dataset.

You want to make sure you have lots of data to train your AI model. The more examples you can give the model, the better it will perform. The data should be balanced between your different classes or labels. You want to have approximately the same number of examples for each class, in order to prevent bias for one over the other. 

As you gather training examples for your dataset, remember to keep a portion of your examples to test the trained model. You will need some data examples that were not used in training the model to test with. You need to see if your model has learned well enough to predict when presented with a new example. A good way to do this is to create two folders on your computer - one labelled Training Data and one labelled Testing Data. The bulk of your data examples should go towards training the model, but a small portion, 10-20%, should be kept aside to test for accuracy.

You want to provide diverse examples. For example, say you are creating an AI model to detect if someone is wearing a face mask or not. You should gather images that reflect varied examples:

  • Different types and colors of masks
  • Different people - genders, ethnicities, ages
  • Different backgrounds - indoors, outdoors, light, dark
  • Different head angles
  • Different placement of head in frame - close, far, left side, right side

 

What if you only trained your model using images of white men with blue surgical masks for your mask class? What happens when a female of color wearing a purple mask uses your model? How do you think it will be classified? Will your model perform well or not?

Photo credit: A Tiny CNN Architecture for Medical Face Mask Detection for Resource-Constrained Endpoints by Puranjay Mohan, Aditya Jyoti Paul, and Abhay Chirani

And of course, a dataset must be the right kind of data. You can train AI models  using numbers, text, images, and sounds. Make sure you choose the data type that is right for your project!

Determining what goes into your dataset gives you immense power! You must be careful to use lots of data, different data, and the  right type of data. Otherwise, your AI model will not be very accurate and could make bad predictions and take the wrong action. Taking the time to collect data that will make for a healthy dataset is critical to a successful model.

 

As a reminder there are three possible ways to collect data for training an AI model.

Activity: Planning your Dataset

  1. Choose what data you want to collect. Decide what information you will need to train your model to work for your project. Think about types of data (sound, images, text, numbers)
  2. Decide where you will collect the data for your dataset. Will it be community, sensors, or public datasets? 
  3. How will you collect the data? Give more detail behind your answer to #2.
  4. Decide what the classes or labels will be for your model? What will your model predict, so what categories will be necessary? You will need examples for each class you include.
  5. Decide on a target number of examples for each class. 50 per class should be a minimum.

 

Once you’ve got your dataset planning started, fill in more details:

  • If gathering from the community, start your survey or questions.
  • If gathering with sensors, list the sensors you will need.
  • If gathering from public datasets, start listing possible datasets you can use.

Additional Resources: Advanced Integrations

Hardware and Sensors

This video gives good information on the microcontroller hardware we recommend for projects using sensors: Sparkfun Workshop: Microcontrollers and Machine Learning

For a comprehensive list of sensors, check out this Wikipedia article.

 

Public datasets

If you choose to go with a public dataset, these sites will be invaluable for your project.

Artificial Intelligence 2: Find Patterns with AI

Find Patterns with AI

Artificial Intelligence 2

This lesson will help you earn points in explaining your dataset and AI model in the Demo video part of the rubric.

In this lesson, you will…

  • Learn about datasets
  • Learn how to train an artificial intelligence model to predict something

Key Terms and Concepts

  • Datasets - large sets of data that are used to teach AI to recognize patterns and predict something
  • AI model - artificial intelligence that is trained on a dataset to recognize patterns to predict or classify something
  • Supervised Learning - machine learning where a model is trained by giving it correct outputs
  • Class - a label that is provided to an AI model so it learns how to classify inputs by its class

You learned that artificial intelligence, or AI, has 3 parts:

The process of gathering data for a dataset and training it to find patterns is called building an AI model. The model can then be used within a program or app to make a prediction and take action.

Datasets 

AI models need a lot of data to make good and accurate predictions. Here we’ll look at some different ways to collect good data for your invention.

There are three possible ways to collect data (data such as images, numbers, sounds, or text) for training an AI model. 

  1. Collect training data from your community
    • The data you want to train makes sense to gather from your community
  2. Invention gathers data with sensors or user input
    • You want your invention to be able to collect data on its own (Tip: You’ll probably still need training data from one of the other methods.)
  3. Use data from public datasets for training data

You need more data than you can gather in your community, or you’re working on a solution for a problem that is more global.

We’ve gathered some free public datasets for you to explore in the Additional Resources section.

Remember the key features of a good dataset!

  • Matches your problem/solution
  • Lots of data - the more, the better
  • Accurate 
  • Comprehensive and unbiased - representing different situations
  • Have permission from the people who shared the images or information

Training the Model

Once you have gathered your dataset, you need to train it to create a model. There are many available platforms where you can train an AI model, using supervised learning. Supervised learning is just as it sounds - you supervise how the model learns by telling it the correct answer. For example, if you want the model to classify dogs vs cats, you feed it lots and lots of pictures of dogs and cats, and let it know which are the dogs and which are the cats, so you help it learn the difference. 

Unsupervised learning happens when the model is given inputs, but no labels so it has to learn patterns itself and determine which inputs are similar and in which ways.

In this curriculum, we will focus on supervised learning. You will gather your dataset and train a model by labelling classes. Each class is a label to identify something you want to classify. For example, “dog” and “cat” would be classes in our animal classifier. 

There are many free and open source platforms available to create AI classification models. Because you want to take the next step and have your model take action, we have curated this list of programs and platforms where you can build your model to make a prediction, and also use your model in a program or mobile app to perform an action based on the prediction.

Let’s look at another example. YouTube uses AI to predict what video you might want to watch.

Its dataset is all of your previous videos watched and videos that other people with similar tastes to you have watched.

It learns your patterns for video watching from this model and predicts what you might want to watch, both when you first visit the website and what you might want to watch after viewing  a particular video.

The action it takes is to display thumbnails of videos you might click on and watch. 

The AI model creation platforms we recommend for ease of use for novices and integration with software to allow your model to take action are:

Platform Classification Types Integration
Teachable Machine by Google Text, sounds, poses Scratch via MIT Media Lab, Scratch via Stretch3, other integrations
MachineLearningForKids Images, sounds, text, numbers Scratch, App Inventor, Python
MIT App Inventor Images, sounds Mobile apps with App Inventor
Stretch3 Sound, images, poses Scratch
mblock Images Scratch, Python

Activity: Train Your Own AI Model

For this first experience training an AI model, we will use MachineLearningForKids. This site is a great place to learn how AI works, and to get your feet wet creating your first model. 

NOTE: It is important for your teacher or Technovation mentor to sign up for a teacher account first. That way you will be able to log in and save your projects. You can try out MachineLearningForKids without an account, but you won’t be able to save your model or full project for use later.

As a simple example, you will create an AI model to recognize images of rock, paper, scissors so you can interact with the computer to play the game. Here is a video tutorial to show you how to make the model and complete the project. If you prefer, you can also use this worksheet and follow the instructions.

If you do get your teacher or mentor to sign up and give you an account on the MachineLearningForKids website, your project, which includes your AI model, will be saved and you can return and access it again. However, the Scratch project does not get saved. In order to save a copy, when you are in Scratch 3 from within MachineLearningForKids, save your Scratch project to your computer.  Then, if needed, you can load it from your computer the next time you visit MachineLearningForKids.

Reflection

You’ve made your first AI model! This should give you a glimpse into the process for making the model. You identify your labels or classes, then gather your data to input, train the model, and use it in a project - in this case, a Scratch project. No matter which platform you use from the suggestions above, they all work in a similar fashion, although the interfaces may differ slightly. 

  1. What did you think about your AI model? Was it successful in detecting rock, paper or scissors?
  2. Was it made with a “healthy” dataset?
  3. How could you make the dataset better?
  4. If a friend or person in a different location from you used your model and project, would it perform as well? Why or why not?

Additional Resources: Advanced Integrations

Build your own AI Models

Here are some of the platforms we use to explore AI:

 

Research datasets

Here are some places to start exploring datasets out there.

 

SolveIt Series by Technovation

Technovation created a video series that challenges you to expand your mind and tackle new problems. A lot of these concepts apply directly to creating your app and using AI!

Artificial Intelligence 1: All About AI

All About AI

Artificial Intelligence 1

This lesson will help you earn points in the technology section of the Demo video part of the rubric.

In this lesson, you will…

  • Learn how artificial intelligence works

Key Terms and Concepts

  • Artificial intelligence (AI) - machines/programming that can do tasks normally thought to be done only by humans
  • Machine Learning - the subset of AI where a technology is trained with data and “learns” to recognize patterns in order to make predictions.
  • Datasets - information that is used to teach AI to recognize patterns and predict something.

Earlier, you learned how artificial intelligence is being used in different areas to make an extraordinary impact on our daily lives. Now, let’s look at how it actually works.

AI has 3 basic parts:

For AI to work, it needs a LOT of data to learn from. Technological advancements have allowed more information to be gathered faster than ever before. That’s one thing that makes AI possible now! The AI learns from the data and finds patterns on its own.  Then when it takes in new data, it can make a prediction.

Think about yourself as a human. What sort of predictions do people make? What inputs and patterns do you consider when making these predictions? Many people love predicting the weather, some people like to predict scores of sports games, and other people like to predict what might happen in a movie. 

Data can come in different forms. In these lessons, we’ll be using sounds, numbers, text, or pictures. 

In your household, what sorts of data do you create every day via technology?

Did you think of any of these?

  • Every Google search, words you type into emails
  • Every question you ask Alexa/Siri/OK Google
  • Connected devices - each time you turn on lights, Air conditioner temperature
  • Taps you make on your cell phone
  • Anything you purchase online
  • Who you are connected to on social sites
  • Songs you listen to
  • Steps you take

Zooming back out, let’s look at an example with Google Maps.

How does Google Maps use AI to give you directions to where you want to go?

Step 1

Google Maps takes in the following inputs to make a prediction:

  • Current location
  • Destination
  • Mode (walk, car, public transport)
  • Traffic

Step 2

But does AI stop at just making a prediction?

Step 3

Google Maps also takes action after making a prediction.

Step 4

The action/decision Google Maps takes: Shows you the best route.

Collecting Good Data

AI models are programmed algorithms that are trained on data that replicate human decision making. AI models need a lot of data to make good and accurate predictions. Here we’ll look at some different ways to collect good data for your invention.

But… What is “good” data? Consider the following:

  • Matches your problem/solution
  • Plentiful - the more, the better
  • Accurate 
  • Comprehensive - good variety representing different situations
  • Has permission from the people who shared the images or information 

There are three popular ways to collect data (data such as images, numbers, sounds, or text) for training and using in their AI models. Here they are with a reason why you might want to follow a certain method:

1. Collect your own training data from your community

You can gather a lot of data you need from your community from a variety of situations, and you can get permission to use it.

2. Invention gathers data with sensors or user input

You want your invention to be able to collect data on its own. You'll learn more about mobile sensors in Coding Lesson 11.

Tip: You’ll probably still additionally need training data from one of the other methods.

3. Use data from public datasets for training data*

You need more data than you can gather in your community, or you’re working on a solution for a problem that is more global.

*Small warning: Lots of free public datasets exist. You can see if you can use one! But sometimes these datasets require a lot of work to even decide if/what you can use. Some people have created whole jobs of making public datasets into good resources and then selling them. 

We’ve gathered some free public datasets for you to explore in the Additional Resources section.

Activity: Putting It All Together

Write down some of the apps or technologies that you and people you know use that utilize AI. Try to identify at least 3 different examples. Then consider the following:

What sort of data does it use to make predictions? How does it collect data?

What prediction is it making?

What action is taken after making its prediction?

Activity: Testing Out AI

You’ve brainstormed about apps on your own, now let’s try out some examples of AI in action.

Explore some of the following websites and get a taste of what AI can do. As you explore, think about further applications of the technology you see. Could you take the concept in front of you and apply it in a different context? AI uses lots and lots of data (images, text, sounds) and machine learning to create a model that can predict something. What sort of data would be necessary to make these applications work?

Imaginary Soundscape

Imagine you’re traveling and visiting another city. Many of us would think of what that place would look like, but have you ever thought about what it would sound like? This technology does just that. Based on an image, AI generates what it believes you would hear if you were actually there.

AutoDraw

How many times have you had this clear image in your head and when you tried to draw it, it didn’t come out like you had in mind? With this website, AI takes your doodling and predicts what it is you’re trying to draw.

X Degrees of Separation

This one gives a glimpse of how AI “thinks” by taking two art pieces and showing us a bridge of similar artworks that connects the two together.

Want to explore some more?

Experiments with Google

NVIDIA AI Playground

Reflection

Now that you have a glimpse of what it takes to create artificial intelligence, you might want to think about the problem you are solving, and how AI might be useful. Can you think about your problem and possible solution in terms of the three parts of AI - data, pattern, prediction? How would you address all three in your solution?

Additional Resources: Advanced Integrations

Build your own AI Models

Here are some of the platforms we use to explore AI:

 

Research datasets

Here are some places to start exploring datasets out there.

 

SolveIt Series by Technovation

Technovation created a video series that challenges you to expand your mind and tackle new problems. A lot of these concepts apply directly to creating your app and using AI!

Artificial Intelligence 2: AI and Your App

AI and Your App

Artificial Intelligence 2

This is a bonus lesson that will expand your knowledge about AI and how it works. It will not be a scored part of your submission but it could be helpful to learn how you could integrate AI with your app. If you need a refresher on how AI works, check out the other AI lesson in Week 2: All About AI.

In this lesson, you will…

  • Consider the ethics of AI
  • Decide if AI is right for your app
  • Learn how to integrate some AI elements with App Inventor and Thunkable

Key Terms and Concepts

  • Ethics - a set of moral principles that affect how people decide what’s right or wrong
  • Bias - preconceived ideas somebody has that are often unfair to some people or groups

We’ll be taking a deeper dive into AI application and your app today. If you need a refresher, review Artificial Intelligence 1: All About AI

Most importantly, remember the 4 basic parts of AI:

If you need an example, check out this video from our friends at Google that shows how someone used AI to act on a problem:

Google and Technovation also held a workshop on an AI tool called Dialogflow that you can check out in the additional resources section!

By this point, you likely have ideas about what you want your app to do. Will AI be important for your app? Let’s consider a few more things before we answer that.

Ethics and Actions

Ethics is a part of philosophy that has to do with what is right or wrong. This is an important topic in the world of AI today for many reasons! You’ll want to make sure your app is making decisions that help people and society instead of causing harm, even if the harm is by accident. Here is a story from Joy Buolamwini about her experiences with technological bias. 

Put yourself in her shoes, how would you respond to the findings? If you were the inventor, would you feel comfortable releasing similar technology? How can we make technology more fair and representative of all kinds of people?

One place to start is by thinking about “healthy” data. Removing bias in your data helps make your model healthier, so that it can make better decisions.

We’ve already learned about some details around bias in training data, here’s a video that outlines more:


 

The Weed Puller

But data isn’t the only area in AI in which we can think about ethics . We can also think about the ethics of the use or the results of the AI. Imagine an AI technology called the Weed Puller that predicts what plants are weeds and pulls them out. Using the Weed Puller as a base, consider the following:

This may be a lesson focused on AI, but these ethical questions are applicable to your entire app

As creators, we bear responsibility on how our technology interacts with people and we should always be mindful of the potential impact our creations can have.


 

Which solutions need AI?

Not every problem requires technology or AI. Think about what sort of problems are best suited for AI such as:

  • Situations that are not programmable (ex - Google Maps)
  • Situations that can predict something when given new information (ex- when given a new x-ray, an AI tech could identify cancer)
  • Often connected to actions (ex - smart vacuum)

Carefully consider if your app should use AI. Does it add significant value to your core design functions?

idea-3222605_1280

 

AI Features in App Inventor and Thunkable

Here are the main features available in App Inventor and Thunkable that utilize AI:

ai-bee-logo.png
Image Recognition Speech Recognition Language Translation
App Inventor¹²
  • LookExtension
  • PersonalImageClassifier
  • PersonalAudioClassifier
  • Speech Recognition
  • Language Translation
Thunkable
Thunkable.png

¹ Artificial intelligence blocks are not natively on the App Inventor platform and need to be downloaded as extensions. Official extensions are available here. Learn how to integrate extensions in your App Inventor platform here.

² Machine Learning for Kids, a website where you can train your own machine learning models, can integrate with App Inventor. Watch this video for an example on how to build your first model. After you train your first model, you'll be able to import your ML4K project as an App Inventor extension.

As you can see, the functions available in App Inventor and Thunkable are a bit basic; you won’t be training your own AI model without outside integration.

However, in Coding 13: Cloud Storage and APIs, you’ll learn about what APIs are and how to connect other resources to your own app. You can find other programs out there that will give you more control of your AI model and integrate them to your app that way instead.

More resources can be found in the Additional Resources section of this lesson.

Activity: Image Recognition

Let's code a simple image recognition app. The steps for App Inventor and Thunkable are outlined in their respective sections:

App Inventor

Thunkable

Setting Up 1

First, let’s get all our screen components in place.

 

 

 

 

 

 

 

 

 

We have two Labels, a WebViewer, two Buttons within a Horizontal Row, and the LookExtension (more on that in Step 2).

 

 

Setting Up 1

First, let’s get all our screen components in place.

We have an Image, a Button, two Labels, a Camera, and an Image Recognizer.

---

Setting Up 2

You’ll notice that there’s a LookExtension1 that’s not normally available in App Inventor. Here is the official list of App Inventor Extensions. Here are the instructions on how to import an extension into your project.


 

Coding 1

There are two buttons available to the user: Classify and Toggle. Here are the tasks we want to accomplish:

  1. Let the user from using the Classify button when LookExtension is ready. (The LookExtension is that it needs a bit of time to load and ready up.)
  2. When the Classify button is clicked, it should call LookExtension to analyze what the camera sees.
  3. When the Toggle button is clicked, the camera that the app uses should switch to another available camera on the device.
  4. Then finally, once LookExtension has analyzed the image from the camera, the app should tell through the StatusLabel what its prediction is.

Coding 1

There is one button available to the user: Classify. Here are the tasks we want to accomplish:

  1. When the Classify button is clicked, the Camera should first take a picture.
  2. Then Image Recognizer should  analyze the image from the camera we got from the camera.
  3. Finally, the app should show the user the photo, inform the user what Image Recognizer predicts the photo is, and how confident the app is in its prediction. We want to inform the user by setting the Image displayed and changing the text Labels.

---

---

1. Let the user use the Classify button when LookExtension is ready.

Hint: Look through the blocks available in LookExtension and see if there are any that would be useful for you.

1. When the Classify button is clicked, the Camera should first take a picture.

Hint: Where should we look when we want something to happen when a button is clicked?

---

---

2. When the Classify button is clicked, it should call LookExtension to analyze what the camera sees.

Hint: Nothing too fancy here. "When a Button is clicked, do something." How do we do that?

2. Then Image Recognizer should  analyze the image from the camera we got from the camera.

Hint: Look in the blocks under Image Recognizer. Which one would be useful here? What input does it need?

---

---

3. When the Toggle button is clicked, the camera that the app uses should switch to another available camera on the device.

Hint: Another button! Remember to look in the available blocks under LookExtension.

3. Finally, the app should show the user the photo, inform the user what Image Recognizer predicts the photo is, and how confident the app is in its prediction. We want to inform the user by setting the Image displayed and changing the text Labels.

Hint: We’re changing two labels and one image here. Use blocks from the appropriate section.

---

---

4. Once LookExtension has analyzed the image from the camera,, the app should tell through the StatusLabel what its prediction is.

Hint: LookExtension stores its predictions as a list of possible answers. This list is named “result” and its most confident prediction is the first item in this list.

---

You’re ready to test!

Connect your mobile device to Thunkable and try it out!


 

Coding 2

We also need to handle how this app deals with errors. 

Many apps have a way to handle errors. For example, think of when you enter the incorrect password to a login. Most of the time, it’ll say “Invalid Password, try again”.

The text there is purely for humans to understand what the app thinks is wrong. Even when things don’t go right, it’s important to understand what’s happening!

Copy the following code into your project. Read through it and make an effort to understand what’s going on.

You’re ready to test!

Connect your mobile device to App Inventor and try it out!

Reflection

Once you’ve taken a few pictures with your new image recognition app, take a moment and reflect:

  • What improvements to this app could you make? 
  • Could image recognition be used to help solve a problem in your community? What about speech recognition or other types of AI?
  • Using whatever you answered with in the previous question, consider the ethics behind your solution.
    • Who would be affected by your solution?
    • What good could it do? What harm could it do?
    • How can you make it cause the most good and least harm to direct and indirect users?

Additional Resources: Advanced Integrations

Advanced AI Integrations

Remember to go over Coding 13: Cloud Storage and APIs to learn more about integrating outside services to your app. Some of these are only compatible with more advanced coding languages (such as Java or Swift) but they’re definitely worth a look regardless if you intend to use it in your app.

  • Dialogflow
    • Great for creating conversational AI apps (scroll down a bit more for a video!).
  • TensorFlow
    • Lots of tools like transcribing handwritten numbers, pose guessing, and more!
  • Google
    • Google has a large library of AI tools to use such as translations, image recognition, and more. Check out this overview video for a nice summary. Note: If you decide to use these tools, be sure to double check the pricing. Some tools are free to use depending on how many users use your app.

 

SolveIt Series by Technovation

The SolveIt series has some more topics for you to reflect on. Here are a few that discuss about ethics and technology.

For more about ethics, you can also check out the MIT AI Ethics Curriculum

AI in Action

Technovation partnered up with VianAI and Google to do a couple workshops involving AI. Watch coding with AI in action!