10 Steps – Bring AI in Containers to Edge

Background

“Hey Marvin, what if the internet connection drops? Will your intelligent app still run as expected?”
“uh… no.. as the AI service used by the app is hosted on the cloud”

When this question hit me months back, I really have no answer for that. I hope we can deploy some models, but I got to train the model by myself and it takes time.

The cloud AI services like Microsoft Cognitive Services are great. But, there are some scenarios that we are required to run AI services on the edge devices and micro chips that may not be connected to Internet all the time and the AI services hosted on the cloud really can’t do much about it.

Don’t get me wrong, Microsoft Cognitive Services are really good, but it depends on the use cases you are working on. If your application always require internet connection and it is always internet connected focused, then cloud AI services are good.

Till months ago, Microsoft has launched Azure Cognitive Services containers. Though it is still in preview, but it would be really great to get to try it out!

Azure Cognitive Services Containers

Azure Cognitive Services in Containers allows developers to use the same rich APIs that Microsoft Cognitive Service has and be deployed on the edge devices with Dockers.

Worth to mention, it supports following services at the point of writing:

  • Computer Vision
  • Face
  • LUIS
  • Text Analytics (Key Phrase Extraction, Language Detection, Sentiment Analysis)

The containerized AI service will not only allow app to consume offline but also decrease the latency as it is just right in the device!

Of course, it comes with some down sides where developers may need to plan how to update the Azure Cognitive Services Containers in the devices.

Pre-requisites

Get Our Experiment ON with Text Analysis for Sentiment!

  1. First, install and setup Docker on your machines.
  2. Create a Cognitive Services subscription on the Azure Portal
    • Fill everything and select a datacenter (Important to note this). For this demo, I will select Southeastasia.
    • A popup will show up when it is created successfully.
  3. Go to the service and you should see the following screen.
  4. After created the service, grab the APIKey from the subscription under “Key” tab.
  5. Then, start your docker with Linux Containers and run following script on Powershell to pull latest
    Text Analysis – Sentiment service model into the container. docker pull mcr.microsoft.com/azure-cognitive-services/sentiment:latest
  6. Once the image is ready, type the following docker script to start the Text Analysis – Sentiment service on Docker. docker run –rm -it -p 5000:5000 –memory 4g –cpus 1 mcr.microsoft.com/azure-cognitive-services/sentiment Eula=accept Billing=https://
    <Replace with the selected data center in step 2> .api.cognitive.microsoft.com/text/analytics/v2.0 ApiKey=<Replace with the key obtained in step 3>
  7. Logon to http://localhost:5000/swagger/index.html and use swagger to test the services.

  8. The result is returned fairly quickly as it is running on the device itself, of course, in this case, my own machine. 🙂
  9. The same service is now running on Docker and it also supports multi-language as same Microsoft Cognitve Service on Azure.
  10. To stop the service on container, simply press CTRL+C on your PowerShell/CMD/Terminal.

What I wish next for AI Service in Containers!

The Azure Cognitive Service in Container is still in preview mode and it looks super promising. I definitely hope Microsoft would bring the entire available service on cloud to this containerized feature and I hope one more extra would be Bot Framework to be brought into the cool technology too!

LIKE Techconnect.io on Facebook:
https://www.facebook.com/techconnectio

Follow our Techconnect Authors:
Cheah Eng Soon, Microsoft MVP – @CheahEngSoon
Cheah Eng Teong, Microsoft MVP – @walkercet
Goh Chun Lin, CTO & .NET Foundation Member – @goh_chunlin
Marvin Heng, Microsoft MVP – @hmheng

About hmheng

Marvin possess a strong passion in smart AI, bot & cloud development. He is good in planning, executing duties along with stakeholders to achieve goals. Equipped with broad experiences and skills to work across different industries. Awarded with Microsoft MVP in 2018. “To me, bringing new ideas into reality is great excitement and achievement.”

View all posts by hmheng →

Leave a Reply

Your email address will not be published. Required fields are marked *