TinyML Survey Report

ML technology has become a reality for endpoint devices. What challenges and opportunities lie ahead for developers?

The future of ML shifts to the edge

New survey reveals how developers are accelerating tinyML innovation

Introduction

Opportunity at the edge

In just a few short years, ML technology has become a reality for endpoint devices. The ability to run compact but powerful algorithms on tinyML devices – endpoint systems that run machine learning locally rather than in the cloud – means more intelligence at the edge. It also means lower latency, improved security and energy efficiency and faster time-to-results.

But getting there has been an adventure, as myriad approaches to hardware, operating system software, training, data acquisition and model development and deployment have emerged. This often has the effect of confusing potential adopters, but fortunately, as with most technological transformations, the area is maturing.

Arm, with support from AWS, RaspberryPi, Arduino, the tinyML Foundation, and Edge Impulse, set out to gauge the current state of tinyML development, what’s worked and what remains a challenge. From June-July 2021, we surveyed global developers and received 667 valid responses.

The results were fascinating.

AI/ML spreads its wings

Development is moving quickly from cloud to endpoints

How can we accelerate change?

The arc of AI adoption has been decades in the making, but in just the past few years the flood gates of innovation have burst open, first around AI development in the cloud and now out at the endpoints, where the real world meets the digital. This rapid adoption is a testament to the early successes of AI and ML but also of the robust, heterogeneous IP, tools offerings and developer ecosystem that are fast emerging.

This is clearly reflected in the survey results: Embedded developers are quickly embracing the power and potential of tinyML on endpoints as a new way to design more intelligent autonomous systems. AI developers with data expertise are driving adoption of tinyML as well. This transformation is yielding much faster prototypes and much shorter time-to-market for software development teams.

Strong correlation between data science, DSP skills and embedded skills for professional developers.

However, nearly half the respondents struggle with modeling optimization, and best practices are not yet cemented around tools and educational channels. In short, opportunity awaits.

Embedded developers and data scientists converge on the same point.

As endpoint design is fundamentally an embedded application, it’s perhaps no surprise that embedded designers and developers are embracing tinyML at the endpoint. Most of the respondents are well versed in Python, C++ and embedded software, and they have a strong correlation between data science expertise and DSP skills.

Experienced embedded designers are devouring information to learn more about data and the discipline of data science to expand their skills in this area.

Popular tinyML implementations include sensor fusion, object classification and anomaly detection.

The development of endpoint AI also doesn’t appear to have created siloed areas of expertise inside companies, according to the data. A significant number of developers are involved in both the data and modelling sides of the development equation. Nearly 60 percent are involved in the system architecture development, and a similar number are also involved in data pre-processing and feature extraction as well as training and ML model experimentation. And while that might suggest that that cohort has more expertise in data science, that’s the not the case: Those with strong embedded skills but weaker data expertise are just as likely to be involved in data processing and model training as their data-science expert counterparts.

What lies at those endpoints?

Gauging the long-term success and popularity of any emerging technology can usually be judged by the breadth of early applications, and endpoint AI may be a technology with some of the broadest application that we’ve seen in decades.

Teams are implementing tinyML in many different use cases, including face, object, anomaly and gesture detection, biometric awareness, speech recognition object classification, real-time recognition and more. The most popular tinyML implementations include sensor fusion, object classification and anomaly detection.

Implementation is also spread broadly into vertical segments, although the most popular today are smart cities and homes, wearables and electronics, industrial automation and healthcare.

Accelerating innovation

The other feature of endpoint AI that’s energizing developers is ease of design that’s leading to faster
proof-of-concept (PoC).

A majority of the respondents said they are developing their first PoCs within four weeks. The popular choices of hardware include development boards from silicon vendors, Raspberry Pi, existing custom hardware, and Arduino. Many developers chose to develop their first PoCs just on their development machine/PC because they don't want to be limited by the hardware constraints of the development boards in the early stage of development.

Some respondents agreed that development approaches, such as leveraging Raspberry Pi, were excellent for developing ideas quickly.

And often the development is model-first, in which ML engineers focus on the accuracy of the algorithm, experimenting with different ones to find the one with the best accuracy. This is not developed with particular hardware in mind. But once a team has settled on an algorithm, then hardware consideration begins.

The benefit of that dynamic is that production timelines are quickly shrinking. Nearly three-quarters of the respondents said they moved from concept to production in no more than a year. Additionally, there’s not much difference when it comes to industry segment how long it takes to move into production, with the notable exception of industrial (76 percent of industrial networking projects can be finished in six months, followed by manufacturing at roughly 60 percent).

41% of professional developers have tinyML projects in production

Real-world innovations

This acceleration of innovation is yielding results from tinyML in the real world. Consider Edge Impulse’s grid-monitoring system.

The RAM-1 – an intelligent grid sensor developed in collaboration between Edge Impulse, research institute IRNAS, and equipment manufacturer Izoelektro – seeks to help grids prevent fires from electrical faults before they happen. Specifically, the RAM-1 device analyzes electrical waveforms on an ongoing basis. If the AI algorithms on the device detect an anomaly, data from the select event is sent to the cloud for further classification. If a problem is indicated, warnings are relayed to control rooms and field crews. Integrating this functionality on an MCU means the product can operate for years on a single battery charge, which is an important feature on grids that are often miles from civilization.

Or consider Rainforest Connection’s bio-acoustic monitoring technology systems that seek to prevent illegal deforestation and halt animal poaching.

Rainforest Connection CEO Topher White with one of his devices.

In addition to those benefits, there’s another important one according to respondents: leaner teams. While teams of 11 or more tend to deliver production systems in six months or under, most teams numbering fewer than 10 developers are easily hitting year-and-under production schedules.

Vast range of use cases being explored professionally and for hobbyist projects

Choosing ML models and hardware

Earlier we mentioned that many developers are building PoCs without hardware constraints, but what happens deeper into development? How do they handle model and hardware alignment?

The survey results also indicated that half the teams decide on hardware direction before choosing their ML model, while another third choose their model at the same time they decide on hardware. Smaller companies tend to be more flexible in the timing of their hardware selection, while larger teams tend to defer their hardware decision to either the point at which the ML is chosen or after.

Public or company-developed datasets are most popular for model development.

As part of that model development, teams use multiple approaches to obtain data. Teams often favor either using public datasets (45 percent) or an existing data set their company has compiled (40 percent), but they also lean toward developing new data sets (55 percent). They also tend to leverage efficiencies by using existing models (40 percent), or pre-trained models (24 percent), although a third of the respondents are developing models from scratch, according to the survey.

Approaches to obtaining data and models

Training ML models

When it comes to training those models, nearly two-thirds of developers have a positive attitude about leveraging the cloud. They can, for example, develop solutions entirely in the cloud, using accurate SoC models to build and test software before and after silicon and hardware availability.

The attractiveness of the cloud training extends to endpoint AI models; however, developers also favor leveraging local GPUs, CPUs, edge devices and MCUs to train their models, and those models generally are updated within the first six months of deployment.

Developers are drawn to the features of devices such as Arm Cortex-M4, M7 and M0+ running on convolutional neural networks (CNNs), recurrent neural networks (RNNs) and multilayer perceptron networks (MLP). They tended to leverage TensorFlow, Keras and PyTorch for training frameworks and TensorFlow Lite for inference frameworks.

Microcontrollers such as the Arm Cortex-M family are an ideal platform for ML because they’re already ubiquitous. They perform real-time calculations quickly and efficiently, so they’re reliable and responsive, and because they use very little power, can be deployed in places where replacing the battery is difficult or inconvenient.

Perhaps even more importantly, they’re cheap enough to be used just about anywhere. The market analyst IDC reports that 28.1 billion microcontrollers were sold in 2018, with that annual shipment volume growing to 38.2 billion by 2023.

As this technology is pushed to the edge and endpoint it will open new and exciting applications, offering designers an opportunity to breathe new life into their product offerings.

Kevin Gordon, VP of AI Technologies, NexOptic

Challenges and opportunities

Challenges

Any developing technological trend is not without its early challenges, and endpoint AI is no exception. Optimizing the system’s crown jewel – the ML model – looms as the biggest challenge now. In the past, it’s been seen as a time-consuming process of trial-and-error.

Within this, the nature of tinyML
devices comes to the fore:

Optimizing models for power, size and latency at the endpoint is the biggest challenge (39 percent list it as their chief challenge)
The next-closest concerns are training and experimenting with the ML model (24 percent)
And data collection (23 percent).

After, that numerous challenges concern developers from production deployment to firmware to integration testing and more.

ML model optimization

So, what about model optimization is tricky? Hardware alignment. The biggest concern that pops up is determining the best match of model and hardware (29 percent). However, right behind that are measuring power usage, picking the right NN architecture, integrating the right tools for performance optimization and optimizing operation implementations.

A possible solution for these challenges is the usage of cloud-native tooling that gives access to virtual models. Keil Studio Cloud is the next-generation development tool from Arm that runs in your browser. It supports all devices covered by CMSIS-Packs (over 9,500) and is being enabled with access to Arm Virtual Hardware that offers ML developers CPU models to test drive their networks in the cloud for rapid software engineering.

Summary: Next-level tinyML

The potential use cases of tinyML are almost unlimited. Developers are already working with tinyML to explore all sorts of new ideas: responsive traffic lights that change signalling to reduce congestion, industrial machines that can predict when they’ll need service, sensors that can monitor crops for the presence of damaging insects, store shelves that can request restocking when inventory gets low, healthcare monitors that track vitals while maintaining privacy. The list goes on.

TinyML can make endpoint devices more consistent and reliable, since there’s less need to rely on busy, crowded internet connections to send data back and forth to the cloud. Reducing or even eliminating interactions with the cloud has major benefits including reduced energy use, significantly reduced latency in processing data and security benefits, since data that doesn’t travel is far less exposed to attack.

It’s worth nothing that these tinyML models, which perform inference on the microcontroller, aren’t intended to replace the more sophisticated inference that currently happens in the cloud. What they do instead is bring specific capabilities down from the cloud to the endpoint device. That way, developers can save cloud interactions for if and when they’re needed.

TinyML also gives developers a powerful new set of tools for solving problems. ML makes it possible to detect complex events that rule-based systems struggle to identify, so endpoint AI devices can start contributing in new ways. Also, since ML makes it possible to control devices with words or gestures, instead of buttons or a smartphone, endpoint devices can be built more rugged and deployable in more challenging operating environments.

Industry players have been quick to recognize the value of tinyML and have moved rapidly to create a supportive ecosystem. Developers at every level, from enthusiastic hobbyists to experienced professionals, can now access tools that make it easy to get started and more quickly sparked innovations for the future.

Check out our latest resources: