Categories
Embedded IoT

5 Reasons Why I Dread Writing Embedded GUIs

A consequence of the massive adoption of Internet of Things technologies across all industries is an increasing need for embedded development skills. Yet, embedded development has historically been a pretty complex domain, and not something that one can add to their skillset overnight.

Luckily, over the last decade, silicon vendors have put a lot of effort into simplifying embedded development, especially for people with little to no experience in the domain. Communities such as Arduino and PlatformIO have also immensely contributed to providing easy-to-use tools and high-level libraries that can hide most of the scary details—yes, assembly, I’m looking at you!—of embedded programming while still allowing for professional applications to be written.

In my experience though, there is at least one area where things remain overly cumbersome: graphical user interface (GUI) development. Many applications require at least some kind of graphical user interface: the display might be small and monochrome, with hardly any buttons for the user to press, but it’s still a UI, eh?

I am sure many of you will relate: GUI development can be a lot of fun… until it isn’t!

In this article, I have compiled 5 reasons why I tend to not enjoy writing embedded GUI code so much anymore. And since you might not be interested in simply reading a rant, I am also sharing some tips and some of the tools I use to help keep GUI development enjoyable 🙂.

Hardware Integration & Portability

Most display devices out there come with sample code and drivers that will give you a head start in being able to at least display something.

But there is more to a GUI than just a screen, as an Interface is also made up of inputs, right? How about those push buttons, touch screen inputs, and other sensors in your system that may all participate in your interactions?

Image credit: Seeed Studio.

It might not seem like much, but properly handling simple hardware inputs such as buttons being pressed can be a lot of work when running on a constrained system, and you can quickly end up having to deal with complex timing or interrupt-management issues (see Event Management and “super-loop” section below for more). And as these often involve low-level programming, they tend to be pretty hardware-dependent and not easily portable.

A lot of embedded development is done using C, so, except for low-level bootstrapping code, embedded code can in theory be fairly portable. However, writing portable GUI code is a whole different story and unless you’re building on top of an existing framework such as LVGL or Azure RTOS GUIX, it requires a lot of effort to abstract all the hardware dependencies, even more so when trying to keep the performance optimal.

Of course, it is not always necessary (or possible) to have GUI code that’s 100% portable. However, in these times of global chip shortage, it can prove very handy to not have a hard dependency on a specific kind of micro-controller or LCD display.

Memory Management

Like I mentioned in the introduction, manipulating pixels can be a lot of fun—it really is! However, constrained systems have limited amounts of memory, and those pixels that you’re manipulating in your code can quickly add up to thousands of bytes of precious RAM and Flash storage.

Let’s take the example of a tiny 128×64 pixels monochrome display. As the screen only supports black & white, each pixel can be represented in memory using just a single bit, meaning a byte can hold up to 8 pixels—yay! But if you do the maths:

A 32x32px bitmap uses 128 bytes of memory!

That’s already 1KB of RAM, which is quite significant if your MCU only has, say, 16KB total. Interested in displaying a handful of 32×32px icons? That will be an additional 128 bytes for each of these tiny icons!

In short: your graphical user interface will likely eat up a lot of your memory, and you need to be extra clever to leave enough room for your actual application. As an example, a quick way to save on graphics memory is to double-check whether some of your raster graphics (ex. icons) can be replaced by a vector equivalent: surely it takes a lot less code and RAM to directly draw a simple 32x32px red square on the screen, instead of having it stored as a bitmap in memory.

Resource Management

It can be tricky to properly manage the various resources that make up a GUI project.

More specifically, and whether you are lucky enough to work with a graphics designer or not, your GUI mockups will likely consist of a variety of image files, icons, fonts, etc. However, in an embedded context, you typically can’t expect to be able to directly manipulate that nice transparent PNG file or TrueType font in your code! It first needs to be converted in a format that allows it to be manipulated in your embedded code.

If you are a seasoned embedded developer, I am sure that you (or someone at your company) have probably developed your very own macros and tools to help you streamline the conversion/optimization of your binary assets, but in my experience it’s always been a lot of tinkering, and one-off, quick and dirty, conversion scripts, which impact long-term maintainability. Add version control to the mix, and it becomes pretty hairy to keep your assets tidy at all times!

Event Handling & Performance

Graphical programming is by nature very much event-driven. It is therefore quite natural to expect embedded GUI code to look as follows (pseudo-code):

Button btnOK;
btnOK.onClick = btnOK_click;

btnOK_click = function() {
  // handle click on button btnOK
  // ...
}

As you can imagine, things are not always that simple…

Firstly, C, which is still the most used embedded programming language, is not exactly object-oriented. As a consequence, even if it’s perfectly possible to aim for a high-level API that looks like the above, there is a good chance you will find yourself juggling with error-prone function pointers whenever adding/updating an event handler to one of your UI elements.

Assuming that you have indeed found an elegant way to associate event handlers to the various pieces of your UI, you still need to implement some kind of event loop. Indeed, you must regularly process the events happening in your system (“button A pressed”, etc.) and dispatch them to the proper event handlers. A common pattern in embedded programming consists in doing so through a so-called super loop: the program is running an infinite loop that invokes each task the system needs to perform, ad infinitum.

int main() {
    setup();
    
    while (1) {
        read_sensors();
        refresh_ui();
        // etc.
    }
    
    /* program's execution will never reach here */
    return 0;
}

A benefit of this approach is that the execution flow remains pretty readable and straightforward, and it also avoids some potential headaches that may be induced by complex multi-threading or interrupt handling. However, any event handler running for too long (or crashing!) can compromise the performance and stability of your main application.

As embedded real-time operating systems such as FreeRTOS, or Azure RTOS ThreadX are becoming more popular, a more modern approach is to have the UI event loop run in a dedicated background task. The operating system can therefore ensure that this task, given its lower priority, will not compromise the performance of your main application.

An embedded GUI does not always need to be performant as in fast and responsive. However, it is considered a good practice to use embedded resources as efficiently as possible. Making sure that your GUI & app code are as performant as reasonably possible can potentially save you a lot of money as it means you can stick to using the smallest possible MCU for the task.

Tooling

Last but not least: tooling. To be honest, I have never been a big fan of designing graphical user interfaces using a WYSIWYG (What You See Is What You Get) approach. That being said, coding a graphical user interface that has more than just a couple of screens requires at least some tooling support, since most of the “boring” glue code can often be automatically generated.

What’s more, testing an embedded GUI can quickly become painful, as re-compiling your app and downloading it to your target can take quite some time. This can be very frustrating when you need to wait minutes to e.g test how that new fancy animation you coded looks like. 😒

Over the past few months, I have started to use Renode more often. It is a pretty complete open-source tool suite for emulating embedded hardware—including their display!—straight from your computer. In a future post, I plan on sharing more on how I started to use Renode for drastically shortening the “inner embedded development loop”, i.e. the time between making a code change and being able to test it live on your—emulated!—device.


I would be curious to hear about your experience, and your pain points when working with embedded GUIs. Let me know in the comments below!

Like I already mentioned, stay tuned for upcoming articles where I will be covering some of the tools and frameworks that I have started to use (and love) and that make my life SO much easier when it comes to GUI development!

If you enjoyed this article, don’t forget to subscribe to be notified of upcoming publications! And of course, you can also always find me on Twitter.

Categories
AI/ML IoT

How I Built a Connected Artificial Nose (and How You Can Too!)

Over the past few months, I have worked on a pretty cool project that some of you might have already heard about as it sort of went viral. I built a DIY, general-purpose, artificial nose that can smell virtually anything you teach it to recognize!

The artificial nose in action, smelling coffee ☕ and whiskey 🥃.
SeeedStudio’s Multichannel Gas Sensor v2.

It is powered by the Wio Terminal (an Arduino-compatible prototyping platform), a super affordable electronic gas sensor, and a TinyML neural network that I trained using the free online tool Edge Impulse.

Cover of Make: Magazine Vol. 77.

The project was recently featured on the cover of Make: Magazine, and I encourage you to check out the article I wrote for them before reading further.

The Make: Magazine article covers a lot about how you can build the artificial nose for yourself, so I want to use this blog post to dive deeper into why this project is so important to me. In particular, I want to share with you how it helped me understand more about AI than I’d ever thought, and how I eventually ended up connecting the “nose” to an IoT platform (namely, Azure IoT).

Making Neural Networks Tangible

Despite being passionate about all things software, Machine Learning (ML) has always been a field that’s eluded me, perhaps because it tends to be too abstract and too much maths for my visual brain?

Sample images from the MNIST test dataset.

Speaking of visual things, every time I have tried to open a book promising to be an introduction to ML, most of the introductory examples involved image classification (ex. automatically recognizing handwritten digits from the MNIST database). And, sadly, those innocent pixels would be anything but visual to me, as they would quickly turn into abstract matrices.

So when I started to think of implementing an artificial nose, I didn’t initially approach it as a Machine Learning problem. Instead, I tried to use my intuition: “What characterizes a smell?”. And my intuition was telling me that somehow I needed to establish a correlation between the concentration of the various gasses measured by the gas sensor (carbon monoxide, ethyl alcohol, etc.), and the associated smell. However, doing a simple read of the gasses concentration at a given point in time would probably not cut it: how would it make the difference between a really strong alcohol smell, and one that was maybe more volatile?

Quickly, I realized that acquiring a couple seconds of sensor data would probably be just enough to “capture” the olfactory fingerprint of each smell. With these few seconds of sensor data, I could look at the variation (min, max, average, etc.) of the concentration of each gas, and this would hopefully characterize each smell.

It turns out that once I had extracted those characteristics—something that I can now refer to as feature extraction, like the AI grown-ups, and which was really easy to do using the Edge Impulse tool suite—all that was left was to effectively establish the correlation between them and the expected smells. However, I didn’t really know what kind of neural network architecture I would need, let alone what a neural network was anyway. So, once again, I leveraged the Edge Impulse environment.

It turns out the kind of classification problem I was looking at was reasonably simple: given the minimum/maximum/average/… concentration of each gas on a given time period (I found 1.5s to be the sweet spot), what is the predicted smell? And one simple way to “solve” that equation, is to use a so-called fully-connected neural network, like you see below.

During the training phase, the training data represents the ground truth (ex. “This is 100% coffee!”) and is used to tweak the parameters of the equation—the weights of the neurons—based on how much each characteristic (ex. the average concentration of NO2) accrues to each smell.

Once the model has been trained, and during the inference phase, a given input/olfactory fingerprint entering the network (left-hand side of the diagram), ends up being “routed” to the appropriate output bucket (right-hand side). effectively giving a prediction about what smell it corresponds to.

Building an actual nose

When I initially shared my project on social media back in May last year, I quickly realized lots of people were interested in it.

This motivated me to go further and to turn my initial prototype into an actual nose! I had never done that before, so I ended up teaching myself how to use 3D CAD software so that I could design an actual enclosure for my device. I picked Blender—which I would not recommend for pure CAD stuff as there are better alternatives out there, ex. TinkerCAD—, and 3D-printed the resulting plastic enclosure.

A screen capture from the thingiverse.com website titled "Artificial Nose Enclosure" that show a blue 3D rendering of a nose.
The Nose Enclosure on Thingiverse.

Turning the nose into an IoT device

An interesting aspect of TinyML is that it enables scenarios where your low-power, constrained, microcontroller-based equipment is completely autonomous when it comes to performing machine learning inference (ex. guessing a smell). It is very powerful, as it means your sensor data never has to leave your device and you don’t need to rely on any sort of cloud-based AI service. But on the other hand, it also means that your smart device might not be so smart if it ends up living in its own echo chamber, right?

At the heart of an IoT solution is often the “thing” itself, and it makes a lot of sense to design it to be as smart as possible for there are many reasons why relying on any form of network communication or cloud-based processing is at best impractical, sometimes plain impossible.

Connecting the Artificial Nose to Azure IoT Central

The Artificial Nose is effectively an IoT Plug and Play device.

As soon I was happy with how it performed at smelling things, and once I had completed the development of the graphical user interface, I did use the Azure IoT SDK (and some of the work I had done last year) to enable the nose to talk to the Azure IoT services.

It means you can very easily connect the device to Azure IoT Central (using the Wio Terminal’s Wi-Fi module), and get access to gas sensor data telemetry in near-realtime, see what the device is smelling, etc.

More importantly, you can automatically trigger rules when, for example, a bad smell is being detected, therefore allowing the nose to be much smarter than if it were just a standalone, offline, device.

Connecting the Artificial Nose to Azure IoT Central – Real-time telemetry.

If you built the artificial nose for yourself—and I hope many of you will consider doing so!—here are the simple steps for you to connect it to Azure IoT Central:

  • First, make sure that your Wio Terminal is running an up-to-date WiFi firmware by following these instructions ;
  • Create a new Azure IoT Central application (if you already have one you want to use, that works too!) ;
    • In the Administration section of the IoT Central application, look for the Device Connection menu.
    • Open the SAS-IoT-Devices enrollment group and take note of the following credentials that you will need to connect your AI nose(s):
      • ID Scope
      • SAS Primary Key
  • Flash the Wio Terminal with the latest Artificial Nose firmware (or deploy your own custom build) ;
  • While the Wio Terminal is powered, keep the three buttons (A, B, C) at the top pressed, and slide the reset button. The device should now be showing a black screen ;
  • Connect to the Wio Terminal over serial and check that it’s running the configuration prompt by typing help, which should show you the list of supported commands. Type the following commands to configure the WiFi connection and the Azure IoT credentials
    • set_wifissid <your_wifi_ssid>
    • set_wifipwd <your_wifi_password>
    • set_az_iotc <id_scope> <sas_primary_key> <device_id> (id_scope and sas_primary_key as per earlier, and device_id being the ID you want to give your device in Azure IoT Central)
  • Reset the Wio Terminal, and voila! You should now see a new device popping up in the Devices section of your IoT Central application.

Digital Twins meet virtual senses

Like I mentioned above, having the nose talking to an IoT platform enables scenarios where e.g. you trigger an alert when a bad smell is being picked up. But what is a bad smell anyway? This might depend on a lot of different factors, just like the final destination for the actual alert might be highly dynamic.

Let me try to illustrate this with an example of a real estate cleaning company in charge of buildings all around the city of Chicago. Their information system already allows them to keep track of their personnel and associated cleaning schedules, but in a pretty static way: cleaning people are going to their assigned location once a day, no matter what. From time to time, it turns out that the location doesn’t really require urgent cleaning (hello, COVID-19 and slow office spaces!), in which case the cleaning staff would have been better off going to a place that actually required servicing.

Beyond the apparent buzzword, the concept of Digital Twins consists in nothing more than augmenting the information system (staff directory, building inventory, cleaning schedules, etc.) and overall knowledge graph of the cleaning company with entities that correspond to physical, connected, assets.

With that in mind, a mere “it doesn’t smell so good in here” signal sent by a sniffing device sitting in an office building can immediately be contextualized, and appropriate actions can be taken. Based on where the device is effectively located, it becomes easy to figure out who is the person responsible for cleaning that space on that particular day, and to notify them accordingly.

Connecting the Artificial Nose to a Digital Twins environment.

Get started today!

Many people have already started to build the device for themselves and to experiment what adding “virtual smell” to their devices and applications could mean. If this blog post inspired you to join them, I will leave you with the only two links that you really need to get started:

TinyML powered Artificial Nose Project kit with Wio Terminal
Dark Mode

artificial-nose (this link opens in a new window) by kartben (this link opens in a new window)

Instructions, source code, and misc. resources needed for building a Tiny ML-powered artificial nose.

If you enjoyed this article, don’t forget to subscribe to be notified of upcoming publications! And of course, you can also always find me on Twitter.

Categories
IoT

Deploying a LoRaWAN network server on Azure

There is something oddly fascinating about radio waves, radio communications, and the sheer amount of innovations they’ve enabled since the end of the 19th century.

What I find even more fascinating is that it is now very easy for anyone to get hands-on experience with radio technologies such as LPWAN (Low-Power Wide Area Network, a technology that allows connecting pieces of equipment over a low-power, long-range, secure radio network) in the context of building connected products.

A portrait of Heinrich Rudolf Hertz

It’s of no use whatsoever […] this is just an experiment that proves Maestro Maxwell was right—we just have these mysterious electromagnetic waves that we cannot see with the naked eye. But they are there.

— Heinrich Hertz, about the practical importance of his radio wave experiments

A picture of a LoRaWAN indoor gateway, featuring 3 antennas.
LG02 Indoor LoRaWAN Gateway from Dragino, available for less than $100.

Nowadays, not only is there a wide variety of hardware developer kits, gateways, and radio modules to help you with the hardware/radio aspect of LPWAN radio communications, but there is also open-source software that allows you to build and operate your very own network. Read on as I will be giving you some insights into what it takes to set up a full-blown LoRaWAN network server in the cloud!

A quick refresher on LoRaWAN

LoRaWAN is a low-power wide-area network (LPWAN) technology that uses the LoRa radio protocol to allow long-range transmissions between IoT devices and the Internet. LoRa itself uses a form of chirp spread spectrum modulation which, combined with error correction techniques, allows for very high link budgets—in other terms: the ability to cover very long ranges!

Data sent by LoRaWAN end devices gets picked up by gateways nearby and is then routed to a so-called network server. The network server de-duplicates packets (several gateways may have “seen” and forwarded the same radio packet), performs security checks, and eventually routes the information to its actual destination, i.e. the application the devices are sending data to.

A round, gray, sensor with a label that says "BOSCH" imprinted on it.
Bosch’s Smart Parking Lot sensor.

LoRaWAN end nodes are usually pretty “dumb”, battery-powered, devices (ex. soil moisture sensor, parking occupancy, …), that have very limited knowledge of their radio environment. For example, a node may be in close proximity to a gateway, and yet transmit radio packets with much more transmission power than necessary, wasting precious battery energy in the process. Therefore, one of the duties of a LoRaWAN network server is to consolidate various metrics collected from the field gateways to optimize the network. If a gateway is telling the network server it is getting a really strong signal from a sensor, it might make sense to send a downlink packet to that device so that it can try using slightly less power for future transmissions.

As LoRa uses an unlicensed spectrum and granted one follows their local radio regulations, anyone can freely connect LoRa devices, or even operate their own network.

My private LoRaWAN server, why?

The LoRaWAN specification puts a really strong focus on security, and by no means do I want to make you think that rolling out your own networking infrastructure is mandatory to make your LoRaWAN solution secure. In fact, LoRaWAN has a pretty elegant way of securing communications, while keeping the protocol lightweight. There is a lot of literature on the topic that I encourage you to read but, in a nutshell, the protocol makes it almost impossible for malicious actors to impersonate your devices (messages are signed and protected against replay attacks) or access your data (your application data is seen by the network server as an opaque, ciphered, payload).

So why should you bother about rolling your ow LoRaWAN network server anyway?

Coverage where you need it

In most cases, relying on a public network operator means being dependant on their coverage. While some operators might allow a hybrid model where you can attach your own gateways to their network, and hence extend the coverage right where you need it, oftentimes you don’t get to decide how well a particular geographical area will be covered by a given operator.

When rolling out your own network server, you end up managing your own fleet of gateways, bringing you more flexibility in terms of coverage, network redundancy, etc.

Data ownership

While operating your own server will not necessarily add a lot in terms of pure security (after all, your LoRaWAN packets are hanging in the open air a good chunk of their lifetime anyway!), being your own operator definitely brings you more flexibility to know and control what happens to your data once it’s reached the Internet.

What about the downsides?

It goes without saying that operating your network is no small feat, and you should obviously do your due diligence with regards to the potential challenges, risks, and costs associated with keeping your network up and running.

Anyway, it is now high time I tell you how you’d go about rolling out your own LoRaWAN network, right?

The Things Stack on Azure

The Things Stack is an open-source LoRaWAN network server that supports all versions of the LoRaWAN specification and operation modes. It is actively being maintained by The Things Industries and is the underlying core of their commercial offerings.

A typical/minimal deployment of The Things Stack network server relies on roughly three pillars:

  • A Redis in-memory data store for supporting the operation of the network ;
  • An SQL database (PostgreSQL or CockroachDB are supported) for storing information regarding the gateways, devices, and users of thje network ;
  • The actual stack, running the different services that power the web console, the network server itself, etc.

The deployment model recommended for someone interested in quickly testing out The Things Stack is to use their Docker Compose configuration. It fires up all the services mentioned above as Docker containers on the same machine. Pretty cool for testing, but not so much for a production environment: who is going to keep those Redis and PostgreSQL services available 24/7, properly backed up, etc.?

I have put together a set of instructions and a deployment template that aim at showing how a LoRaWAN server based on The Things Stack and running in Azure could look like.

The Things Stack running on Azure - Deployment diagram.
The Things Stack running on Azure – Deployment diagram.

The instructions in the GitHub repository linked below should be all you need to get your very own server up and running!

In fact, you only have a handful of parameters to tweak (what fancy nickname to give your server, credentials for the admin user, …) and the deployment template will do the rest!

OK, I deployed my network server in Azure, now what?

Just to enumerate a few, here are some of the things that having your own network server, running in your own Azure subscription, will enable. Some will sound oddly specific if you don’t have a lot of experience with LoRaWAN yet, but they are important nevertheless. You can:

  • benefit from managed Redis and PostgreSQL services, and not have to worry about potential security fixes that would need to be rolled out, or about performing regular backups, etc. ;
  • control what LoRaWAN gateways can connect to your network server, as you can tweak your Network Security Group to only allow specific IPs to connect to the UDP packet forwarder endpoint of your network server ;
  • completely isolate the internals of your network server from the public Internet (including the Application Server if you which so), putting you in a better position to control and secure your business data ;
  • scale your infrastructure up or down as the size and complexity of the fleet that you are managing evolves ;
  • … and there is probably so much more. I’m actually curious to hear in the comments below about other benefits (or downsides, for that matter) you’d see.

I started to put together an FAQ in the GitHub repository so, hopefully, your most obvious questions are already answered there. However, there is one that I thought was worth calling out in this post, which is: How big of a fleet can I connect?.

It turns out that even a reasonably small VM like the one used in the deployment template—2 vCPUs, 4GB of RAM—can already handle thousands of nodes, and hundreds of gateways. You may find this LoRaWAN traffic simulation tool that I wrote helpful in case you’d want to conduct your own stress testing experiments.

What’s next?

You should definitely expect more from me when it comes to other LoRaWAN related articles in the future. From leveraging DTDL for simplifying end application development and interoperability with other solutions, to integrating with Azure IoT services, there’s definitely a lot more to cover. Stay tuned, and please let me know in the comments of other related topics you’d like to see covered!