Categories
IoT

Deploying a LoRaWAN network server on Azure

There is something oddly fascinating about radio waves, radio communications, and the sheer amount of innovations they’ve enabled since the end of the 19th century.

What I find even more fascinating is that it is now very easy for anyone to get hands-on experience with radio technologies such as LPWAN (Low-Power Wide Area Network, a technology that allows connecting pieces of equipment over a low-power, long-range, secure radio network) in the context of building connected products.

A portrait of Heinrich Rudolf Hertz

It’s of no use whatsoever […] this is just an experiment that proves Maestro Maxwell was right—we just have these mysterious electromagnetic waves that we cannot see with the naked eye. But they are there.

— Heinrich Hertz, about the practical importance of his radio wave experiments

LG02 Indoor LoRaWAN Gateway from Dragino, available for less than $100.

Nowadays, not only is there a wide variety of hardware developer kits, gateways, and radio modules to help you with the hardware/radio aspect of LPWAN radio communications, but there is also open-source software that allows you to build and operate your very own network. Read on as I will be giving you some insights into what it takes to set up a full-blown LoRaWAN network server in the cloud!

A quick refresher on LoRaWAN

LoRaWAN is a low-power wide-area network (LPWAN) technology that uses the LoRa radio protocol to allow long-range transmissions between IoT devices and the Internet. LoRa itself uses a form of chirp spread spectrum modulation which, combined with error correction techniques, allows for very high link budgets—in other terms: the ability to cover very long ranges!

Data sent by LoRaWAN end devices gets picked up by gateways nearby and is then routed to a so-called network server. The network server de-duplicates packets (several gateways may have “seen” and forwarded the same radio packet), performs security checks, and eventually routes the information to its actual destination, i.e. the application the devices are sending data to.

Bosch’s Smart Parking Lot sensor.

LoRaWAN end nodes are usually pretty “dumb”, battery-powered, devices (ex. soil moisture sensor, parking occupancy, …), that have very limited knowledge of their radio environment. For example, a node may be in close proximity to a gateway, and yet transmit radio packets with much more transmission power than necessary, wasting precious battery energy in the process. Therefore, one of the duties of a LoRaWAN network server is to consolidate various metrics collected from the field gateways to optimize the network. If a gateway is telling the network server it is getting a really strong signal from a sensor, it might make sense to send a downlink packet to that device so that it can try using slightly less power for future transmissions.

As LoRa uses an unlicensed spectrum and granted one follows their local radio regulations, anyone can freely connect LoRa devices, or even operate their own network.

My private LoRaWAN server, why?

The LoRaWAN specification puts a really strong focus on security, and by no means do I want to make you think that rolling out your own networking infrastructure is mandatory to make your LoRaWAN solution secure. In fact, LoRaWAN has a pretty elegant way of securing communications, while keeping the protocol lightweight. There is a lot of literature on the topic that I encourage you to read but, in a nutshell, the protocol makes it almost impossible for malicious actors to impersonate your devices (messages are signed and protected against replay attacks) or access your data (your application data is seen by the network server as an opaque, ciphered, payload).

So why should you bother about rolling your ow LoRaWAN network server anyway?

Coverage where you need it

In most cases, relying on a public network operator means being dependant on their coverage. While some operators might allow a hybrid model where you can attach your own gateways to their network, and hence extend the coverage right where you need it, oftentimes you don’t get to decide how well a particular geographical area will be covered by a given operator.

When rolling out your own network server, you end up managing your own fleet of gateways, bringing you more flexibility in terms of coverage, network redundancy, etc.

Data ownership

While operating your own server will not necessarily add a lot in terms of pure security (after all, your LoRaWAN packets are hanging in the open air a good chunk of their lifetime anyway!), being your own operator definitely brings you more flexibility to know and control what happens to your data once it’s reached the Internet.

What about the downsides?

It goes without saying that operating your network is no small feat, and you should obviously do your due diligence with regards to the potential challenges, risks, and costs associated with keeping your network up and running.

Anyway, it is now high time I tell you how you’d go about rolling out your own LoRaWAN network, right?

The Things Stack on Azure

The Things Stack is an open-source LoRaWAN network server that supports all versions of the LoRaWAN specification and operation modes. It is actively being maintained by The Things Industries and is the underlying core of their commercial offerings.

A typical/minimal deployment of The Things Stack network server relies on roughly three pillars:

  • A Redis in-memory data store for supporting the operation of the network ;
  • An SQL database (PostgreSQL or CockroachDB are supported) for storing information regarding the gateways, devices, and users of thje network ;
  • The actual stack, running the different services that power the web console, the network server itself, etc.

The deployment model recommended for someone interested in quickly testing out The Things Stack is to use their Docker Compose configuration. It fires up all the services mentioned above as Docker containers on the same machine. Pretty cool for testing, but not so much for a production environment: who is going to keep those Redis and PostgreSQL services available 24/7, properly backed up, etc.?

I have put together a set of instructions and a deployment template that aim at showing how a LoRaWAN server based on The Things Stack and running in Azure could look like.

The Things Stack running on Azure – Deployment diagram.

The instructions in the GitHub repository linked below should be all you need to get your very own server up and running!

In fact, you only have a handful of parameters to tweak (what fancy nickname to give your server, credentials for the admin user, …) and the deployment template will do the rest!

OK, I deployed my network server in Azure, now what?

Just to enumerate a few, here are some of the things that having your own network server, running in your own Azure subscription, will enable. Some will sound oddly specific if you don’t have a lot of experience with LoRaWAN yet, but they are important nevertheless. You can:

  • benefit from managed Redis and PostgreSQL services, and not have to worry about potential security fixes that would need to be rolled out, or about performing regular backups, etc. ;
  • control what LoRaWAN gateways can connect to your network server, as you can tweak your Network Security Group to only allow specific IPs to connect to the UDP packet forwarder endpoint of your network server ;
  • completely isolate the internals of your network server from the public Internet (including the Application Server if you which so), putting you in a better position to control and secure your business data ;
  • scale your infrastructure up or down as the size and complexity of the fleet that you are managing evolves ;
  • … and there is probably so much more. I’m actually curious to hear in the comments below about other benefits (or downsides, for that matter) you’d see.

I started to put together an FAQ in the GitHub repository so, hopefully, your most obvious questions are already answered there. However, there is one that I thought was worth calling out in this post, which is: How big of a fleet can I connect?.

It turns out that even a reasonably small VM like the one used in the deployment template—2 vCPUs, 4GB of RAM—can already handle thousands of nodes, and hundreds of gateways. You may find this LoRaWAN traffic simulation tool that I wrote helpful in case you’d want to conduct your own stress testing experiments.

What’s next?

You should definitely expect more from me when it comes to other LoRaWAN related articles in the future. From leveraging DTDL for simplifying end application development and interoperability with other solutions, to integrating with Azure IoT services, there’s definitely a lot more to cover. Stay tuned, and please let me know in the comments of other related topics you’d like to see covered!

Categories
Eclipse

How many lines of open source code are hosted at the Eclipse Foundation?

Spoiler alert: 162 million!

That’s right, as of August 1st, there are 330 active open-source projects hosted at the Eclipse Foundation and if you look across the 1120 Git repositories that this represents, you will find over 162 million physical source lines of code. But beyond this number, let’s look at how it was obtained, and what it really means.

I’ve blogged several times about the importance of using metrics to monitor the health (and hopefully, growth!) of an open source project/community, and lines of code are just one. You should always have other metrics on your radar like the number of contributors, diversity, etc.

There are many ways, and many tools available out there, to count source lines of code. Openhub (previously known as ohloh) used to be a really good tool, but it doesn’t seem to be actively maintained. For a few years now, I’ve been relying on a home-made script to analyze Eclipse IoT projects, and it’s only recently that I realized I should probably run it against the entire eclipse.org codebase!

In this blog post, I will briefly talk about how the aforementioned script works, why you should make sure to take these metrics with a pinch of salt and finally, go through some noteworthy findings.

Line counting process

The script used to count the number of lines of code is available on Github. It takes a list of Eclipse projects’ identifiers (e.g ‘iot.paho’) and a given time range as an input and outputs a consolidated CSV file.

The main script (main.js) uses the Eclipse Project Management Infrastructure (PMI) API to retrieve the list of Git repositories for the requested projects and then proceeds to clone the repos and run the cloc command-line tool against each repo. The script also allows computing the statistics for a given time period, in which case it looks at the state of each repository at the beginning of each month for that period.

Once the main script has completed (and it can obviously take quite some time), thecsv-concat.js script can be used to consolidate all the produced metric files into one single CSV file that will contain the detailed breakout of lines of code per project and per programming language, the affiliation of the project to a particular top-level projects, the number of blanks or comment lines, etc.. It is pretty easy to then feed this CSV into Excel or Google Spreadsheets, and use it as the source for building pivot tables for specific breakouts.

Caveats

Just like virtually any KPI, you want to take the number of lines of code in your project with a grain of salt. Here are a few things to keep in mind:

All lines of code are not created equal

There is an incredible diversity of projects at Eclipse, and while a majority is using Java as their main programming language, there’s also a lot of C, C++, Python, Javascript, … 10M lines of Java code probably don’t carry the same value (i.e. how much effort has been needed to produce them) as 10M lines of C code.

Trends are more important than snapshots

It is nice to know that as of today there are 162 million lines of code in the Eclipse repositories, but it is, in my opinion, more important to look at trends over time. Is a particular programming language becoming more popular? Are all the top-level projects equally active?

I didn’t have a chance to run the scripts for a longer time period yet, but I will make sure to share the results when I get a chance!

Generated code, should it count?

There is a fair amount of generated code in some projects (in the Modeling top-level project in particular, of course), which certainly accounts for a few million lines of code. However, generated code often is customized, so I think it doesn’t necessarily skew the numbers as much as one would think.

Development does not always happen in a single branch

My script just looks at the code stored in the main (HEAD) branch of the Git repository. Some projects may have more than one development stream and may e.g. have a “develop” branch that is ahead of the main stable branch. Therefore, there is very likely more code in our repositories than what this quick analysis shows.

Additional findings

As my script outputs pretty detailed statistics, it is interesting to have a quick look at e.g. how the different top-level projects and programming languages compare.

Top 3 top-level projects: Runtime, Technology & Modeling

Top-level projectPhysical SLOC
rt54,961,728
technology28,887,621
modeling27,140,344
tools14,214,182
webtools9,651,900
eclipse6,401,518
ee4j5,809,126
ecd3,114,768
polarsys3,105,229
iot2,930,217
birt2,235,624
science1,670,051
datatools939,424
mylyn767,652
soa752,774

Top programming language: Java

Programming languagePhysical SLOC
Java72,349,870
HTML61,119,106
XML7,543,689
ANTLR Grammar3,161,339
JSON2,313,556
JavaScript2,251,418
C++2,245,759
C1,446,013
XMI1,355,914
C/C++ Header1,019,368
TTCN923,098
Maven884,271
CSS805,073
Assembly717,771
XSD688,764
PHP459,237
Python316,553
Markdown304,421
XSLT256,857
Scala229,560
Bourne Shell214,142
Go184,306
SWIG152,062
JSP142,190
Gencat NLS125,251
Ant113,133
TypeScript108,217
AsciiDoc105,552
Windows Module Definition64,843
TITAN Project File Information64,014
Groovy55,261
Sass53,915
XQuery51,432
XHTML51,166
DTD51,052
make48,021
Perl43,643
DITA42,526
yacc39,876
TeX36,400
m434,438
AspectJ33,717
Ruby28,355
Scheme27,484
YAML26,348
CMake25,182
Lua23,646
LESS18,712
SQL16,070
Cucumber15,454
IDL12,564
INI12,171
Bourne Again Shell11,978
Pascal11,915
lex11,795
DOS Batch11,675
Windows Resource File10,278
Blade8,295
C#7,983
Tcl/Tk7,611
Stylus7,477
Fortran 907,211
ERB7,048
Vuejs Component6,281
Visualforce Component5,047
MSBuild script4,538
Freemarker Template4,077
Dockerfile3,696
Velocity Template Language3,649
awk3,068
Rust2,903
Qt2,772
CUDA2,533
Puppet2,084
diff1,880
Haml1,819
Oracle PL/SQL1,778
ProGuard1,739
Objective C1,469
ActionScript1,459
Visual Basic1,365
Mathematica1,247
RobotFramework1,074
Korn Shell1,023
D1,007
Smalltalk911
R887
TOML826
Ada668
Lisp618
Objective C++589
Fortran 77588
Arduino Sketch480
MATLAB476
sed461
Protocol Buffers454
WiX source446
JavaServer Faces440
PowerShell284
Qt Project176
Windows Message File139
Expect120
NAnt script110
Smarty109
HCL78
CoffeeScript78
Skylark74
Forth69
Qt Linguist61
WiX include52
XAML49
QML48
Handlebars46
Clojure38
Prolog37
Razor32
PO File29
Haskell27
JSX24
ASP.NET21
HLSL15
F#11
Swift10
GLSL8
Kotlin7
C Shell7
Mustache1

If you end up using my script and have any question, please let me know in the comments or directly on Github!

Categories
Eclipse IoT

On making standards organizations and open source communities work hand in hand

eclipse-and-standardsDid you know that the Eclipse Foundation is home to many open source implementations of industry standards?

From IETF to ISO to oneM2M or OASIS, we have many open source projects that provide industrial-grade implementations that anyone can use to evaluate a given standard, or to effectively use it in their commercial solution.

We do believe that open source is key to the adoption of standards, and in a presentation I gave last week at an Open Source Think Tank organized by IEEE, I shared some thoughts on what makes a standard successful, as well as how Eclipse has proved with recent success stories that open source and open communities are a key factor.

The two examples I used in my presentation (see the slides at the end of this post) originate from the Eclipse IoT community.

OMA (Open Mobile Alliance) LWM2M is a standard for doing device management of IoT devices (i.e remotely monitor the device’s health, upgrade its firmware over-the-air, etc.). The first drafts of the standard have been published less than 4 years ago and today, LWM2M is already used in commercial products, and has a thriving community of developers and contributors gathered around two Eclipse open source projects: Eclipse Wakaama, and Eclipse Leshan. I think you will agree that this is the kind of timeline you would like to see for all standards!

The other example is MQTT, a very popular IoT protocol that I’m sure you’ve heard about! 🙂 In just a few years, it went from a de-facto standard to an actual OASIS and ISO/IEC standard. Having a rich ecosystem of open source MQTT implementations (including Eclipse Paho clients, and the Eclipse Mosquitto server) certainly helped the standards organizations to pin down the issues that need to be fixed in the spec much faster. What’s more, open source projects will also fuel the future of the MQTT specification, as they allow for new ideas to be explored (see e.g this recent work on MQTT-SN).

My hope is that Standards Developing Organizations will start embracing open source initiatives more and more. Open source communities are a great place for innovation, and can host standard implementations that sometimes actually become reference implementation. They also complement very well the role of the SDOs, which are here to enforce some needed processes when it comes to evolving a standard, anticipating incompatibilities or corner cases, etc.

As mentioned above, here are the slides I used during my presentation. I am looking forward to hearing your comments and feedback.

On making standards organizations & open source communities work hand in hand from Benjamin Cabé