My Second Project at Metosin!

IntroductionLink to Introduction

I started working at Metosin at the beginning of 2020. For the first one and a half years, I worked on an interesting Clojure & cloud project, first implementing an AWS infrastructure in the devops team and then working with full-stack Clojure apps in a development team. You can read more about that project in my previous blog post My First Project at Metosin! This new blog post describes my second project at Metosin, an interesting IoT case. I have written four blog posts related to that project and the technologies used in the project. You might want to read them as well:

• AWS IoT First Reflections
• AWS IoT Storage Considerations
• On Superorganisms
• Clojurescript Frontend Development For Novices

ProjectLink to Project

I came from my summer vacation in September. I was lucky since an exciting project was waiting for me - one of our customers wanted us to implement an IoT platform and an IoT visualization app for them. The customer needed a minimum viable product at this stage - our customer uses the IoT platform and the MVP app for demonstration purposes for its clients. The customer had an IoT specialist whose responsibility was to implement the IoT sensors used in the project. My responsibility was to implement an IoT platform for ingesting the IoT sensor data and an application to visualize the IoT sensor data.

TechnologiesLink to Technologies

• AWS: The cloud platform.
• AWS IoT Core: The part of the infrastructure for ingesting the IoT sensor data.
• AWS IoT Analytics: For storing the raw IoT messages in the S3 data lake.
• AWS Lambda: I used Lambda functions to process the sensor data events and store the events into RDS.
• AWS Kinesis: Kinesis data stream is a superb way to fan out data and buffer it for the next processing phase.
• AWS S3: The AWS object storage. I used a customer managed S3 data lake for storing the raw IoT sensor data events processed by AWS IoT Analytics.
• AWS RDS: The AWS relational database as a service. I used RDS to store the latest IoT data for the visualization application.
• Terraform: The infrastructure as code tool.
• AWS CDK: AWS Cloud Development Kit. Another IaC tool. At the time of implementing the infrastructure Terraform did not support AWS IoT Analytics, and therefore I had to use AWS native IaC tool to implement that part of the infrastructure.
• Clojure: The Clojure programming language. I used Clojure to implement the backend application.
• ClojureScript: Clojure on the browser (transpiled into Javascript). I used ClojureScript, Reagent and re-frame to implement the frontend IoT visualization application.
• Reagent: Minimalistic React for ClojureScript.
• Re-frame: A web app framework built around Reagent.
• Vega-Lite: The visualization library.

IoT PlatformLink to IoT Platform

I implemented the IoT platform using AWS. AWS provides excellent components for ingesting IoT sensor data (AWS IoT Core), processing the data further, and storing it for various purposes. If you use AWS services, you can glue together a nice solution with minimal code. The actual code implemented in the infrastructure was relatively minimal, just some Lambda code to fan out data and store it to S3 and RDS. I used Terraform and AWS CDK as infrastructure as code tools (AWS CDK for implementing IoT Analytics - Terraform didn't support it at the time of implementation). Therefore, the infrastructure was fully automated using infrastructure as code paradigm - it was easy to create dedicated development environments for both myself and the other IoT specialist and a dedicated demonstration environment for the customer. I'm not going to describe the IoT infrastructure more in this new blog post since you can read more about the IoT infrastructure in my previous blog post AWS IoT Storage Considerations.

IoT Virtual DeviceLink to IoT Virtual Device

I created an IoT virtual device for development and testing purposes. The IoT virtual device was just a Python script that generates "IoT sensor events" and publishes them to the IoT Core. This way, I could generate real IoT-like data to the IoT storages of my IoT solution. The IoT virtual device served two purposes: 1. It was easier to develop the visualization app when I had a lot of various IoT sensor like data in my IoT storages, and 2. I used the IoT virtual device also in the demonstration environment to stream IoT events to the demonstration environment IoT storages - our customer could try the latest application version with real-looking IoT data.

IoT Data ModelLink to IoT Data Model

I designed a dynamic and open data model. I.e., I didn't fix the fields (e.g., temperature, vibration), but I used "dimension-name" and "dimension-value" fields. This was a good design choice. In one of our weekly demonstration sessions, our customer asked if it was difficult to add a new field that they would like to use when demonstrating the application to their end customer. I replied: "No, it is quite trivial," and edited the IoT virtual device script (added the requested field and min/max values) and re-started the virtual IoT device. After a while, the new IoT data reached our IoT storages, and I showed our customer that the new field is in the application selection box, and we can visualize that data.

Clojure ApplicationLink to Clojure Application

Clojure application development was so much fun! I created a Clojure backend application using reitit. It was pretty trivial to fetch the IoT data from RDS, process it a bit, and return it through the REST API. For frontend development, I didn't have that much experience. But with the help of the Metosin superb frontend gurus, I was pretty productive and able to create a nice-looking IoT visualization frontend application. I used Vega-Lite to implement the charts in the frontend. Vega-Lite is an excellent graphics library; I recommend checking it out if you need to visualize some data.

Regarding the Clojure development, you might want to read these blog posts:

• Clojurescript Frontend Development For Novices: This is basically the frontend technology stack I used (+ Vega-Lite).
• On Superorganisms: In this blog post, I describe that you don't have to be a guru in every area of software development if you are working in a company with competent and helpful colleagues. I'm pretty mediocre in frontend development, but with the help of Metosin's excellent frontend gurus, I was amazingly productive when implementing a nice-looking frontend.
• Vega-Lite Experimentation Bench: I later implemented a small Vega-Lite experimentation bench that I used to study Vega-Lite more.
• Vega-Lite Experimentation Part 2: This blog post describes the actual Vega-Lite experiments using that experimentation bench.

ConclusionsLink to Conclusions

I loved the project. I gave this project the highest 5 points on my scale from 0 to 5 that I use to score projects regarding how interesting they are related to technology and learning possibilities. What I especially appreciated in that project was the possibility of implementing everything myself: the IoT infrastructure and the full-stack Clojure application. It is a wonderful feeling not to depend on some other specialist but to make progress yourself in all areas of the project. I pondered this and wrote a blog post about it: Real Full-Stack Developer. A real full-stack developer is not a specialist who can do both frontend and backend, but a specialist who first implements the cloud infrastructure and then implements the backend and frontend applications running in that infrastructure.

Another insight was that you are not an island. You don't have to know 100% of all areas of software development. Working in a superorganism-like company gives you the superpower of all specialists in the company - with the help of your colleagues, you can make fast progress and not be stuck on some specific problem.

There are many interesting Clojure & cloud projects at Metosin. I strongly recommend contacting Eeva Lennon and reading our Careers page if you are interested in starting a Clojure career.

By the way. I took the picture the same day I wrote this blog post, in January 2022. I took a couple of hours off and went to Haltiala forest and fields for a cross-country skiing trip. It was a beautiful sunny winter day, some -13 degrees celsius. The sun was shining from the blue sky, and the snow shone magically. I really love to live and work in Finland - there is a lot of nature around and excellent opportunities for outdoor activities.