Global
What is it good for?
MQTT is an OASIS standard messaging protocol for the Internet of Things (IoT). It is designed as an extremely lightweight publish/subscribe messaging transport that is ideal for connecting remote devices with a small code footprint and minimal network bandwidth.
From the sea of protocols aimed to send data from clients to the cloud, MQTT is the most suitable for IoT use cases. Nowadays, we see it more and more used in real projects. Its implementations are confidently holding the position of reliable communication platforms for environments with unreliable connection and thousands of small devices.
As this lightweight protocol is best suited for IoT solution, we will look on different aspects and products for its usage in our projects.
Nowadays in IoT systems like factories, production plants, homes and digital twins, the most used protocols to transfer data from sensors to cloud are AMQP, MQTT and CoAP. Every option has its advantages and disadvantages in different scenarios so why is then MQTT used the most? To answer this question, we have to compare them and evaluate their usage.
But first, let’s talk about one of the key features of any type of communication middleware, Quality of Service. To have better picture of reliability of services, we distinguish three types:
QoS 0 - fire and forget
- At most one sent message is received. If there is any failure in network, communication or supporting applications, message is not delivered.
QoS 1 - at least once
- Every message is delivered to the receiver. Due to failures, multiple same messages or messages out of order can be delivered.
QoS 2 - exactly once
- Messages are delivered exactly once in same order they were sent.
AMQP
Advanced Message Queuing Protocol is an open standard message-oriented protocol which features are orientation, queuing, routing, reliability and security. Previous API level middleware focused on standardizing programmer interaction with different implementations, rather than on providing interoperability between multiple implementations. It supports wide variety of messaging applications and communications patterns including request/response and publish/subscribe and Quality of Service 0,1 and 2 using TCP protocol. It has general capability to transfer up to 1000 messages per broker instance.
CoAP
Constrained Application Protocol is optimized for constrained low-power devices in environment with unreliable network. CoAP is designed to easily translate to HTTP for simplified integration with the web, while also meeting specialized requirements such as multicast support, very low overhead, and simplicity. It supports request/response communication with Quality of Service defined as confirmable and unconfirmable messages. Using UDP, it supports transfer of up to 100 messages.
MQTT
Formerly Message Queuing Telemetry Transport but now, MQTT abbreviation means nothing specific. It is designed as an extremely lightweight publish/subscribe messaging transport that is ideal for connecting remote devices with a small code footprint and minimal network bandwidth. Supporting QoS 0,1 and 2 can using TCP transfer up to 1000 messages per one broker instance.
When our goal is to transfer
- the most messages,
- from constrained devices,
- easiest way,
- with option to use QoS 0 to 2,
But how it works?
Whole publish/subscribe communication has three types of actors.
-
Publishers publishing messages to the broker to specific topic and queuing them in case of network unavailability.
- Advanced sensors, edge devices and PLCs producing telemetry and sensory data
-
Subscribers subscribed to specific topic or group of topics receiving messages.
- Backend services, message busses, other IoT devices, analytical applications, time-series databases and client devices for further processing
- Broker receiving messages and making sure all subscribed clients will receive the message.
MQTT v5
In 2019, new version of MQTT came out with some very interesting additions that keeps protocol as lightweight as before but provides more options to manage connections and notify clients about different events they can be interested in. New version is naturally compatible with old version so old MQTT v3.1.1 messages are not a problem when communicating with a broker.
- Challenge/response authentication in addition to username/password
- “last will” message. When connecting to the broker, client can provide message, that broker will send to all subscribers in case of “not clean” disconnection
- Interval in CONNECT packet allows broker to drop the client if time after disconnecting elapsed
- Client can limit number of QoS1 and QoS2 messages it is willing to process (note: flooding a client with QoS0 still possible)
- Broker can send the client a disconnect message and reason code e.g. 137 Server busy
- Message expiration to prevent clients from receiving stale data (retain forever in v3.1.1 for clean session flag and QoS 1 or 2)
- Client-to-broker and broker-to-client topic aliases to further reduce packet size
- Shared subscriptions let the broker distribute incoming messages to a group of “competing consumers”, only delivering each message to one of the groups (in non-shared, message is delivered to all subscribers)
- Subscription flags Retain as Published, No Local, and Retain Handling
- Subscriber IDs: optional integer identifier to route message to handler (instead of parsing the topic)
How individual brokers can be different?
Brokers can differ in their internal implementation and provide different set of features that can be crucial in specific use cases, so it is not trivial task to choose one over another. The most clear view on the topic is division into next few categories:
TCP Retransmission broker
This is the simplest broker we can use. It is single, non-clustered broker where we rely only on TCP retransmission for ensuring the delivery of message. Every other failure in the chain of transfer process will result into not delivered message.
Publication Acknowledgements broker
Broker is using acknowledgement notices of received messages by broker and also subscribed clients so message can be resent in case of a failure.
Single instance Broker with Persistence
For the case of broker failure and need to restart broker instance, this type of brokers uses persistence volume to store client’s session data with message queues. With option to use distributed storage, it can be immune also against disk failure.
Clustered Broker
When using clustered broker, even one broker instance failure will not affect broker availability because there are other instances already running. Only already sent messages will be lost in case persistent volume not configured.
- VerneMQ
- EMQ-X3
Clustered Broker with Queue Mirroring
With mirroring of user session and received messages to other nodes, this is the most reliable broker type. As nothing has only advantages, mirroring comes with a slight delay in message processing, so this feature is useful in specific cases.
- Only HiveMQ
But we can still use proprietary implementations
AWS IoT Core
Azure IoT Hub
Yes, when working with cloud solutions, incorporating MQTT capability for communication with devices is trivial task. Just enable AWS IoT Core or Azure IoT Hub, use Azure or AWS MQTT client on its edge device runtime and you are good to go. Great advantage is fast learning curve and ease of use in standard use cases and general implementations. Everything is working as intended but problems can arise after a time of usage, when system will become robust, or when customer choose to change cloud provider.
Brokers are a bit of a black box with not much options to configure or trace and debug problems and MQTT protocol is not fully supported. Proprietary implementation also lacks MQTT v5 support and QoS 2, that can be problem in future. As MQTT protocol implementation is custom with slight differences, it is hard to switch edge device runtime, device onboarding runtime or cloud platform.
Dependency on platform can be big problem when cloud provider decides to change pricing model of its services. Customer business model can be dependent on service cost per device/data/deployment and change can make it not feasible.
How MQTT can help in transportation
Every part of our daily life is in some way affected by IoT. Transportation domain is not different and using IoT approach to problems, it can be even more driven by innovation and data. MQTT is therefore important tool that makes it easy to connect complex IoT infrastructure and enables us to focus on data processing and process optimization. Among the solved problems of transportation devices communication are:
- Persistent Always-on Client Connection: pub/sub architecture allows vehicles to be decoupled. Allows persistent, always-on push connection to the cloud.
- Guaranteed and Reliable Data Delivery: advanced message retention policies and offline message queuing are essential to accommodating network latency and unreliable mobile networks.
- Secure non-addressable clients: Vehicles or devices running MQTT clients are not addressable over the Internet. This makes it virtually impossible for a vehicle to be directly attacked by a hacker from the Internet.
- Efficient network utilization: MQTT was designed specifically to operate over constrained networks. HTTP deals with requests one at a time, with overhead such as authentication being carried out each time. In one HiveMQ test scenario, 100 messages sent using MQTT required 44,748 bytes, compared to 554,600 bytes for 100 messages sent using HTTP.
- Elastic scalability and Auto Heal: user does not see any change in user experience when nodes are started or stopped since the connected vehicle, device or sensor can resume its session on any of the remaining cluster nodes.
- IoT observability: HiveMQ tools allow for a system administrator to pinpoint issues with a specific vehicle and work to resolve the issues. This allows administrators to do real-time fleet monitoring and remote debug and trade of interactions between a vehicle and the broker.
Written by Gabriel Novak, Senior Software Engineer at Ness Digital Engineering
