MQTT Namespace New (DRAFT)

🚧

This Page is Work In Progress (do not publish)

Interactions between EC and the devices take place using an open application protocol built on top of MQTT i.e. the Eclipse Kura Protocol. The protocol defines the MQTT topic namespace and the MQTT payload format used by the parties to exchange information as well as the sequence of messages required by the various types of defined interactions.
Interactions may be solicited or unsolicited. A solicited interaction is started by EC towards the device to initiate a request/response conversation. An unsolicited interaction is started by the device which sends event messages or data messages to EC for example device lifecycle events or measures or other type of data periodically collected by a sensor.

• Unsolicited interactions
  • Device lifecycle events
  • Device application data and events
• Solicited interactions
  • Request/response conversations

The parties, i.e. EC and the device, interact by subscribing and publishing to MQTT topics with well known names and structure. The adopted structure is extensible and allows to create new conversations provided that they follow the protocol rules. The following table recaps the main parts that compose a topic which complies with the protocol:

EC authentication and authorisation services ensure that MQTT connections are established on behalf of valid EC users and that the users are authorised to publish and/or subscribe to topics. EC also ensures the isolation between tenants, i.e. a device that belongs to an account cannot subscribe/publish topics of other accounts.

📘

Prefix for control topics

While EC currently requires “$EDC” as the prefix for MQTT control topics, this prefix may change in the future as MQTT 3.1.1 discourages the use of topics starting with “$” for application purposes.

MQTT Message Payload

The content of the MQTT message payload is an open format. All interactions use the same payload format: lifecycle events, device data and events and request response conversations.

The payload format consists of the following parts:

The position has the following structure:

EC assumes that payload of MQTT messages exchanged with the device client is encoded with Google Protocol Buffers using the Kura Payload message defined by Eclipse Kura (see kurapayload.proto in Eclipse Kura code repo).

Other types of encoding may be supported in the future.

📘

Payload compression

Device client can compress the payload with gzip to save network traffic. EC detects the compression and decompress the payload whenever is necessary.

Device Lifecycle Events

Device Lifecycle events are used to manage the connection status of the device and to represent the device properties and capabilities within EC:

• Connect event
• Birth event
• Apps update event
• Death event
• Disconnect event
• Last Will Testament event

EC keeps listening to the events above and as soon as they occur triggers the processing.

Device Lifecycle Flow

The typical device lifecycle flow is described below:

1. The device establishes a connection to the MQTT Endpoint.
2. The device publishes a BIRTH Certificate message describing its properties and capabilities
3. The device executes its tasks, for example:
   1. publishes data messages
   2. process a request sent by EC via the Web Console on behalf of the user
   3. process a request sent by EC via the REST API on behalf of an application
4. The device sends a Death Certificate message before cleanly close the connection
5. The device closes the connection with the MQTT Endpoint

📘

Terminology Note

The term Certificate should not be interpreted in the context of security certificates (e.g. TLS certificates. Instead it is used in the sense of a document that certifies or declares information regarding the device just like a civil with or death certificate.

Connect event

Before anything else the device has to establish a connection with EC sending an MQTT Connect packet to the MQTT service endpoint. Within the connect packet the device specifies:

• The client-id
• The username on behalf of which the device requests the connection. Depending upon the authentication schema, it might be required to include the credentials (e.g. the password).
• The keep-alive interval
• (Optionally) The Last Will Testament (LWT) topic and message

If authentication is successful and user is authorised, the connection is accepted and registered by EC. The connection status is set to CONNECTED.

EC will process the LWT message only if the LWT Topic is set to:

• $EDC/{account-name}/{client-id}/MQTT/LWT

where {account-name} matches the account of the user on behalf of which the connection is made and {client-id} matches the client-id of the connect packet.

Birth event

With the Birth event the device describe its properties and capabilities to EC. The client sends an MQTT Publish packet adding the Birth Certificate as the packet payload. The Birth Certificate message must be published to the following control topic:

• $EDC/{account-name}/{client-id}/MQTT/BIRTH

EC uses the content of the Birth Certificate to create its own representation of the device and persist it in the Device Registry

If the device is not yet present in the Device Registry it will be created otherwise it will be updated with the content of the latest Birth Certificate. The Device Registry is accessible to authorised users via the Console and to authorised applications via the REST API.

Apps Update event

The Apps Update event is used by device to notify EC that the set of applications supported by the device has changed. The client sends an MQTT Publish packet adding a Birth Certificate with the updated app list as the packet payload. The Birth Certificate with the updated app list must be published to the following control topic:

• $EDC/{account-name}/{client-id}/MQTT/APPS

EC will update the list of the applications in the Device Registry consistently.

Death event

The Death event is used by the device to notify EC that it is closing the session, before cleanly disconnecting from the MQTT Service. The client sends an MQTT Publish packet adding a Death Certificate as the packet payload. The Death Certificate message must be published to the following control topic:

• $EDC/{account-name}/{client-id}/MQTT/DC

EC processes and records the event but does not change the connection status of the device (see Device Events in this guide).

Disconnect event

The device cleanly closes the connection with the MQTT Service sending an MQTT Disconnect packet to the MQTT Service endpoint. EC updates the connection status of the device to DISCONNECTED.

📘

Non-clean Disconnect

The device sets the keep-alive interval in the MQTT Connect packet. If the broker receives nothing from the MQTT client within 1.5 times if the interval, it assumes the connection is lost. If the client device set an LWT topic and message, EC receives the LWT message and sets the status to MISSING. If no LWT topic and message are set or if they not comply with the rules described in the section Connect Event, the connection is just deleted from the broker and the status is set to DISCONNECTED.

Last Will Testament event

If the Last Will Testament is set in the MQTT connection to the MQTT Service, and if a device disappears from the network without issuing an MQTT DISCONNECT packet, the Last Will Testament message will be sent by the broker to the topic:

• $EDC/{account-name}/{client-id}/MQTT/LWT

The LWT informs EC that the connection was lost before receiving a clean MQTT Disconnect. However EC can detect the loss of the connection whether or not the LWT message is received (e.g. when the keep-alive interval is exceeded) and change the connection status consistently.

📘

Last Will Testament in multi-broker MQTT Service instance

When devices connect to a multi-broker MQTT Service instance, delivery of LWT message to clients subscribed to the LWT topic is not guaranteed. This doesn't impact the ability of EC to determine the device connection status.

Device Status Table

The following table recaps how the lifecycle events determine the device connection status in the registry:

Device Application Data and Events

Application data and events are application specific information that is sent to EC to be processed-by or stored-into some upstream application like for example a Data Warehouse or an ERP. The client sends an MQTT Publish packet adding the data/event as the packet payload. The data/event message must be published to the following topic:

• {account-name}/{client-id}/{semantic-topic}

A good practice is to keep the semantic-topic meaningful but possibly compact in length and depth (number of levels in the hierarchical notation) to save network traffic.

EC has two built-in services to process these data. Store them in the Datastore service where they are organised as time series, or route them to external systems via the Route functionality.

Request/Response Conversations

Solicited interactions require a request/response message pattern to be established over MQTT between EC and the device. To initiate a solicited conversation, a remote server first sends a request message to a given application running on a specific device and then waits for a response.

Request/response flow

To ensure the delivery of request messages, applications that support request/response conversations via MQTT should subscribe to the following topic on startup:

$EDC/{account-name}/{client-id}/{app-id}/#

A requester (i.e. the remote server) initiates a request/response conversation through the following events:

1. Generating a conversation identifier known as a request.id (e.g., by concatenating a random number to a timestamp)
2. Subscribing to the topic where the response message will be published, where requester.client.id is the client ID of the requester, such as:

$EDC/{account-name}/{requester.client.id}/{app-id}/REPLY/{request.id}

3. Sending the request message to the appropriate application-specific topic with the following fields in the payload:
   1. request.id (identifier used to match a response with a request)
   2. requester.client.id (client ID of the requester)
      The application receives the request, processes it, and responds on a REPLY topic structured as:

$EDC/{account-name}/{requester.client.id}/{app-id}/REPLY/{request.id}

For more details regarding request/response conversations please refer to the following documentation:

• MQTT Namespace chapter of the Eurotech ESF documentation
• MQTT Namespace chapter of the Eclipse Kura documentation

Pre-defined request/response conversations

ESF and Kura support several request/response conversations out of the box. ESF and Kura publish the supported conversations in the applications field of the Birth Certificate message. Cloud side, EC implements support for these conversations with the device management features; check the DEVICE MANAGEMENT chapter in this guide for more information regarding the supported device management conversations starting from Gateway Management section and following with System Management, Device Assets, etc.

Asynchronous notifications

Package Deployment

Package Deployment application (DEPLOY-V2) has operations that can take long time to execute: download, install and uninstall, if EC had to wait until the completion of the operation the request timeout may expire. For this application a notification mechanism was implemented. The request returns immediately with code 200 if accepted then the execution starts. The device reports progress by publishing notification to the following control topic:

• $EDC/{account-name}/{requester.client.id}/DEPLOY-V2/NOTIFY/{client-id}/{resource-id}

The resource-id is one of download, install, uninstall. Notifications are correlated to the original request by the job ID metric in the the message payload. For more information refer to the following documentation: