Kafka Route
Parameters for Apache Kafka Routes
Parameter Name | Description | Default |
|---|---|---|
Kafka Brokers | Comma separated list of Kafka brokers in the format host:port (or ip:port) | 127.0.0.1:9092 |
Header Property (1) | Header property name in the Camel message where Camel Kafka component will find the composed URL representing the endpoint to invoke. | evaluatedKafkaEndPoint |
Original Topic (1) | The name of the topic where the message was originally sent to by the device. By default it is read from the originalTopic property of the Camel message. It is used by Kafka to calculate the partition key. | $simple{in.header.originalTopic} |
Message Converter | Message converter to convert the incoming message before forwarding it to Kafka
| none |
Key Serializer | Kafka-record key serializer. | org.apache.kafka.common.serialization.StringSerializer |
Key Header (2) | Kafka-record key assignment method. Can be either NONE or TOPIC. The recommended value is NONE. When the method is NONE, no key is assigned to the record so that Kafka will manage which topic partition the record will be assigned to. When the method is TOPIC the key is set to the original topic of the message (e.g. the MQTT topic of the telemetry message sent to EC by the device). | NONE |
Serializer | Kafka-record value serializer. | org.apache.kafka.common.serialization.StringSerializer |
URL Options | Additional Kafka endpoint options. This field allows to specify more parameters to be used by the Kafka client. The options are added to the composed URL. For the available client parameters please refer to the Kafka client documentation (see URL Options below). | none |
Topic Matching | Describes how to map the incoming messages topics to Kafka topics (see Topic Matching Example below). | .*,${ACCOUNT} |
(1) The parameter has been removed starting from release 5.8.1. If specified it will be ignored.
(2) The parameter has been introduced starting from release 5.9.3. Using the TOPIC assignment method all the messages from the same message topic will be sent to the same Kafka topic partition. This choice could impact the performances of your Kafka instance. Routes created with versions earlier the 5.9.3, will work as if they have the parameter set to NONE.
Supported Kafka Client
The currently supported Kafka client version is 0.10.2.0
Output Json example
{
"id": "550e8400-e29b-41d4-a716-446655440000",
"scopeId": "AQ",
"deviceId": "RyO9_XKGRsw",
"clientId": "client-id-0",
"receivedOn": 1543486113,
"sentOn": 1543486113,
"capturedOn": 1543486113,
"position": {
"longitude": 0.0,
"latitude": 0.0,
"precision": 50,
"heading": 240,
"speed": 110,
"timestamp": 1543486113,
"satellites": 2,
"status": 0
},
"channel": {
"semanticParts": [ "one", "two", "three" ]
},
"payload": {
"metrics": {
"metric_one": {
"value": "string_value",
"valueType": "string"
},
"metric_two": {
"value": 123,
"valueType": "integer"
},
"metric_three": {
"value": true,
"valueType": "boolean"
}
},
"body": "VGhpcyBpcyBhIGJvZHkgY29udGVudA=="
}
}
Topic matching example
This is a topic matching example.
Please note that the incoming message use the JMS syntax (so the topic levels are . (dot) separated instead of / (slash) separated)
(?:[^.]*?\.){2}topic1\.topic1_1,kafka_topic_1
(?:[^.]*?\.){2}topic2,kafka_topic_2
(?:[^.]*?\.){2}topic3\.topic3_1,kafka_topic_3
This configuration forwards the messages for all the account's clients to different destination topics depending on 3 semantic topics:
MQTT semantic topic | Kafka topic |
|---|---|
topic1/topic1_1 | kafka_topic_1 |
topic2 | kafka_topic_2 |
opic3/topic3_1 | kafka_topic_3 |
Topic Names
Kafka allows only few characters in the topic (regex coming from the Kafka source code "[a-zA-Z0-9._-]"). Please take care of this limitation while setting the topic name route field.
URL Options
The route supports various configuration options. For example these are various combinations used for different authentication methods.
Protocol/Mechanism | Description | URL Option |
|---|---|---|
PLAINTEXT (no authentication) | Plaintext, no authentication | &securityProtocol=PLAINTEXT |
PLAINTEXT/PLAIN | Plaintext, basic authentication | &securityProtocol=SASL_PLAINTEXT&saslMechanism=PLAIN |
SSL/PLAIN | SSL/TLS encrypted, basic authentication | &securityProtocol=SASL_SSL&saslMechanism=PLAIN |
PLAINTEXT/GSSAPI | Plaintext, Kerberos authentication | &securityProtocol=SASL_PLAINTEXT&saslMechanism=GSSAPI&saslKerberosServiceName=kafka |
SSL/GSSAPI | SSL/TLS encrypted, Kerberos authntication | &securityProtocol=SASL_SSL&saslMechanism=GSSAPI&saslKerberosServiceName=kafka |
For more details regarding security configurations refer to section Secrets for Helm Charts.
Updated 3 months ago