Export EdgeX events to Azure Event Hubs by Kafka

Overview

Azure Event Hubs provides an Apache Kafka endpoint on an event hub, which enables users to connect to the event hub using the Kafka protocol. Edge Xpert provides a proprietary KafkaSend function to send data to Azure Event Hubs through Kafka. This article will take you through steps to create an Edge Xpert application service to send data as generated by virtual device service to an event hub. For further information on Azure Event Hubs, refer to the Azure Event Hubs for Apache Kafka.

Prerequisites

Understand Edge Xpert Application Services
Launch Edge Xpert core services
```
edgexpert up
```
or use --secret option to run in security mode
```
edgexpert up --secret
```
Create an Azure event hub

Configure an application service to export events to Azure Event Hubs

Edge Xpert provides a proprietary built-in function KafkaSend to produce events to the Azure Event Hubs Kafka endpoint. A sample configuration is provided with Edge Xpert CLI utility. Once you complete the installation of Edge Xpert CLI utility, the sample configuration is available at /usr/share/edgexpert/examples/app-configurable/kafka-export/azure-event-hubs-insecure.toml.

Copy and save the sample configuration file as a separate file (for example, eventhubs.toml) to your working directory
Modify this configuration based on the Azure Event Hubs settings. The Azure Event Hubs Kafka endpoint is {namespace_name}.servicebus.windows.net with port 9093,replace the {namespace_name} to your Azure Event Hubs Namespace. Kafka topic is corresponding to your event hub name. Azure Event Hubs requires to use Shared Access Signature (SAS) to authorize access to your secure resources. Thus, the AuthMode must be set to sasl_ssl

For example, user shall revise KafkaSend with namespace: edgexpert, event hub: test, Kafka Client ID: edgex, Kafka broker port: 9093, and Kafka partition: 0.
```
[Writable.Pipeline.Functions.KafkaSend]
  [Writable.Pipeline.Functions.KafkaSend.Parameters]
  ClientID = "edgex"
  Address = "edgexpert.servicebus.windows.net"
  Port = "9093"
  Topic = "test"
  Partition = "0"
  AuthMode = "sasl_ssl"
  SecretPath = "kafka"
```

If the application service is not launched in secure mode, secrets will be retrieved from Writable.InsecureSecrets configuration. You will need to update Writable.InsecureSecrets.kafka configuration with correct values of azureeventhubsconnectionstring:

[Writable.InsecureSecrets.kafka]
path = "kafka"
  [Writable.InsecureSecrets.kafka.Secrets]
  # When using KafkaSend and AuthMode="sasl_ssl", azureeventhubsconnectionstring is the mandatory secret.
  # Refer to https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-get-connection-string
  azureeventhubsconnectionstring = "{eventhubs_connection_string}"

Once configuration is ready, you can launch the application service in secure mode using command below:
```
edgexpert up --secret app-service --path=<path of the configuration file>
```
If you prefer to launch the application service in insecure mode, use command below:
```
edgexpert up app-service --path=<path of the configuration file>
```
If the application service is launched under secure mode on previous step, secrets will be retrieved from Vault secret store. Users need to add azureeventhubsconnectionstring into Vault secret store. Please refer to EdgeX Foundry Storing Secrets for details about adding secrets into Vault secret store.
Now, an application service is up and ready to export events to Azure Event Hubs. To simulate the device sending readings from southbound, you can start up the virtual device service:
```
# Under secure mode
edgexpert up --secret device-virtual

# Under insecure mode
edgexpert up device-virtual
```
Azure Event Hubs allows users to see the number of received messages. To do this in the Azure Portal, select your event hub and Overview from the menu options.
Azure Event Hubs also allows users to view the messages using the real time insights. Follow the steps as shown in the screenshots.

More configuration details about KafkaSend

Refer to KafkaSend configuration.