Skip to content

Batch Transform Example

The Batch transform example is located in the examples/batch_transform directory. This example shows how to use the Batch Transform component with the MQTT exporter to export batched data from inside a container using JSON configuration files.

When the example is run, a container is instantiated and populated with components. The container uses component factories to manage components, which are specified in the config/main.json file. Each entry for a component in this file specifies the following:

  • The name of the component, for example: batch
  • The name of the registered factory used to create the component, for example: XRT::BatchTransform

The components used in this example are as follows:

  • Logger component
  • ThreadPool component
  • Scheduler component
  • Core Bus component
  • Virtual Device Service component. This publishes data on the Bus
  • Batch Transform component. This subscribes to the data published by the Virtual Device Service, and republishes the batched data on another topic
  • MQTT Exporter component. This subscribes to the batched data available on the Bus on a topic for export
  • Log Exporter component. This subscribes to the data published by the Virtual Device Service on the same topic set by the Batch exporter, and outputs it to the log

Each component can be configured using the associated JSON file; for example, the Batch Transform component is configured using the batch.json file. For further information on the configuration of the Batch Transform component, see Batch Transform Component

Note

There are three instances of the Logger component for each configuration. One is configured to Info level and is used by the Log Exporter component that logs the messages available to the Batch Transform component as input. The second is configured to log Debug level messages and is used by the Batch Transform component. The third is configured to log Warning level messages for reduced verbosity and is used by the other components in this example.

In this example, the Virtual Device Service is configured to generate a reading event every half a second and publish it to the Bus using the device/data topic, which is the default topic for events. The Batch Transform component is configured to subscribe to the device/data topic, and publish all messages on the device/processed topic.

To run the Batch Transform component example for five seconds, enter the following commands:

cd examples/batch_transform
xrt config 5

The output is similar to the following:

[xrt:1622556320623786:console:Info] iot_threadpool_alloc (threads: 1 max_jobs: 500 default_priority: -1 affinity: -1)
[iot-1-0:1622556320632076:console:Debug] Thread iot-1-0 starting
[iot-0-0:1622556320632792:console:Info] device/data:{"device":"xrt-example-device1","resource":"SensorOne","readings":{"Origin": {"value":"Sensor54","type":"string"},"Temp":{"value":123,"type":"uint32"},"Uptime": {"value":456789,"type":"uint64"}}}
[xrt:1622556320764635:console:Info] xrt_mqtt_paho_connect: MQTT client connected
[iot-0-1:1622556321133014:console:Info] device/data:{"device":"xrt-example-device1","resource":"SensorOne","readings":{"Origin": {"value":"Sensor54","type":"string"},"Temp":{"value":122,"type":"uint32"},"Uptime": {"value":456790,"type":"uint64"}}}
[iot-0-0:1622556321633112:console:Info] device/data:{"device":"xrt-example-device1","resource":"SensorOne","readings":{"Origin": {"value":"Sensor54","type":"string"},"Temp":{"value":121,"type":"uint32"},"Uptime": {"value":456791,"type":"uint64"}}}
[iot-0-1:1622556322133259:console:Info] device/data:{"device":"xrt-example-device1","resource":"SensorOne","readings":{"Origin": {"value":"Sensor54","type":"string"},"Temp":{"value":120,"type":"uint32"},"Uptime": {"value":456792,"type":"uint64"}}}
[iot-0-0:1622556322633483:console:Debug] batched data to publish: [{"device":"xrtexample-device-1","resource":"SensorOne","readings":{"Origin": {"value":"Sensor54","type":"string"},"Temp":{"value":123,"type":"uint32"},"Uptime": {"value":456789,"type":"uint64"}}},{"device":"xrt-example-device1","resource":"SensorOne","readings":{"Origin": {"value":"Sensor54","type":"string"},"Temp":{"value":122,"type":"uint32"},"Uptime": {"value":456790,"type":"uint64"}}},{"device":"xrt-example-device1","resource":"SensorOne","readings":{"Origin": {"value":"Sensor54","type":"string"},"Temp":{"value":121,"type":"uint32"},"Uptime": {"value":456791,"type":"uint64"}}},{"device":"xrt-example-device1","resource":"SensorOne","readings":{"Origin": {"value":"Sensor54","type":"string"},"Temp":{"value":120,"type":"uint32"},"Uptime": {"value":456792,"type":"uint64"}}},{"device":"xrt-example-device1","resource":"SensorOne","readings":{"Origin": {"value":"Sensor54","type":"string"},"Temp":{"value":119,"type":"uint32"},"Uptime": {"value":456793,"type":"uint64"}}}]
[iot-0-0:1622556322633620:console:Info] device/data:{"device":"xrt-example-device1","resource":"SensorOne","readings":{"Origin": {"value":"Sensor54","type":"string"},"Temp":{"value":119,"type":"uint32"},"Uptime": {"value":456793,"type":"uint64"}}}
[iot-0-0:1622556322633787:console:Debug] iot_threadpool_add_work jobs/max: 1/500
[iot-0-0:1622556322633813:console:Debug] iot_threadpool_add_work jobs/max: 1/500
[iot-0-0:1622556322633824:console:Debug] iot_threadpool_add_work jobs/max: 2/500
[iot-0-0:1622556322634225:console:Debug] iot_threadpool_add_work jobs/max: 2/500
[iot-0-0:1622556322634242:console:Debug] iot_threadpool_add_work jobs/max: 3/500
...
In this output, the messages from the Virtual Device Service are available on the device/data topic, which is the input to the Batch Transform component, and the batched data to publish entry is the output from the Batch Transform component.

Back to top