Export to InfluxDB

Overview

The following example describes how to configure an application service to export EdgeX events to InfluxDB by using proprietary built-in function InfluxDBSyncWrite. Edge Xpert uses InfluxDB 2.6.x by default. If you are using an earlier version of InfluxDB, please refer to Upgrading InfluxDB.

Note

You can also create application services with EdgeXpert Manager, which is the graphical user interface (GUI) of EdgeXpert. Refer to Supporting Services - InfluxDB for more information.

Prerequisite

Understand Edge Xpert Application Services
Launch Edge Xpert core services
```
  edgexpert up
```
or use --secret option to run in security mode
```
  edgexpert up --secret
```
Launch InfluxDB and obtain the information
- Influx DB server URL
- Influx Organization
- Influx Bucket
- measurement
- measurement field data type
- timestamp precision
- authentication token
Info

Since InfluxDB 2.0.x, the authentication token can be either specified or auto-generated. InfluxDB container will take the value of DOCKER_INFLUXDB_INIT_ADMIN_TOKEN environment variable as the token value during setup. If DOCKER_INFLUXDB_INIT_ADMIN_TOKEN environment variable is not found during setup, InfluxDB will automatically generate the token, and you can obtain the token value through docker exec <influxdb container name> influx auth list --user admin --json command. Edge Xpert sets DOCKER_INFLUXDB_INIT_ADMIN_TOKEN to custom-token by default in the compose files.

Configure an application service to export events to InfluxDB

Edge Xpert provides a proprietary built-in function InfluxDBSyncWrite to convert EdgeX events to InfluxDB points and then synchronously write to InfluxDB. If Edge Xpert CLI utility is installed, a sample configuration can be found in /usr/share/edgexpert/examples/app-configurable/influxdb-write/configuration.toml for InfluxDB.

Copy and save the sample configuration file as a separate file (for example, influxdb-write.toml) to your working directory

Modify this configuration based on the information obtained from step 3 of the Prerequisite. See InfluxDBSyncWrite configuration details.

[Writable.Pipeline.Functions.InfluxDBSyncWrite]
  [Writable.Pipeline.Functions.InfluxDBSyncWrite.Parameters]
    InfluxDBServerURL = 'http://localhost:8086'
    InfluxDBOrganization = 'my-org'
    InfluxDBBucket = 'my-bucket'
    InfluxDBMeasurement = 'readings'
    InfluxDBPrecision = 'us'
    Authmode = 'token'
    Secretpath='influxdb'
    Skipverify = 'true'
    PersistOnError = 'false'
    StoreEventTags = 'false'
    StoreReadingTags = 'false'
    FieldKeyPattern = "{resourceName}"

If the application service is not launched in secure mode, secrets will be retrieved from Writable.InsecureSecrets configuration. You will need to update Writable.InsecureSecrets.influxdb configuration with correct values of token:
```
[Writable.InsecureSecrets.influxdb]
path = 'influxdb'
  [Writable.InsecureSecrets.influxdb.Secrets]
  token = 'custom-token'
```
Once configuration is ready, you can launch the application service in secure mode using command below:
```
$ edgexpert up --secret app-service --path=<path of the configuration file>
```
If you prefer to launch the application service in insecure mode, use command below:
```
$ edgexpert up app-service --path=<path of the configuration file>
```
If the application service is launched in secure mode on previous step, secrets will be retrieved from Vault secret store. You need to add token into Vault secret store.
Note

Follow the steps below to use POST secrets REST API of Application service to add secrets into secret store:
1. Inspect the IPAddress of the running Application service:
```
$ docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' app-configuration
172.17.0.6
```
2. Insert the copied string into the payload of REST call to add the secret. Note that all necessary secrets must be added in a single REST call, otherwise the secrets content will be overwritten by REST calls afterwards. For example, the following curl command adds one secret--token--under the secret path influxdb in the Vault secret store:
```
curl -k -X POST -d '{ "apiVersion": "v2", "path": "influxdb", "secretData": [ {"key":"token","value":"custom-token"}]}' http://172.17.0.6:59700/api/v2/secret
```
Now, an application service is up and ready to export events to InfluxDB. To simulate the device sending readings from southbound, you can start up the virtual device service:
```
# Under secure mode
edgexpert up --secret device-virtual

# Under insecure mode
edgexpert up device-virtual
```

There are various way to monitor and query those events being wrote to InfluxDB, please refer to InfluxDB 2.6 documentation for more details. The following sample uses the Influx command line interface inside the InfluxDB docker container (for example, run command docker exec -it influxdb bash) to query all records that were inserted within the last five minutes under my-bucket :

# run the following command inside the InfluxDB 2.0.x docker container
# influx query 'from(bucket:"my-bucket") |> range(start:-5m)'
Result: _result
Table: keys: [_start, _stop, _field, _measurement, deviceName, resourceName]
_start:time                      _stop:time           _field:string     _measurement:string       deviceName:string     resourceName:string                      _time:time  _value:boolean
------------------------------  ------------------------------  ----------------------  ----------------------  ----------------------  ----------------------  ------------------------------  --------------
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                    Bool                readings   Random-Boolean-Device                    Bool  2022-08-05T11:27:22.203600000Z            true
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                    Bool                readings   Random-Boolean-Device                    Bool  2022-08-05T11:27:32.204440000Z           false
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                    Bool                readings   Random-Boolean-Device                    Bool  2022-08-05T11:27:42.204840000Z           false
Table: keys: [_start, _stop, _field, _measurement, deviceName, resourceName]
_start:time                      _stop:time           _field:string     _measurement:string       deviceName:string     resourceName:string                      _time:time                  _value:float
------------------------------  ------------------------------  ----------------------  ----------------------  ----------------------  ----------------------  ------------------------------  ----------------------------
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                 Float32                readings     Random-Float-Device                 Float32  2022-08-05T11:27:22.196934000Z            -293.3783874511719
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                 Float32                readings     Random-Float-Device                 Float32  2022-08-05T11:27:52.197209000Z            -1479.512939453125
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                 Float32                readings     Random-Float-Device                 Float32  2022-08-05T11:28:22.198764000Z            -2948.382080078125
Table: keys: [_start, _stop, _field, _measurement, deviceName, resourceName]
_start:time                      _stop:time           _field:string     _measurement:string       deviceName:string     resourceName:string                      _time:time                  _value:float
------------------------------  ------------------------------  ----------------------  ----------------------  ----------------------  ----------------------  ------------------------------  ----------------------------
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                 Float64                readings     Random-Float-Device                 Float64  2022-08-05T11:27:22.196633000Z                     -2954.532
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                 Float64                readings     Random-Float-Device                 Float64  2022-08-05T11:27:52.196992000Z                      1619.041
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                 Float64                readings     Random-Float-Device                 Float64  2022-08-05T11:28:22.198991000Z                      732.3111
Table: keys: [_start, _stop, _field, _measurement, deviceName, resourceName]
_start:time                      _stop:time           _field:string     _measurement:string       deviceName:string     resourceName:string                      _time:time                  _value:int
------------------------------  ------------------------------  ----------------------  ----------------------  ----------------------  ----------------------  ------------------------------  --------------------------
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                   Int16                readings   Random-Integer-Device                   Int16  2022-08-05T11:27:22.200017000Z                       -2770
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                   Int16                readings   Random-Integer-Device                   Int16  2022-08-05T11:27:37.200241000Z                        -625
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                   Int16                readings   Random-Integer-Device                   Int16  2022-08-05T11:27:52.200819000Z                        -917
Table: keys: [_start, _stop, _field, _measurement, deviceName, resourceName]
_start:time                      _stop:time           _field:string     _measurement:string       deviceName:string     resourceName:string                      _time:time                  _value:int
------------------------------  ------------------------------  ----------------------  ----------------------  ----------------------  ----------------------  ------------------------------  --------------------------
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                   Int32                readings   Random-Integer-Device                   Int32  2022-08-05T11:27:22.199638000Z                       -3812
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                   Int32                readings   Random-Integer-Device                   Int32  2022-08-05T11:27:37.199731000Z                         897
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                   Int32                readings   Random-Integer-Device                   Int32  2022-08-05T11:27:52.200731000Z                        1435
Table: keys: [_start, _stop, _field, _measurement, deviceName, resourceName]
_start:time                      _stop:time           _field:string     _measurement:string       deviceName:string     resourceName:string                      _time:time                  _value:int
------------------------------  ------------------------------  ----------------------  ----------------------  ----------------------  ----------------------  ------------------------------  --------------------------
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                   Int64                readings   Random-Integer-Device                   Int64  2022-08-05T11:27:22.196715000Z                        -753
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                   Int64                readings   Random-Integer-Device                   Int64  2022-08-05T11:27:37.197740000Z                       -2381
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                   Int64                readings   Random-Integer-Device                   Int64  2022-08-05T11:27:52.198263000Z                       -1510
Table: keys: [_start, _stop, _field, _measurement, deviceName, resourceName]
_start:time                      _stop:time           _field:string     _measurement:string       deviceName:string     resourceName:string                      _time:time                  _value:int
------------------------------  ------------------------------  ----------------------  ----------------------  ----------------------  ----------------------  ------------------------------  --------------------------
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                    Int8                readings   Random-Integer-Device                    Int8  2022-08-05T11:27:22.198269000Z                         -56
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                    Int8                readings   Random-Integer-Device                    Int8  2022-08-05T11:27:37.199519000Z                          35
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                    Int8                readings   Random-Integer-Device                    Int8  2022-08-05T11:27:52.200629000Z                         -98
Table: keys: [_start, _stop, _field, _measurement, deviceName, resourceName]
_start:time                      _stop:time           _field:string     _measurement:string              deviceName:string     resourceName:string                      _time:time                  _value:uint
------------------------------  ------------------------------  ----------------------  ----------------------  -----------------------------  ----------------------  ------------------------------  ---------------------------
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                  Uint16                readings  Random-UnsignedInteger-Device                  Uint16  2022-08-05T11:27:32.200155000Z                         2319
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                  Uint16                readings  Random-UnsignedInteger-Device                  Uint16  2022-08-05T11:27:52.200666000Z                         1483
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                  Uint16                readings  Random-UnsignedInteger-Device                  Uint16  2022-08-05T11:28:12.200916000Z                          806
Table: keys: [_start, _stop, _field, _measurement, deviceName, resourceName]
_start:time                      _stop:time           _field:string     _measurement:string              deviceName:string     resourceName:string                      _time:time                  _value:uint
------------------------------  ------------------------------  ----------------------  ----------------------  -----------------------------  ----------------------  ------------------------------  ---------------------------
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                  Uint32                readings  Random-UnsignedInteger-Device                  Uint32  2022-08-05T11:27:32.199736000Z                         3408
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                  Uint32                readings  Random-UnsignedInteger-Device                  Uint32  2022-08-05T11:27:52.200581000Z                         4849
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                  Uint32                readings  Random-UnsignedInteger-Device                  Uint32  2022-08-05T11:28:12.201070000Z                         1474
Table: keys: [_start, _stop, _field, _measurement, deviceName, resourceName]
_start:time                      _stop:time           _field:string     _measurement:string              deviceName:string     resourceName:string                      _time:time                  _value:uint
------------------------------  ------------------------------  ----------------------  ----------------------  -----------------------------  ----------------------  ------------------------------  ---------------------------
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                  Uint64                readings  Random-UnsignedInteger-Device                  Uint64  2022-08-05T11:27:32.200580000Z                          256
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                  Uint64                readings  Random-UnsignedInteger-Device                  Uint64  2022-08-05T11:27:52.200702000Z                         1882
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                  Uint64                readings  Random-UnsignedInteger-Device                  Uint64  2022-08-05T11:28:12.200837000Z                           46
Table: keys: [_start, _stop, _field, _measurement, deviceName, resourceName]
_start:time                      _stop:time           _field:string     _measurement:string              deviceName:string     resourceName:string                      _time:time                  _value:uint
------------------------------  ------------------------------  ----------------------  ----------------------  -----------------------------  ----------------------  ------------------------------  ---------------------------
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                   Uint8                readings  Random-UnsignedInteger-Device                   Uint8  2022-08-05T11:27:32.200326000Z                          216
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                   Uint8                readings  Random-UnsignedInteger-Device                   Uint8  2022-08-05T11:27:52.200803000Z                           29
2022-08-05T11:23:48.345036205Z  2022-08-05T11:28:48.345036205Z                   Uint8                readings  Random-UnsignedInteger-Device                   Uint8  2022-08-05T11:28:12.200983000Z                          191

More configuration details about InfluxDBSyncWrite

InfluxDBSyncWrite

Uses the influxdb-client-go library to synchronously write EdgeX events to the InfluxDB

Sample configuration:

[Writable.Pipeline.Functions.InfluxDBSyncWrite]
  [Writable.Pipeline.Functions.InfluxDBSyncWrite.Parameters]
    InfluxDBServerURL = 'http://localhost:8086'
    InfluxDBOrganization = 'my-org'
    InfluxDBBucket = 'my-bucket'
    InfluxDBMeasurement = 'readings'
    InfluxDBValueType = 'integer'
    InfluxDBPrecision = 'us'
    authmode = 'token'
    secretpath='influxdb'
    skipverify = 'true'
    persistOnError = 'false'
    StoreEventTags = 'false'
    StoreReadingTags = 'false'
    FieldKeyPattern = '{resourceName}'

Configuration	Description
InfluxDBServerURL	Mandatory configuration to specify the InfluxDB Server address.
InfluxDBOrganization	The configuration setting to specify the InfluxDB Organization. For InfluxDB, an organization is a workspace for a group of users. An organization includes dashboards, tasks, buckets, and members. For more details about InfluxDB Organization, please refer to "Manage organizations" section in InfluxDB Documentation. Default value: `my-org`
InfluxDBBucket	The configuration setting to specify InfluxDB bucket. A bucket is a named location where time series data is stored in InfluxDB. To understand more details about InfluxDB bucket, please refer to "bucket" in "Glossary" section in InfluxDB Documentation. Default value: `my-bucket`
InfluxDBMeasurement	The configuration setting to specify InfluxDB measurement where EdgeX events shall write to. Measurement is the part of the InfluxDB data structure that describes the data stored in the associated fields. To understand more details about measurement, please refer to "measurement" in "Glossary" section in InfluxDB Documentation. Default value: `readings`
InfluxDBValueType	The configuration setting to specify what value type of events shall be written into InfluxDB. As each fields of InfluxDB measurement is restricted to accept one data type, you shall specify the corresponding data type that the target field of `InfluxDBMeasurement` will accept. Note that this configuration is effective only when `FieldKeyPattern` does not contain any replaceable variables. InfluxDBValueType only accepts `float`, `integer`, `uinteger`, `string`, or `boolean`. Default value: `integer`
InfluxDBPrecision	The configuration setting to specify how much timestamp precision is retained in the InfluxDB points. InfluxDB will truncate all incoming timestamps to the given precision. Per InfluxDB Documentation — Frequently Asked Questions, InfluxDB suggests to use coarsest possible timestamp precision when writing data to InfluxDB to maximize performance, so you can adjust the precision per your scenarios. InfluxDBPrecision only accepts `ns`(nanoseconds), `us`(microseconds), `ms`(millieseconds), or `s`(seconds). For more details about InfluxDB precision, please refer to InfluxDB documentation. Default value: `us`
authmode	Specify what authentication mode to write points to InfluxDB. As of release of Edge Xpert 1.7.1, `authmode` only accepts token for `token` authentication or none for no authentication. Default Value: `token`
secretpath	The name of the path in secret provider to retrieve secrets.
skipverify	Specify whether EdgeX app service should verify the server's certificate chain and host name. If `skipverify` is `true`, TLS accepts any certificate presented by the server and any host name in that certificate. In this mode, TLS is susceptible to man-in-the-middle attacks. This should be used only for testing.
PersistOnError	Specify whether the events should be stored for later retry if encountering any error during message publish. If `persistOnError` is `true` and `Store and Forward` is enabled, the data will be stored for later retry. See Store and Forward for more details.
StoreEventTags	Specify whether the events tags should be stored as InfluxDB tags. By default, InfluxDB tags only contain `deviceName` and `resourceName`. If `StoreEventTags` is `true`, the event tags added by Tags built-in functions will be stored to InfluxDB.
StoreReadingTags	Specify whether the reading tags should be stored as InfluxDB tags. By default, InfluxDB tags only contain `deviceName` and `resourceName`. If `StoreReadingTags` is `true`, the tags at the reading level will be stored to InfluxDB. Note that reading tags take higher precedence over event tags when both specifying the same tag key.
FieldKeyPattern	Specify the pattern of the field keys used to store reading values. FieldKeyPattern allows to contain replaceable variables, such as `{resourceName}`, `{deviceName}`, `{profileName}`, and `{valueType}`, which will be replaced during runtime execution. For example, specify FieldKeyPattern to `{profileName}_{resourceName}` will store the reading values with different field keys based on reading's profile name and device name. If `FieldKeyPattern` is not specified, InfluxDBSyncWrite will use `value` as the field key for all reading values.