Export to Google IoT Core

Overview

The Google Cloud IoT Core is a fully managed service to easily and securely connect, manage, and ingest data from globally dispersed devices. This page will describe how customers can configure an application service to use proprietary built-in function XpertMQTTExport to export EdgeX events to Google Cloud IoT Core.

Note

In Edge Xpert V2, GoogleCloudIoTCoreMQTTSend function is replaced by XpertMQTTExport function.

Prerequisites

Before starting this example, you must have the following:

A Google account
Create a Google Cloud project - make a note of the Project ID, this information is needed when configuring Application Service.
Create a Cloud Pub/Sub topic – Google Cloud IoT Core routes device messages to Cloud Pub/Sub for aggregation
Create a device registry – a device registry is a container of devices. The Pub/Sub topic as created in step 3 will be used in this step.
Create a device - currently supports RS256_X509 and ES256_X509 public key formats for device authentication. For testing purpose, you can use openssl utility below to generate public/private key pairs for self-signed X.509 certificates:

For RS256_X509:
```
openssl req -x509 -nodes -newkey rsa:2048 -keyout rsa_private.pem -out rsa_cert.pem -subj "/CN=unused"
```
!!! note In this case, you have to add ras_cert.pem as public key on Google Cloud IoT Core when creating a device.

For ES256_X509:

openssl req -x509 -new -key ec_private.pem -out ec_cert.pem -subj "/CN=unused"

Understand Edge Xpert Application Service
Install Edge Xpert
Launch core services

Sample Configuration

The following code defines the pipeline function XpertMQTTExport:

[Writable.Pipeline.Functions.XpertMQTTExport]
  [Writable.Pipeline.Functions.XpertMQTTExport.Parameters]
  ExportMode = "GoogleCloudIoT"
  BrokerAddress = "tls://mqtt.googleapis.com:8883"
  GoogleCloudProjectID = "{Project_ID}"
  GoogleCloudIoTJWTAlgorithm = "RS256"
  GoogleCloudIoTJWTExpiration = "24h"
  Topic = "/devices/{DEVICE_ID}/events"
  ClientID = "projects/{Project_ID}/locations/{REGION}/registries/{REGISTRY_ID}/devices/{DEVICE_ID}"
  SecretPath = "google"
  Qos = "0"
  AutoReconnect = "true"
  SkipVerify = "false"
  PersistOnError = "false"

The following table describes the configuration parameters used to define the pipeline function XpertMQTTExport to send data to Google Cloud IoT Core:

Configuration	Mandatory	Description
ExportMode	Yes	Must specify `GoogleCloudIoT` to indicate that the data should be exported to Google Cloud IoT Core.
BrokerAddress	Yes	Specifies the MQTT broker address on the Google Cloud IoT Core. At the time of publication, Google defines two MQTT broker addresses: If your system trusts Certificate Authorities as listed in https://pki.goog/roots.pem, use address below: tls://mqtt.googleapis.com:8883 If your systems have memory contraints and need to use minimal Google root CA set, use address below: tls://mqtt.2030.ltsapis.goog:8883 For further information on Google Cloud MQTT broker, please refer to the Google Cloud IoT Guides.
Topic	Yes	Specifies the MQTT topic of Google Cloud IoT Core to publish the message to. At the time of publication, Google requires devices to publish the messages to the topic as defined in the format below: /devices/{DEVICE_ID}/events Note that {Device_ID} is the device ID as registered on Google Cloud IoT Core. For further information, please refer to Google Cloud IoT Core Documentation.
ClientID	Yes	Specifies the MQTT Client ID. Per Google Cloud IoT Guides, the MQTT client ID should be set to the full device path in following format: projects/{PROJECT_ID}/locations/{REGION}/registries/{REGISTRY_ID}/devices/{DEVICE_ID} {PROJECT_ID} is the ID of your Google Cloud project. {REGION} is one of the properties of the device registry, it determines where data is stored for devices in this registry. {REGISTRY_ID} is the ID of the device registry. The device registry is a container of devices. {DEVICE_ID} is the ID of the device.
SecretPath	Yes	Specifies the path in the secret provider from which to retrieve the certificate/private key pair of your Google Cloud IoT device, and Google root CA certificate. Note: Secrets under this path are key-value pairs. By the current design of Application Service, users must add their secrets with the following key names with values in PEM format: `clientcert`: the device X.509 certificate. `clientkey`: the device private key. `googlerootcacert`: the Google root CA set. Specify this secret only when the Google root CA set does not exist in your system root CA pool. If your system has memory constraints and use the minimal Google root CA set (gtsltsr.crt), use the following command to convert it to PEM format: `openssl x509 -in gtsltsr.crt -inform der -out gtsltsr.pem -outform PEM`
GoogleCloudProjectID	Yes	Specifies the ID of Google Cloud project.
GoogleCloudIoTJWTAlgorithm		Specifies the algorithm used to generate a JSON Web Token to authenticate to Google Cloud IoT Core. Cloud IoT Core supports the following signing algorithms: `RS256` `ES256` The default value is `RS256`.
GoogleCloudIoTJWTExpiration		Specifies the expiration time in ISO8601 format for the JSON Web Token. The maximum lifetime of a token is 24 hours + skew. For further information on JWT Expiration time, refer to JWT claims. The default value is `24H`.
Qos		Specifies the Quality of Service (QoS) levels for the Google Cloud IoT Core MQTT connection. At the time of publication, Google Cloud IoT Core only accepts QoS 0 and 1. For further information on MQTT QoS, refer to Google Cloud IoT Guides.
AutoReconnect		Specifies whether the connection is to be automatically reconnected if it is lost.
SkipVerify		Specifies whether the Edge Xpert Application Service verifies the server's certificate chain and host name. If set to true, TLS accepts any certificate presented by the server and any host name in that certificated. This means that TLS is susceptible to man-in-the middle attacks. Use this setting for testing purposes only.
PersistOnError		Specifies whether the events are stored for further attempts if any error is encountered during the message publication. Note: To use persistOnError, Store and Forward must be enabled in the Application Service configuration file. For further information on Store and Forward, refer to the Advanced Topics under the Application Functions SDK section of the EdgeX Foundry documentation. Set to true to enable Set to false to disable

Note

The application service can be running in either secure or insecure mode. When the app service is running under secure mode, secrets will be retrieved from Vault secret store; when the app service is running under insecure mode, secrets will be retrieved from Writable.InsecureSecrets configuration.

Configure an Application service to export events to Google Cloud IoT Core

By end of step 5 in Prerequisite, customers shall expect to obtain Google Cloud Project ID, Region, Registry ID, Device ID, and X.509 Certificate/Private Key. Keep a note on these values, as you will need these values to authenticate with Google Cloud IoT Core.
Edge Xpert offers a proprietary built-in function XpertMQTTExport to send the events to Google Cloud IoT Core via MQTT, and the sample configuration is provided with Edge Xpert CLI utility. Once customers completes the installation of Edge Xpert CLI utility, the sample configuration is available at /usr/share/edgexpert/examples/app-configurable/googlecloudiot-mqtt-export/. Copy and save the sample configuration file as a separate file, e.g. googlecloudiot-mqtt-export.toml, and then revise this configuration per values of Google Cloud Project ID, Region, Registry ID, Device ID, and X.509 Certificate/Private Key as obtained earlier. For example, if users have Google Cloud Project ID:iotech-google, Region: asia-east1, Registry ID: TestRegistry, Device ID: TestDevice, user shall revise XpertMQTTExport with following parameters:
```
[Writable.Pipeline.Functions.XpertMQTTExport]
  [Writable.Pipeline.Functions.XpertMQTTExport.Parameters]
  ExportMode = "GoogleCloudIoT"
  BrokerAddress = "tls://mqtt.googleapis.com:8883"
  GoogleCloudProjectID = "iotech-google"
  GoogleCloudIoTJWTAlgorithm = "RS256"
  GoogleCloudIoTJWTExpiration = "24h"
  Topic = "/devices/TestDevice/events"
  ClientID = "projects/iotech-google/locations/asia-east1/registries/TestRegistry/devices/TestDevice"
  SecretPath = "google"
  Qos = "0"
  AutoReconnect = "true"
  SkipVerify = "false"
  PersistOnError = "false"
```
Note

Users will find two sample configuration files under /usr/share/edgexpert/examples/app-configurable/googlecloudiot-mqtt-export/. If the application service will be running in insecure mode, use configuration-insecure.toml as the sample. If the application service will be running in secure mode, use configuration-secretstore.toml as the sample.

If the application service will be launched under insecure mode, secrets will be retrieved from Writable.InsecureSecrets configuration. Users will need to update Writable.InsecureSecrets.google configuration with correct values of clientcert, and clientkey:

 [Writable.InsecureSecrets.google]
 path = "google"
   [Writable.InsecureSecrets.google.Secrets]
   # When using XpertMQTTExport and ExportMode="GoogleCloudIoT", clientcert and clientkey are mandatory secrets.
   clientcert = '''-----BEGIN CERTIFICATE-----
   MIIDBTCCAe2gAwIBAgIUINqAgewYIJnHFwQXOJXORgDBIDkwDQYJKoZIhvcNAQEL
   -----END CERTIFICATE-----'''
   clientkey = '''-----BEGIN PRIVATE KEY-----
   MIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQDVxFJCQvmARt6f
   -----END PRIVATE KEY-----'''
   # When using XpertMQTTExport, ExportMod="GoogleCloudIoT", SkipVerify="false" and the Google root CA set doesn’t exist in your system root pool, googlerootcacert is a required secret.
   # For further information on Google root CA set, refer to https://cloud.google.com/iot/docs/how-tos/mqtt-bridge#downloading_mqtt_server_certificates.
   #googlerootcacert = '''
   #{Google_root_CA_certificate}
   #'''

Note

When using XpertMQTTExport with ExportMode="GoogleCloudIoT" , SkipVerify="false" and the Google root CA set does not exist in the system root CA pool, googlerootcacert is a required secret. For further information on Google root CA set, refer to the documentation.

Once configuration is ready, users can launch the application service in secure mode using command below:
```
$ edgexpert up --secret app-service --path=<path of the confgiuration file>
```
If users prefer to launch the application service in insecure mode, use command below:
```
$ edgexpert up app-service --path=<path of the confgiuration file>
```
If the application service is launched under secure mode on previous step, secrets will be retrieved from Vault secret store. Users need to add clientkey, and clientcert into Vault secret store. Please refer to Edge Xpert Application Services for details about adding secrets into Vault secret store.
Note

Since Edge Xpert V2.1, IOTech recommends users to add secrets through the POST secrets REST API of application service. Follow steps below to add secrets under secure mode.

1. Inspect the IPAddress of the application service as launched in previous step:
```
$ docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' app-configuration-secretstore
192.168.80.10
```
2. Users may find it's difficult to properly add the content of X.509 PEM files in valid format in a REST request. Use following awk commands to convert the content of X.509 PEM files as a string:
```
$ awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' rsa_cert.pem
-----BEGIN CERTIFICATE-----\nMIIDBTCCAe2gAwIBAgIUINqAgewYIJnHFwQXOJXORgDBIDkwDQYJKoZIhvcNAQEL\n-----END CERTIFICATE-----\n
$ awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' rsa_private.pem
-----BEGIN PRIVATE KEY-----\nMIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQDVxFJCQvmARt6f\n-----END PRIVATE KEY-----\n
```
Make a note on the console output of both awk commands result.
3. Insert the copied string into the payload of REST call to add secrets. Note that all necessary secrets must be added in a single REST call, otherwise the secrets content will be overwritten by REST calls afterwards. For example, following curl command adds two secrets--clientcert, and clientkey--under the secret path google in the Vault secret store:
```
curl -k -X POST -d '{ "apiVersion": "v2", "path": "google", "secretData": [ {"key":"clientcert","value":"-----BEGIN CERTIFICATE-----\nMIIDBTCCAe2gAwIBAgIUINqAgewYIJnHFwQXOJXORgDBIDkwDQYJKoZIhvcNAQEL\n-----END CERTIFICATE-----\n"},{"key":"clientkey","value":"-----BEGIN PRIVATE KEY-----\nMIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQDVxFJCQvmARt6f\n-----END PRIVATE KEY-----\n"}]}' http://192.168.80.10:59700/api/v2/secret
```
Now, an application service is up and ready to export events to Google Cloud IoT Core. To simulate the device sending readings from southbound, start up the virtual device service in secure mode using command below:
```
$ edgexpert up --secret device-virtual
```
If users prefer to launch the virtual device service in insecure mode, use command below:
```
$ edgexpert up device-virtual
```
Monitor the events on the Google Cloud Platform. Navigate to Pub/Sub > Subscriptions page, create a subscription for the Pub/Sub topic used by your Device Registry, then select this subscription and click VIEW MESSAGES on the top navigation bar. Ideally, you should expect to see EdgeX messages in the Message body column: