Export to HTTP endpoints

Overview

To send data to HTTP endpoints, Edge Xpert provides a proprietary XpertHTTPExport function, which is similar to HTTPExport of EdgeX Foundry but with additional feature enhancements, such as OAuth2 Client Credentials grant. Moreover, designated HTTP endpoint may only accept the data payload in the certain format. To satisfy such needs, Edge Xpert provides an experimental function JavascriptTransform to execute the data transformation logic in the syntax of Javascript. Lastly, Edge Xpert also provides a proprietary SetContextVariable function, a handy feature to extract certain value from incoming data per specified GJsonPath and then set a corresponding context-level variable with extracted value. This example assumes that a device service will tag GPS coordination information into each EdgeX event with similar format as shown below:

{
    "apiVersion": "v2",
    "id": "040bd523-ec33-440d-9d72-e5813a465f37",
    "origin": 1602168089665565200,
    "deviceName": "device-001",
    "profileName": "profile-001",
    "sourceName": "source-1",
    "readings":
    [
        {
            "deviceName": "device-001",
            "id": "31569347-9369-43ec-aa6a-59ea9c624a6f",
            "modified": 1594975851631,
            "origin": 1602168089665565200,
            "profileName": "profile-001",
            "resourceName": "resource-001",
            "value": "39.5",
            "valueType": "Float32"
        },
        {
            "create": 1594876281221,
            "deviceName": "device-001",
            "id": "2fd73a5b-969f-483c-9c52-6bb460a06eb1",
            "origin": 1602168089665565200,
            "profileName": "profile-001",
            "resourceName": "resource-001",
            "value": "75",
            "valueType": "Int8"
        }
    ],
    "tags":
    {
        "latitude":"25.024063",
        "longitude":"121.522111"
    }
}

This example will create an application service to deal with such EdgeX event in following processing sequence:

When receiving an EdgeX event, extract the value of latitude from tags.latitude and then store the value to a context variable Latitude.
When receiving an EdgeX event, extract the value of longitude from tags.longitude and then store the value to a context variable Longitude.
Issue an HTTP GET request to https://postman-echo.com/get?latitude={Latitude}&longitude={Longitude} where {Latitude} is the value stored in context variable Latitude and {Longitude} is the value stored in context variable Longitude.
Transform the HTTP response as received from https://postman-echo.com/get?latitude={Latitude}&longitude={Longitude} to another designated JSON format using JavascriptTransform function.
Send the transformation result as body payload in an HTTP PATCH request to https://postman-echo.com/patch endpoint.

Note

This example will send data to Postman Echo, which is a service used to test REST clients.

Prerequisite

Edge Xpert CLI installation
Understand Edge Xpert Application Service
Launch Edge Xpert core services
Obtain following information of the target HTTP endpoint where data should export to
- URL
- HTTP Method
- MimeType
- Authentication mechanism
If the target HTTP endpoint requires OAuth2 client credential authentication, obtain following information from your OAuth2 tenant service provider:
- client id
- client secret
- token url
- token server certificate, if your OAuth2 token server uses self-signed certificate
- token scopes, if required
- additional parameters, if required, to obtain the oauth2 token

Configure an Application service to export events to an HTTP endpoint

We will need a configuration file to start up an application service. The recommended way is to copy the sample configuration as provided by Edge Xpert CLI, and then revise the configuration to satisfy your needs. Copy /usr/share/edgexpert/examples/app-configurable/javascriptTransform-xpertHTTPExport/configuration.toml sample configuration file to your working directory.
Open the copied configuration.toml file in a text editor and update parameters under [Writable.Pipeline.Functions.SetContextVariable_1.Parameters] section as shown below:
```
[Writable.Pipeline.Functions.SetContextVariable_1]
  [Writable.Pipeline.Functions.SetContextVariable_1.Parameters]
  VariableName = "Latitude"
  ValueJSONPath = "tags.latitude"
  ContinueOnError = "false"
```
Note

SetContextVariable sets the context variable according to the specified parameters. The name of context variable will be the value of parameter VariableName, and the value of context variable will be the value extracted from the specified ValueJSONPath of data as passed into the function. Set ContinueOnError to true if users would like the pipeline to continue even when there is error during the execution of SetContextVariable. When ContinueOnError is true and error occurs during the execution of SetContextVariable, the function will return incoming data rather than error. If incoming data doesn't contain path of specified ValueJSONPath, the context variable will be set to empty string "".

Update parameters under [Writable.Pipeline.Functions.SetContextVariable_2.Parameters] section as shown below:

[Writable.Pipeline.Functions.SetContextVariable_2]
  [Writable.Pipeline.Functions.SetContextVariable_2.Parameters]
  VariableName = "Longitude"
  ValueJSONPath = "tags.longitude"
  ContinueOnError = "false"

Add a new parameter section [Writable.Pipeline.Functions.XpertHTTPExport_GET] and update parameters under [Writable.Pipeline.Functions.XpertHTTPExport_GET.Parameters] section as shown below:
```
[Writable.Pipeline.Functions.XpertHTTPExport_GET]
  [Writable.Pipeline.Functions.XpertHTTPExport_GET.Parameters]
  Method = "GET"
  MimeType = "application/json"
  Url = "https://postman-echo.com/get?latitude={Latitude}&longitude={Longitude}"
  PersistOnError = "false"
  ContinueOnSendError = "false"
  ReturnInputData = "false"
  AuthMode = "none"
```
Note

This sample demonstrates the case that the target URL doesn't require any authentication, so AuthMode is set to none. If your designated HTTP endpoint require OAuth2 Client Credential grant, you have to specify AuthMode to oauth2_clientcredentials. When AuthMode is oauth2_clientcredentials, XpertHTTPExport will look for mandatory secrets--client_id, client_secret, and token_url--from secret provider when obtaining OAuth2 access token. If your Edge Xpert is running under secure mode, follow instruction as described in Run Application Services in Secure Mode to add your mandatory secrets into secret store. If your Edge Xpert is running under insecure mode, simply edit the Writable.InsecureSecrets.http configuration as shown below:
```
[Writable.InsecureSecrets.http]
path = "http"
  [Writable.InsecureSecrets.http.Secrets]
  client_id = "{CLIENT_ID}"
  client_secret = "{CLIENT_SECRET}"
  token_scopes = "{TOKEN_SCOPES}"
  token_url = "{TOKEN_URL}"
```
If obtaining oauth2 access token requires scopes, you must specify optional secret token_scopes in the secret provider. Multiple scopes can be provided with comma separator, for example, "scope1,scope2". Additional request parameters required to obtain OAuth2 access token can also be specified in the secret provider.
If your oauth2 token server uses self-signed certificate that is not signed by a trusty authority, to ensure that your app service can successfully obtain the token, you will have to specify secret token_cert with the self-signed token server certificate in the secret provider. If obtaining the self-signed token server certificate is not possible, you can also specify SkipVerify = "true" in your XpertHTTPExport function configuration.

Update parameters under [Writable.Pipeline.Functions.JavascriptTransform.Parameters] section as shown below:

[Writable.Pipeline.Functions.JavascriptTransform]
  [Writable.Pipeline.Functions.JavascriptTransform.Parameters]
  # TransformScript defines the transformation logic in the valid javascript syntax
  TransformScript = "var outputObject = { responseUserAgent: inputObject.headers[\"user-agent\"] };\n    return outputObject;"

Note

When preparing the TransformScript, you can expect an embedded javascript object inputObject available for representing the data as passed into the JavascriptTransform function. The response from https://postman-echo.com/get?latitude={Latitude}&longitude={Longitude} will be in following format:

{
  "args": {
    "latitude": "25.024063",
    "longitude": "121.522111"
  },
  "headers": {
    "x-forwarded-proto": "https",
    "x-forwarded-port": "443",
    "host": "postman-echo.com",
    "x-amzn-trace-id": "Root=1-61f007ec-3313b0cc762d84010b3a7e7c",
    "user-agent": "PostmanRuntime/7.26.8",
    "accept": "*/*",
    "postman-token": "88452a27-106b-44ca-a19d-3e2ea9c3ad84",
    "accept-encoding": "gzip, deflate, br",
    "cookie": "sails.sid=s%3AfMLtC0tDqM-qPbXkb_t_NGOkipK297sg.ojNajoujC5cUK11tzbJiNREdZEF5uMI4KEBxsfy904w"
  },
  "url": "https://postman-echo.com/get?latitude=25.024063&longitude=121.522111"
}

In our sample script above, we simply extract the value of headers["user-agent"] and then assign to a field responseUserAgent, so the transformation result will look like following:

{
  "responseUserAgent" : "PostmanRuntime/7.26.8"
}

Add a new parameter section [Writable.Pipeline.Functions.XpertHTTPExport_PATCH] and update parameters under [Writable.Pipeline.Functions.XpertHTTPExport_PATCH.Parameters] section as shown below:

[Writable.Pipeline.Functions.XpertHTTPExport_PATCH]
  [Writable.Pipeline.Functions.XpertHTTPExport_PATCH.Parameters]
  Method = "PATCH"
  MimeType = "application/json"
  Url = "https://postman-echo.com/patch"
  PersistOnError = "false"
  ContinueOnSendError = "false"
  ReturnInputData = "false"
  AuthMode = "none"

Specify the execution order of the pipeline function, as shown in the following example:

[Writable.Pipeline]
 ExecutionOrder = "SetContextVariable_1,SetContextVariable_2,XpertHTTPExport_GET,JavascriptTransform,XpertHTTPExport_PATCH"

In order to monitor events sent to designated HTTP endpoint, update the log level to DEBUG, as shown in the following example:
```
[Writable]
  LogLevel = "DEBUG"
```
Now, the application service is ready to start up. If your Edge Xpert is running under secure mode, enter the following command, replacing the path with your configuration file:
```
edgexpert up --secret app-service --path=<path your configuration file>
```
If your Edge Xpert is running under insecure mode, enter the following command, replacing the path with your configuration file:
```
edgexpert up app-service --path=<path your configuration file>
```

Trigger the function pipeline from an HTTP request via the curl command or API tool and post the above example as the body.

curl command

curl -d "@data.json" -H "Content-Type: application/json" -X POST http://<the IP address of app-service>:59704/api/v2/trigger

API tool

http://<the IP address of app-service>:59704/api/v2/trigger

Monitor the log from the running app service. If everything works fine, you should observe similar log as shown below:

level=DEBUG ts=2022-01-25T15:22:32.041239076Z app=app-javascriptTransform-xpertHTTPExport-test source=context.go:29 msg="Setting context variable Latitude in pipeline 'default-pipeline'"
level=DEBUG ts=2022-01-25T15:22:32.041321969Z app=app-javascriptTransform-xpertHTTPExport-test source=context.go:29 msg="Setting context variable Longitude in pipeline 'default-pipeline'"
level=DEBUG ts=2022-01-25T15:22:32.041377297Z app=app-javascriptTransform-xpertHTTPExport-test source=http.go:135 msg="HTTP Exporting in pipeline 'default-pipeline'"
level=DEBUG ts=2022-01-25T15:22:32.041421167Z app=app-javascriptTransform-xpertHTTPExport-test source=http.go:160 msg="Issuing HTTP GET request to https://postman-echo.com/get?latitude=25.024063&longitude=121.522111 in pipeline 'default-pipeline'"
level=DEBUG ts=2022-01-25T15:22:33.085751307Z app=app-javascriptTransform-xpertHTTPExport-test source=http.go:187 msg="Sent 627 bytes of data in pipeline 'default-pipeline'. Response status is 200 OK"
level=DEBUG ts=2022-01-25T15:22:33.08618545Z app=app-javascriptTransform-xpertHTTPExport-test source=conversion.go:228 msg="Prepare to run JavascriptTransform in pipeline 'default-pipeline'"
level=DEBUG ts=2022-01-25T15:22:33.086300967Z app=app-javascriptTransform-xpertHTTPExport-test source=conversion.go:240 msg="JavascriptTransform in pipeline 'default-pipeline' receives data: {\"args\":{\"latitude\":\"25.024063\",\"longitude\":\"121.522111\",\"apiVersion\":\"v2\",\"id\":\"040bd523-ec33-440d-9d72-e5813a465f37\",\"deviceName\":\"device-001\",\"profileName\":\"profile-001\",\"sourceName\":\"source-1\",\"origin\":1602168089665565200,\"readings\":[{\"id\":\"31569347-9369-43ec-aa6a-59ea9c624a6f\",\"origin\":1602168089665565200,\"deviceName\":\"device-001\",\"resourceName\":\"resource-001\",\"profileName\":\"profile-001\",\"valueType\":\"Float32\",\"value\":\"39.5\"},{\"id\":\"2fd73a5b-969f-483c-9c52-6bb460a06eb1\",\"origin\":1602168089665565200,\"deviceName\":\"device-001\",\"resourceName\":\"resource-001\",\"profileName\":\"profile-001\",\"valueType\":\"Int8\",\"value\":\"75\"}],\"tags\":{\"latitude\":\"25.024063\",\"longitude\":\"121.522111\"}},\"headers\":{\"x-forwarded-proto\":\"https\",\"x-forwarded-port\":\"443\",\"host\":\"postman-echo.com\",\"x-amzn-trace-id\":\"Root=1-61f015b8-100c6d420dea7c3166c40f9a\",\"content-length\":\"627\",\"content-type\":\"application/json\",\"accept-encoding\":\"gzip\",\"user-agent\":\"Go-http-client/2.0\"},\"url\":\"https://postman-echo.com/get?latitude=25.024063&longitude=121.522111\"}"
level=DEBUG ts=2022-01-25T15:22:33.091007028Z app=app-javascriptTransform-xpertHTTPExport-test source=conversion.go:271 msg="result returned from running script: {\"responseUserAgent\":\"Go-http-client/2.0\"}"
level=DEBUG ts=2022-01-25T15:22:33.091189612Z app=app-javascriptTransform-xpertHTTPExport-test source=http.go:135 msg="HTTP Exporting in pipeline 'default-pipeline'"
level=DEBUG ts=2022-01-25T15:22:33.09132988Z app=app-javascriptTransform-xpertHTTPExport-test source=http.go:160 msg="Issuing HTTP PATCH request to https://postman-echo.com/patch in pipeline 'default-pipeline'"
level=DEBUG ts=2022-01-25T15:22:33.329903212Z app=app-javascriptTransform-xpertHTTPExport-test source=http.go:187 msg="Sent 42 bytes of data in pipeline 'default-pipeline'. Response status is 200 OK"

More configuration details about XpertHTTPExport

Configuration	Description
Method	Mandatory configuration to specify HTTP Method. Valid values are `GET`, `POST`, `PUT`, `PATCH` or `DELETE`.
MimeType	Mandatory configuration to specify the MimeType in a HTTP request sending to destination.
Url	Mandatory configuration to specify the URL of destination. Url allows to contain placeholder text, {key-name}, which will be replaced with any matching context variable found during runtime execution. For example, specify `Url` to `http://myhost.com/edgex-events/{devicename}` will send requests to different endpoints base on their device name.
PersistOnError	Specify whether the events should be stored for later retry if encountering any error during message publish. Default value is `false`. If persisOnError is `true` and Store and Forward is enabled, the data will be stored for later retry. See Store and Forward for more details.
ContinueOnSendError	Specify ContinueOnSendError to `true` will allow execution of subsequent chained senders after errors. Default value is `false`. *Note that ContinueOnSendError=true can only be used when ReturnInputData=true and cannot be use when PersistOnError=true*
ReturnInputData	ReturnInputData enables chaining multiple HTTP senders if true. Default value is `false`.
SecretPath	Specify the name of the path in secret provider to retrieve secrets.
AuthMode	Specify the authentication mode for accessing the designated Url. Valid values are `none`, `header_secret`, and `oauth2_clientcredentials`. Default value is `none`. Set `AuthMode` to `none` when there is no authentication required. Set `AuthMode` to `header_secret` when the designated Url requires specific value to be included in the header of each HTTP request. When `AuthMode` is `header_secret`, `SecretPath`, `HeaderName`, and `SecretName` must be specified. Set `AuthMode` to `oauth2_clientcredentials` when the designated Url requires OAuth2 client credentials grant. When `AuthMode` is `oauth2_clientcredentials`, `SecretPath` must be specified and manadatory secrets--`client_id`,`client_secret`,`token_url`--are expected to be retrieved from secret provider. Optional secrets--`token_scopes` and addiitonal parameters--should be added into secret provider when required for obtaining OAuth2 access token. Set `AuthMode` to `aws_signature` when the designated Url is Amazon REST API. When `AuthMode` is `aws_signature`, `SecretPath` and `AWSV4SignerConfigs` must be specified. `AWSV4SignerConfigs` is a json string which must include `region` and `service`.
HeaderName	Specify the name of the header key to add to the HTTP header. Only take effect when `AuthMode` is `header_secret`.
SecretName	Specify name of the secret for the header value in the secret store. Only take effect when `AuthMode` is `header_secret`.
SkipVerify	Specify whether to verify the server's certificate chain and host name. If `Auth_Mode` is `oauth2_clientcredentials`, this configuration will also apply to the oauth2 token server when the app service obtains the oauth2 access token.
HTTPRequestHeaders	Specify the HTTP request headers. See the example in `/usr/share/edgexpert/examples/app-configurable/javascriptTransform-xpertHTTPExport/configuration.toml`.
AWSV4SignerConfigs	Specify the `region` and `service` for AWS signature calculation. See the examples under `/usr/share/edgexpert/examples/app-configurable/awss3-http-export`.

Note

If the designated HTTP endpoints requires TLS with Self-Signed CA certificate, you have to add cacert secret with the PEM format of self-signed certificate into secret store. Otherwise, you may experience x509: certificate signed by unknown authority error while sending data to the target HTTP endpoint.
If Auth_Mode is oauth2_clientcredentials and the oauth2 token server requires TLS with Self-Signed CA certificate, you have to add token_cert secret with the PEM format of self-signed token server certificate into secret store. Otherwise, you may experience x509: certificate signed by unknown authority error while obtaining access token from the oauth2 token server.