Skip to content

Export to HTTP endpoints

Overview

To send data to HTTP endpoints, Edge Xpert provides a proprietary XpertHTTPExport function, which is similar to HTTPExport of EdgeX Foundry but with additional feature enhancements, such as OAuth2 Client Credentials grant. Moreover, designated HTTP endpoint may only accept the data payload in certain format. To satisfy such needs, Edge Xpert provides an experimental function JavascriptTransform to execute the data transformation logic in the syntax of Javascript. Lastly, Edge Xpert also offers a proprietary SetContextVariable function, a handy feature to extract certain value from incoming data per specified GJsonPath and then set a corresponding context-level variable with extracted value. This example assumes that a device service will tag GPS coordination information into each EdgeX event with similar format as shown below:

{
    "apiVersion": "v2",
    "id": "040bd523-ec33-440d-9d72-e5813a465f37",
    "origin": 1602168089665565200,
    "deviceName": "device-001",
    "profileName": "profile-001",
    "sourceName": "source-1",
    "readings":
    [
        {
            "deviceName": "device-001",
            "id": "31569347-9369-43ec-aa6a-59ea9c624a6f",
            "modified": 1594975851631,
            "origin": 1602168089665565200,
            "profileName": "profile-001",
            "resourceName": "resource-001",
            "value": "39.5",
            "valueType": "Float32"
        },
        {
            "create": 1594876281221,
            "deviceName": "device-001",
            "id": "2fd73a5b-969f-483c-9c52-6bb460a06eb1",
            "origin": 1602168089665565200,
            "profileName": "profile-001",
            "resourceName": "resource-001",
            "value": "75",
            "valueType": "Int8"
        }
    ],
    "tags":
    {
        "latitude":"25.024063",
        "longitude":"121.522111"
    }
}
This example will create an application service to deal with such EdgeX event in following processing sequence:

  1. When receiving an EdgeX event, extract the value of latitude from tags.latitude and then store the value to a context variable Latitude.
  2. When receiving an EdgeX event, extract the value of longitude from tags.longitude and then store the value to a context variable Longitude.
  3. Issue an HTTP GET request to https://postman-echo.com/get?latitude={Latitude}&longitude={Longitude} where {Latitude} is the value stored in context variable Latitude and {Longitude} is the value stored in context variable Longitude.
  4. Transform the HTTP response as received from https://postman-echo.com/get?latitude={Latitude}&longitude={Longitude} to another designated JSON format using JavascriptTransform function.
  5. Send the transformation result as body payload in an HTTP PATCH request to https://postman-echo.com/patch endpoint.

Note

This example will send data to Postman Echo, which is a service used to test REST clients.

Prerequisite

  1. Edge Xpert CLI installation
  2. Understand Edge Xpert Application Service
  3. Launch Edge Xpert core services
  4. Obtain following information of the target HTTP endpoint where data should export to
    • URL
    • HTTP Method
    • MimeType
    • Authentication mechanism
  5. If the target HTTP endpoint requires OAuth2 client credential authentication, obtain following information from your OAuth2 tenant service provider:
    • client id
    • client secret
    • token url
    • token scopes, if required
    • additional parameters, if required, to obtain the oauth2 token

Configure an Application service to export events to an HTTP endpoint

  1. We will need a configuration file to start up an application service. The recommended way is to copy the sample configuration as provided by Edge Xpert CLI, and then revise the configuration to satisfy your needs. Copy /usr/share/edgexpert/examples/app-configurable/javascriptTransform-xpertHTTPExport/configuration.toml sample configuration file to your working directory.
  2. Open the copied configuration.toml file in a text editor and update parameters under [Writable.Pipeline.Functions.SetContextVariable_1.Parameters] section as shown below:

    [Writable.Pipeline.Functions.SetContextVariable_1]
      [Writable.Pipeline.Functions.SetContextVariable_1.Parameters]
      VariableName = "Latitude"
      ValueJSONPath = "tags.latitude"
      ContinueOnError = "false"
    

    Note

    SetContextVariable sets the context variable per specified parameters. The name of context variable will be the value of parameter VariableName, and the value of context variable will be the value extracted from the specified ValueJSONPath of data as passed into the function. Set ContinueOnError to true if users would like the pipeline to continue even when there is error during the execution of SetContextVariable. When ContinueOnError is true and error occurs during the execution of SetContextVariable, the function will return incoming data rather than error. If incoming data doesn't contain path of specified ValueJSONPath, the context variable will be set to empty string "".

  3. Update parameters under [Writable.Pipeline.Functions.SetContextVariable_2.Parameters] section as shown below:

    [Writable.Pipeline.Functions.SetContextVariable_2]
      [Writable.Pipeline.Functions.SetContextVariable_2.Parameters]
      VariableName = "Longitude"
      ValueJSONPath = "tags.longitude"
      ContinueOnError = "false"
    

  4. Add a new parameter section [Writable.Pipeline.Functions.XpertHTTPExport_GET] and update parameters under [Writable.Pipeline.Functions.XpertHTTPExport_GET.Parameters] section as shown below:

    [Writable.Pipeline.Functions.XpertHTTPExport_GET]
      [Writable.Pipeline.Functions.XpertHTTPExport_GET.Parameters]
      Method = "GET"
      MimeType = "application/json"
      Url = "https://postman-echo.com/get?latitude={Latitude}&longitude={Longitude}"
      PersistOnError = "false"
      ContinueOnSendError = "false"
      ReturnInputData = "false"
      SecretPath = "http"
      AuthMode = "none"
    

    Note

    This sample demonstrates the case that the target URL doesn't require any authentication, so AuthMode is set to none. If your designated HTTP endpoint require OAuth2 Client Credential grant, you have to specify AuthMode to oauth2_clientcredentials. When AuthMode is oauth2_clientcredentials, XpertHTTPExport will look for mandatory secrets--client_id, client_secret, and token_url--from secret provider when obtaining OAuth2 access token. If your Edge Xpert is running under secure mode, follow instruction as described in Run Application Services in Secure Mode to add your mandatory secrets into secret store. If your Edge Xpert is running under insecure mode, simply edit the Writable.InsecureSecrets.http configuration as shown below:

    [Writable.InsecureSecrets.http]
    path = "http"
      [Writable.InsecureSecrets.http.Secrets]
      client_id = "{CLIENT_ID}"
      client_secret = "{CLIENT_SECRET}"
      token_scopes = "{TOKEN_SCOPES}"
      token_url = "{TOKEN_URL}"
    
    If obtaining oauth2 access token requires scopes, user must specify optional secret token_scopes in the secret provider. Multiple scopes can be provided with comma separator, e.g. "scope1,scope2". Additional request parameters required to obtain OAuth2 access token can also be specified in the secret provider.

  5. Update parameters under [Writable.Pipeline.Functions.JavascriptTransform.Parameters] section as shown below:

    [Writable.Pipeline.Functions.JavascriptTransform]
      [Writable.Pipeline.Functions.JavascriptTransform.Parameters]
      # TransformScript defines the transformation logic in the valid javascript syntax
      TransformScript = "var outputObject = { responseUserAgent: inputObject.headers[\"user-agent\"] };\n    return outputObject;"
    

    Note

    When preparing the TransformScript, you can expect an embedded javascript object inputObject available for representing the data as passed into the JavascriptTransform function. The response from https://postman-echo.com/get?latitude={Latitude}&longitude={Longitude} will be in following format:

    {
      "args": {
        "latitude": "25.024063",
        "longitude": "121.522111"
      },
      "headers": {
        "x-forwarded-proto": "https",
        "x-forwarded-port": "443",
        "host": "postman-echo.com",
        "x-amzn-trace-id": "Root=1-61f007ec-3313b0cc762d84010b3a7e7c",
        "user-agent": "PostmanRuntime/7.26.8",
        "accept": "*/*",
        "postman-token": "88452a27-106b-44ca-a19d-3e2ea9c3ad84",
        "accept-encoding": "gzip, deflate, br",
        "cookie": "sails.sid=s%3AfMLtC0tDqM-qPbXkb_t_NGOkipK297sg.ojNajoujC5cUK11tzbJiNREdZEF5uMI4KEBxsfy904w"
      },
      "url": "https://postman-echo.com/get?latitude=25.024063&longitude=121.522111"
    }
    
    In our sample script above, we simply extract the value of headers["user-agent"] and then assign to a field responseUserAgent, so the transformation result will look like following:
    {
      "responseUserAgent" : "PostmanRuntime/7.26.8"
    }
    

  6. Add a new parameter section [Writable.Pipeline.Functions.XpertHTTPExport_PATCH] and update parameters under [Writable.Pipeline.Functions.XpertHTTPExport_PATCH.Parameters] section as shown below:

    [Writable.Pipeline.Functions.XpertHTTPExport_PATCH]
      [Writable.Pipeline.Functions.XpertHTTPExport_PATCH.Parameters]
      Method = "PATCH"
      MimeType = "application/json"
      Url = "https://postman-echo.com/patch"
      PersistOnError = "false"
      ContinueOnSendError = "false"
      ReturnInputData = "false"
      SecretPath = "http"
      AuthMode = "none"
    

  7. Specify the execution order of the pipeline function, as shown in the following example:

    [Writable.Pipeline]
     ExecutionOrder = "SetContextVariable_1,SetContextVariable_2,XpertHTTPExport_GET,JavascriptTransform,XpertHTTPExport_PATCH"
    

  8. In order to monitor events sent to designated HTTP endpoint, update the log level to DEBUG, as shown in the following example:

    [Writable]
      LogLevel = "DEBUG"
    

  9. Now, the application service is ready to start up. If your Edge Xpert is running under secure mode, enter the following command, replacing the path with your configuration file:
    edgexpert up --secret app-service --path=<path your confgiuration file>
    
    If your Edge Xpert is running under insecure mode, enter the following command, replacing the path with your configuration file:
    edgexpert up app-service --path=<path your confgiuration file>
    
  10. Trigger the function pipeline from an HTTP request via the curl command or API tool and post the above example as the body.

    1. curl command
      curl -d "@data.json" -H "Content-Type: application/json" -X POST http://<the address of app-service>:59704/api/v2/trigger
      
    2. API tool
      http://<the address of app-service>:59704/api/v2/trigger
      
    3. Monitor the log from the running app service. If everything works fine, you should observe similar log as shown below:
      level=DEBUG ts=2022-01-25T15:22:32.041239076Z app=app-javascriptTransform-xpertHTTPExport-test source=context.go:29 msg="Setting context variable Latitude in pipeline 'default-pipeline'"
      level=DEBUG ts=2022-01-25T15:22:32.041321969Z app=app-javascriptTransform-xpertHTTPExport-test source=context.go:29 msg="Setting context variable Longitude in pipeline 'default-pipeline'"
      level=DEBUG ts=2022-01-25T15:22:32.041377297Z app=app-javascriptTransform-xpertHTTPExport-test source=http.go:135 msg="HTTP Exporting in pipeline 'default-pipeline'"
      level=DEBUG ts=2022-01-25T15:22:32.041421167Z app=app-javascriptTransform-xpertHTTPExport-test source=http.go:160 msg="Issuing HTTP GET request to https://postman-echo.com/get?latitude=25.024063&longitude=121.522111 in pipeline 'default-pipeline'"
      level=DEBUG ts=2022-01-25T15:22:33.085751307Z app=app-javascriptTransform-xpertHTTPExport-test source=http.go:187 msg="Sent 627 bytes of data in pipeline 'default-pipeline'. Response status is 200 OK"
      level=DEBUG ts=2022-01-25T15:22:33.08618545Z app=app-javascriptTransform-xpertHTTPExport-test source=conversion.go:228 msg="Prepare to run JavascriptTransform in pipeline 'default-pipeline'"
      level=DEBUG ts=2022-01-25T15:22:33.086300967Z app=app-javascriptTransform-xpertHTTPExport-test source=conversion.go:240 msg="JavascriptTransform in pipeline 'default-pipeline' receives data: {\"args\":{\"latitude\":\"25.024063\",\"longitude\":\"121.522111\",\"apiVersion\":\"v2\",\"id\":\"040bd523-ec33-440d-9d72-e5813a465f37\",\"deviceName\":\"device-001\",\"profileName\":\"profile-001\",\"sourceName\":\"source-1\",\"origin\":1602168089665565200,\"readings\":[{\"id\":\"31569347-9369-43ec-aa6a-59ea9c624a6f\",\"origin\":1602168089665565200,\"deviceName\":\"device-001\",\"resourceName\":\"resource-001\",\"profileName\":\"profile-001\",\"valueType\":\"Float32\",\"value\":\"39.5\"},{\"id\":\"2fd73a5b-969f-483c-9c52-6bb460a06eb1\",\"origin\":1602168089665565200,\"deviceName\":\"device-001\",\"resourceName\":\"resource-001\",\"profileName\":\"profile-001\",\"valueType\":\"Int8\",\"value\":\"75\"}],\"tags\":{\"latitude\":\"25.024063\",\"longitude\":\"121.522111\"}},\"headers\":{\"x-forwarded-proto\":\"https\",\"x-forwarded-port\":\"443\",\"host\":\"postman-echo.com\",\"x-amzn-trace-id\":\"Root=1-61f015b8-100c6d420dea7c3166c40f9a\",\"content-length\":\"627\",\"content-type\":\"application/json\",\"accept-encoding\":\"gzip\",\"user-agent\":\"Go-http-client/2.0\"},\"url\":\"https://postman-echo.com/get?latitude=25.024063&longitude=121.522111\"}"
      level=DEBUG ts=2022-01-25T15:22:33.091007028Z app=app-javascriptTransform-xpertHTTPExport-test source=conversion.go:271 msg="result returned from running script: {\"responseUserAgent\":\"Go-http-client/2.0\"}"
      level=DEBUG ts=2022-01-25T15:22:33.091189612Z app=app-javascriptTransform-xpertHTTPExport-test source=http.go:135 msg="HTTP Exporting in pipeline 'default-pipeline'"
      level=DEBUG ts=2022-01-25T15:22:33.09132988Z app=app-javascriptTransform-xpertHTTPExport-test source=http.go:160 msg="Issuing HTTP PATCH request to https://postman-echo.com/patch in pipeline 'default-pipeline'"
      level=DEBUG ts=2022-01-25T15:22:33.329903212Z app=app-javascriptTransform-xpertHTTPExport-test source=http.go:187 msg="Sent 42 bytes of data in pipeline 'default-pipeline'. Response status is 200 OK"
      

More configuration details about XpertHTTPExport

Configuration Description
Method Mandatory configuration to specify HTTP Method. Valid values are GET, POST, PUT, PATCH or DELETE.
MimeType Mandatory configuration to specify the MimeType in a HTTP request sending to destination.
Url Mandatory configuration to specify the URL of destination. Url allows to contain placeholder text, {key-name}, which will be replaced with any matching context variable found during runtime execution. For example, specify Url to http://myhost.com/edgex-events/{devicename} will send requests to different endpoints base on their device name.
PersistOnError Specify whether the events should be stored for later retry if encountering any error during message publish. Default value is false. If persisOnError is true and Store and Forward is enabled, the data will be stored for later retry. See Store and Forward for more details.
ContinueOnSendError Specify ContinueOnSendError to true will allow execution of subsequent chained senders after errors. Default value is false. Note that ContinueOnSendError=true can only be used when ReturnInputData=true and cannot be use when PersistOnError=true
ReturnInputData ReturnInputData enables chaining multiple HTTP senders if true. Default value is false.
SecretPath Specify the name of the path in secret provider to retrieve secrets.
AuthMode Specify the authentication mode for accessing the designated Url. Valid values are none, header_secret, and oauth2_clientcredentials. Default value is none.
  • Set AuthMode to none when there is no authentication required.
  • Set AuthMode to header_secret when the designated Url requires specific value to be included in the header of each HTTP request. When AuthMode is header_secret, SecretPath, HeaderName, and SecretName must be specified.
  • Set AuthMode to oauth2_clientcredentials when the designated Url requires OAuth2 client credentials grant. When AuthMode is oauth2_clientcredentials, SecretPath must be specified and manadatory secrets--client_id,client_secret,token_url--are expected to be retrieved from secret provider. Optional secrets--token_scopes and addiitonal parameters--should be added into secret provider when required for obtaining OAuth2 access token.
HeaderName Specify the name of the header key to add to the HTTP header. Only take effect when AuthMode is header_secret.
SecretName Specify name of the secret for the header value in the secret store. Only take effect when AuthMode is header_secret.

Note

If the designated HTTP endpoints requires TLS with Self-Signed CA certificate, users have to add cacert secret with the PEM format of self-signed certificate into secret store. Otherwise, user may experience x509: certificate signed by unknown authority error while sending data to the target HTTP endpoint.

Back to top