Skip to content

Export EdgeX events to AWS S3

Overview

Amazon Simple Storage Service (Amazon S3) is cloud object storage. Edge Xpert provides a proprietary XpertHTTPExport function to send data to AWS S3 through the REST API. This article will take you through steps to create an Edge Xpert application service to export data as generated by virtual device service to AWS S3.

Prerequisites

  1. Understand Edge Xpert Application Services
  2. Launch Edge Xpert core services 
    edgexpert up
    or use --secret option to run in security mode 
    edgexpert up --secret
  3. Set up Amazon S3
  4. Create access keys that consist of two parts: an access key ID and a secret access key. The access keys are required by App Service to caculate AWS Signuatures for HTTPS requests.

Configure an application service to export events to AWS S3

Edge Xpert provides a proprietary built-in function XpertHTTPExport to produce events to the AWS S3. A sample configuration is provided with Edge Xpert CLI utility. Once you complete the installation of Edge Xpert CLI utility, the sample configuration is available at /usr/share/edgexpert/examples/app-configurable/awss3-http-export/configuration-insecure.toml.

  1. Copy and save the sample configuration file as a separate file (for example, aws-s3.toml) to your working directory

  2. Modify this configuration based on the AWS S3 settings.

    Config Value Description
    URL https://{AWSS3BucketName}.s3.{AWSRegion}.amazonaws.com/event_{timestamp}.json Replace the {AWSS3BucketName}(e.g. shellopenxpot), {AWSRegion}(e.g. us-east-1) that set in Prerequisites Step 3
    AWSV4SignerConfigs '{"region": "{AWSRegion}", "service": "s3"}' Replace the {AWSRegion}(e.g. us-east-1)

  3. If the application service is not launched in secure mode, secrets will be retrieved from Writable.InsecureSecrets configuration. You will need to update Writable.InsecureSecrets.AWS configuration with correct values of aws_access_key and aws_secret_key that obtain from Prerequisites Step 4: 

    [Writable.InsecureSecrets.AWS]
path = "aws"
  [Writable.InsecureSecrets.AWS.Secrets]
  aws_access_key=''
  aws_secret_key=''

  4. Once configuration is ready, you can launch the application service in secure mode using command below: 

    edgexpert up --secret app-service --path=<path of the configuration file>
    If you prefer to launch the application service in insecure mode, use command below: 
    edgexpert up app-service --path=<path of the configuration file>

  5. If the application service is launched under secure mode on previous step, secrets will be retrieved from Vault secret store. Users need to add aws_access_key and aws_secret_key into Vault secret store. Please refer to EdgeX Foundry Storing Secrets for details about adding secrets into Vault secret store.

  6. Now, an application service is up and ready to export events to AWS S3. To simulate the device sending readings from southbound, you can start up the virtual device service: 

    # Under secure mode
edgexpert up --secret device-virtual

# Under insecure mode
edgexpert up device-virtual

  7. The data can be viewed from the AWS Management Console by selecting an object and clicking Actions > Query with S3 Select

    AWS S3 query data

  8. AWS S3 also allows users to view the data with a HTTPS URL that can be retrieved by clicking Share with a presigned URL

    AWS S3 data url

More configuration details about XpertHTTPExport

Refer to XpertHTTPExport configuration.