Core Data

Introduction

The Core Data provides centralized persistence for data collected by devices. Device Services that collect sensor data call on the Core Data to store the sensor data on the edge system, such as in a gateway until the data gets moved to the "north" and then exported to the enterprise and cloud systems. Core data persists the data in a local database. Redis is used by default, but a database abstraction layer allows for other databases to be used.

Other services and systems, both within Edge Xpert and outside of Edge Xpert, access the sensor data through the Core Data. Core data could also provide a degree of security and protection of the data collected while the data is at the edge.

Sensor data can be sent to Core Data via two different means:

1. Services (like Devices Services) and other systems can put sensor data on a message bus topic and Core Data can be configured to subscribed to that topic. This is the default means of getting data to Core Data. Any service (like the Application Service or rules engine service) or 3rd system could also subscribe to the same topic. If the sensor data does not need to persisted locally, Core Data does not have to subscribe to the message bus topic - making Core Data completely optional. By default, the message bus is implemented using MQTT in Edge Xpert. Redis Pub/Sub can be used as an alternate message bus implementation, and it's the default message bus in the EdgeX Foundry community.

   

2. Services and systems can call on the Core Data REST API to send data to Core Data and have the data put in the local storage. Prior to Edge Xpert 2.0, this was set in default and only means to send data to Core Data. Today, it is an alternate means to send data to Core Data. When data is sent via REST to Core Data, Core Data republishes the data to message bus so that other services can subscribe to it.

   

Core data moves data to the Application Service (and edge analytcs) via MQTT by default in Edge Xpert. Redis Pub/Sub or ZeroMQ can be used as alternatives. Using MQTT requires installing a broker such as ActiveMQ, and Edge Xpert uses Eclipse Mosquitto as the default MQTT broker. There is a messaging infrastructure abstraction that allows other message bus implementations (such as AMQP) to be created and used.

Core Data "Streaming"

By default, all data sent to it from services and other systems is kept by Core Data. However, if the data is too sensitive to be stored at the Edge, or if it is not being used at the Edge by other local services (e.g., an analytics microservice), the data can be "streamed" through Core Data without being stored. A configuration change to Core Data (Writable.PersistData=false) causes Core Data to send data to the Application Service without persisting it. This option has the advantage of reducing latency through this layer and reducing storage requirements at the network edge. The result, however, is that no historical data is available for analysis, which requires looking into the past to make a decision.

Events and Readings

Data collected from sensors is marshalled into EdgeX event and reading objects (delivered as JSON objects or a binary object encoded as CBOR to Core Data). An event represents a collection of one or more sensor readings. Some sensors or devices are only providing a single value – a single reading - at a time. Other sensors spew multiple values whenever they are read.

An event must have at least one reading. Events are associated to a sensor or device – the “thing” that sensed the environment and produced the readings. Readings represent a sensing on the part of a device or sensor. Readings only exist as part of (are owned by) an event. Readings are essentially a simple key/value pair of what was sensed (the key - called a ResourceName) and the value sensed (the value). A reading may include other bits of information to provide more context (for example, the data type of the value) for the users of that data. Consumers of the reading data could include things like user interfaces, data visualization systems and analytics tools.

In the diagram below, an example event/reading collection is depicted. The event coming from the “motor123” device has two readings (or sensed values). The first reading indicates that the motor123 device reported the pressure of the motor was 1300 (the unit of measure might be something like PSI).

The value type property (shown as type above) on the reading indicates that the value is an integer, 64 bits. The second reading indicates that the motor123 device also reported the temperature of the motor was 120 at the same time it reported the pressure (perhaps in degrees Fahrenheit).

Data Model

The following diagram shows the Data Model for Core Data. Device Services send Event objects containing a collection or Readings to Core Data when a device captures a sensor reading.

Data Retention

In use cases, since core data persists data in the local database indefinitely, there is a need to persist the latest recent readings only and clean up the old ones, as keeping an infinite number of readings is considered computationally expensive and can lead to a lack of machine memory. Thus, a retention mechanism is placed on core data to keep a certain number of readings.

Under this mechanism, the maximum readings capacity is called MaxCap and the minimum readings capacity is called MinCap. Core data will create an internal schedule according to the Interval configuration to check if the number of readings are higher than the MaxCap. When the number of readings reach the MaxCap, Core data will purge the amount of the readings to the MinCap.

For example, the MaxCap is set to 10, the MinCap is set to 2 and the Interval is set to 3s. Now, core data will check how many readings are in the local database every 3 seconds. When the number of readings reach 10, core data will check the 3rd reading to find the related event's origin timestamp and perform function DeleteEventsByAge to delete events by age. This way the related readings will also be deleted.

For more information, see Data Retention and Persistent Caps for more details.

Data Dictionary

Configuration Properties

Please refer to the general Common Configuration documentation for configuration properties common to all services. Below are only the additional settings and sections that are not common to all EdgeX Services.

V2 Configuration Migration Guide

Refer to the Common Configuration Migration Guide for details on migrating the common configuration sections such as Service.

Writable

The following settings have been removed from the Writable section

• DeviceUpdateLastConnected
• MetaDataCheck
• ServiceUpdateLastConnected
• ValidateCheck
• ChecksumAlgo

MessageQueue

The following MessageQueue setting values have changed:

• Host - Override value for docker is now mqtt-broker
• Protocol = "mqtt"
• Port = 1883
• Type = 'mqtt'

The following setting has been removed from the MessageQueue section

• Topic

The following new settings have been added to MessageQueue section

• PublishTopicPrefix = 'edgex/events/core'

• SubscribeTopic = 'edgex/events/device/#'

• AuthMode = 'none'

• SecretName = 'mqtt-bus'
• PublishTopicPrefix = 'edgex/events/core'
• SubscribeEnabled = true

MessageQueue.Optional

The following settings have been removed from MessageQueue.Optional section for when using MQTT for the MessageBus. Secure MessageBus using MQTT is not yet supported and will be retrieved from the Secret Store in a future release.

• Username
• Password