Data Transformation

🔁 Data Transformation Engine

Overview

MAPS Messaging includes a transformation engine that allows administrators to modify, enrich, or normalize data in transit. This is useful when integrating devices that produce different data formats, when enriching messages with metadata, or when converting between protocol-specific schemas.

The transformation rules are defined in the TransformationManager.yaml file. Transformations are matched based on URI-style patterns that describe the source of the message.

🧾 Configuration Format

Defined in TransformationManager.yaml:

transformations:
  data:
    - pattern: "*://*/*/*"
      transformation: ""

Each rule consists of:

pattern: A matching rule for the source connection.
transformation: The transformation script or function to apply (currently empty in the example).

🔍 Pattern Matching

The pattern field uses a simplified URI-style syntax:

(protocol)://(host)/(protocol)/(username)

Wildcards (*) can be used to match any value in each segment.

✅ Examples

Pattern	Matches
`tcp:///mqtt/`	All MQTT connections over TCP
`:///*/fred`	Any connection from user `fred`
`i2c://1/29/*`	I2C device `29` on bus `1`
`://localhost/coap/`	CoAP clients connecting via localhost

The transformation engine processes the rules from top to bottom. The first match wins. Order matters!

🧠 Transformation Logic

The transformation script or logic is referenced as a string (possibly a named script, embedded expression, or function reference). Depending on implementation, these may include:

JSON-to-JSON transformations (e.g., field mapping, renaming)
Protocol translation (e.g., GPS NMEA to structured JSON)
Field injection (e.g., add timestamp, geolocation)
Value normalization (e.g., converting units)

🚧 Note: The format of transformation logic is implementation-specific and may require external scripting support (e.g., JavaScript, Groovy, or internal DSL).

🧪 Example Use Case

transformations:
  data:
    - pattern: "tcp://*/mqtt/sensor01"
      transformation: "normalizeSensorPayload"

This rule applies the normalizeSensorPayload transformation to messages published by sensor01 over MQTT on TCP.

✅ Best Practices

Use specific patterns before general wildcards to avoid unintended matches.
Keep transformation scripts modular and reusable.
Document custom transformations clearly if using embedded logic.
Consider performance impact for complex or large-scale data rewrites.
Use transformations to enforce consistent schemas across protocols.