Skip to main content

Schemas

Schemas define the structure and format of data flowing through MAPS. They ensure that incoming events can be validated, interpreted, transformed, and processed consistently, regardless of protocol or payload format.

This page provides a high‑level overview of schema concepts within MAPS. Detailed format‑specific documentation is available in the dedicated schema pages.

Why Schemas Matter​

Schemas provide a foundation for data correctness and interoperability. MAPS uses schemas to:

  • Validate incoming payloads
  • Enable typed filtering and accurate expression evaluation
  • Support transformations between schema types
  • Perform statistics using correct numeric and string types
  • Convert data safely across supported formats

Without a schema, MAPS treats payloads as opaque, and processing capabilities are limited.

What a Schema Looks Like in MAPS​

Every schema in MAPS is defined as a SchemaConfig object, containing:

  • Metadata
    name, versionId, description, lifecycle timestamps

  • Labels
    Used for discovery and classification (resource, interface, uniqueId, comments)

  • Format
    The schema type (e.g., json, protobuf, avro, cbor, cbc, messagepack, csv, native)

  • Schema Definition
    Provided as:

    • JSON Schema (JSON, CBOR, MessagePack)
    • Avro schema
    • Protobuf descriptor set
    • CBC field layout
    • CSV structural conventions
    • Native scalar declaration

  • Optional Matching Rules
    Such as matchExpression for topic binding

This configuration is the canonical definition used across ingestion, filtering, transformations, routing, and output encoding.

Where Schemas Are Used​

Schemas influence several key components of the MAPS data‑processing pipeline:

1. Ingestion​

Schemas allow MAPS to decode incoming payloads into Typed Events.

2. Filtering​

Typed values enable accurate comparisons, boolean logic, and timestamp handling.

3. Transformations​

Schema‑based transformations support structured mapping, enrichment, and cross‑format conversion.

4. Statistics​

Schemas ensure correct aggregation for:

  • numeric fields
  • string fields
  • time‑series summaries

5. Routing​

Routing expressions gain consistency from typed fields and stable structure.

6. Format Conversion​

Schemas allow safe and deterministic conversion between supported formats.

Schema Registration​

Schemas may be bound to:

  • specific topics
  • topic patterns

After registration, the schema becomes active and is used automatically where applicable.

Full details are provided in the Schema Registration documentation.

Supported Schema Types​

MAPS supports:

  • Avro
  • CBC (bit‑packed binary format)
  • CBOR Schema
  • CSV (best‑effort)
  • JSON Schema
  • MessagePack Schema
  • Native scalar values
  • Protobuf
  • RAW (opaque, no schema)

Each format has dedicated documentation describing its capabilities and structure.

Typed Events​

Once decoded, MAPS produces a Typed Event, containing:

  • typed fields
  • nested structures
  • normalised timestamps
  • stable lookup paths
  • validated content

This is the unified internal representation used across the MAPS pipeline.

When No Schema Is Available​

MAPS can still process the message, but with reduced capability:

  • filtering is only available on meta data not the payload
  • statistics are not available
  • transformations are limited
  • cross‑format conversion are not available

Registering schemas is recommended even for simple payloads.

Schema Details​

See the detailed schema pages for specific formats: