Skip to main content

Protobuf Schemas

Protocol Buffers (Protobuf) is a language- and platform-neutral mechanism for serialising structured data.
In MAPS, Protobuf is used wherever you want:

  • strong typing
  • explicit schema evolution
  • compact binary representation
  • multi-language interoperability

MAPS integrates Protobuf schemas through SchemaConfig with format: "protobuf".

1. Description​

Protobuf schemas are defined in .proto files, compiled to language-specific code and a binary descriptor set.
MAPS uses the descriptor(s) plus a small JSON wrapper to:

  • know which message type is the root
  • resolve nested message types
  • decode/encode payloads to/from Typed Events

Once decoded, Protobuf behaves like any other schema type in MAPS: all downstream processing sees a Typed Event, not raw Protobuf messages.

2. Protobuf Schema​

2.1 Example .proto definition​

syntax = "proto3";

package maps.sensors;

message Bme688Reading {
  string device_id   = 1;
  int64  timestamp   = 2; // epoch millis
  double temperature = 3; // °C
  double humidity    = 4; // %RH
  double pressure    = 5; // hPa
  double gas         = 6; // Ω
  string heater_mode = 7;
  string gas_mode    = 8;
}

This is the canonical Protobuf schema.
MAPS does not rewrite this; it consumes the compiled descriptors.

3. SchemaConfig for Protobuf​

Protobuf schemas in MAPS are stored as a SchemaConfig with:

  • format = "protobuf"
  • schemaBase64 = base64-encoded descriptor set (e.g. FileDescriptorSet)
  • schema (JSON) = metadata about root type, package, and options

3.1 Example SchemaConfig​

{
  "versionId": "1",
  "name": "Bme688-Protobuf",
  "description": "BME688 sensor reading using Protobuf encoding.",
  "labels": {
    "uniqueId": "e63f4f0f-1ef9-4b6a-8c7a-7c7b27e9f111",
    "resource": "sensor",
    "interface": "sensor.bme688",
    "comments": "BME688 telemetry encoded as Protobuf"
  },
  "format": "protobuf",
  "schema": {
    "package": "maps.sensors",
    "messageName": "Bme688Reading",
    "fileName": "bme688.proto"
  },
  "schemaBase64": "<base64-encoded FileDescriptorSet here>"
}

Notes:

  • schema.package and schema.messageName tell MAPS which message to treat as the root.
  • schemaBase64 contains the compiled descriptor(s).
    This allows MAPS to decode/encode without needing the .proto file at runtime.

4. Java Usage​

4.1 Registering the Protobuf schema​

Assuming you have already built a FileDescriptorSet and encoded it as Base64:

SchemaConfig config = SchemaConfig.builder()
    .name("Bme688-Protobuf")
    .format("protobuf")
    .description("BME688 Protobuf sensor reading")
    .schema(protobufMetaJsonObject)       // { package, messageName, ... }
    .schemaBase64(descriptorBase64String) // FileDescriptorSet as base64
    .build();

schemaRepository.register(config);

4.2 Processing Protobuf payloads​

At runtime, MAPS:

  1. Resolves the appropriate SchemaConfig for the topic/context.
  2. Loads and caches the Protobuf descriptors from schemaBase64.
  3. Locates the root message type using schema.package + schema.messageName.
  4. Decodes the binary Protobuf payload into:
    • a dynamic Protobuf message
    • then a Typed Event (MAPS internal representation)
  5. The Typed Event goes through:
    • filtering
    • transformations
    • statistics
    • format conversion (e.g. Protobuf → JSON / Avro / CBC / CSV)

Encoding is the inverse: MAPS takes a Typed Event and:

  1. Maps fields to Protobuf message fields.
  2. Builds a dynamic message.
  3. Serialises it as a Protobuf binary payload.

5. Protobuf-Specific Notes​

  • Strong typing & evolution
    Protobuf supports field addition/removal via field numbers and reserved tags.
    MAPS uses the active descriptor set; versioning is handled by versionId and your deployment practices.

  • Unknown fields
    Protobuf can preserve unknown fields. When decoding into Typed Events, MAPS focuses on known fields defined in the active schema.

  • Timestamps
    Common options:

    • int64 epoch millis (as shown above)
    • google.protobuf.Timestamp
      MAPS can normalise either to epoch millis internally.

  • Enums
    Protobuf enums become typed fields in Typed Events.
    MAPS preserves both the numeric value and the symbolic name when needed.

  • Oneof
    oneof fields are represented as a set of mutually exclusive fields in the Typed Event; only the active one is populated.

  • Embedded / nested messages
    These are converted to nested objects in Typed Events and can be used for filtering and transformations like any other structured schema format.

Protobuf is the right choice in MAPS when you want strict schemas, strong cross-language compatibility, and efficient binary encoding, especially for more complex domain models than “just JSON but smaller.”