Skip to main content

Anomaly Detection

Real-time anomaly detection in Maps is executed exclusively via JMS selectors that call ML operations. Models are loaded by the ML Model Manager (file/S3/Nexus/Maps backends) and evaluated inline to accept/reject/route events.

What Maps Detects​

  • Message pattern irregularities
  • Protocol behavior deviations
  • System performance shifts
  • Environment/sensor anomalies (e.g., indoor air-quality drift)
  • Security-relevant outliers (e.g., unusual client behavior)

Model choice (Maps practice)​

  • Random Forest (RF) is the primary built-in “learning” algorithm in Maps.
  • Use RF for:
    • Classification: normal vs. abnormal.
    • Regression + residuals: predict a metric and flag large prediction error.
  • Other models (ONNX/TF) can be used; execution still goes through JMS selectors.

Example: Room AQI anomaly detection (48‑hour learning)​

Goal: Learn “normal” conditions for a specific room over ~2 days, then flag anomalies in real time.

Signals (metric units): AQI, CO₂ (ppm), temperature (°C), humidity (%), PM2.5 (µg/m³), optionally VOC index/NOx index. Input stream: per-room topic, e.g., /building/level1/room-201/sensors.

Phase 1 — Learn (collect baseline ~48h)​

  • Accumulate events for 2 days (or until maxTrainEvents is met).
  • Train RF classifier on “normal” samples.
    • If you do not label anomalies during learning, treat the 48‑hour dataset as normal and synthesize boundaries (e.g., quantile banding per feature or cluster labels used as “normal”).
    • Alternative: RF regression to predict AQI from other features; compute residual = |predicted − observed|.

Phase 2 — Detect (continuous run)​

  • At inference time, the JMS selector computes:
    • Classification probability of “normal” (e.g., P(normal)).
    • Or residual threshold if using regression.
  • If below threshold / residual above limit → publish to an outlier topic and optionally trigger alerts.

Minimal configuration (illustrative)​

1) Model Store & Manager (see full reference in the ML overview)​

MLModelManager:
  enableCaching: true
  autoRefresh:
    enabled: true
    intervalMinutes: 10
  modelStore:
    type: file
    config:
      file:
        path: "{{MAPS_DATA}}/models"
  preloadModels: room201_rf.zip   # optional

2) Event stream with 48‑hour learning window​

eventStreams:
  - id: room201.aqi.learn
    topicFilter: /building/level1/room-201/sensors
    schemaId: room-sensor-v1
    # Learn phase: train a Random Forest model on 48h of data
    selector: rf.train(room201_rf, AQI, CO₂, temperature, humidity, pm2_5, vocIndex, noxIndex)
    maxTrainEvents: 20000           # cap the sample size
    maxTrainTimeSeconds: 172800     # ≈ 48 hours
    retrainThreshold: 0.03          # trigger retrain on drift (optional)

3) Runtime detection via JMS selector​

Classification probability approach:

selector: rf.classifyprob(room201_rf, AQI, CO₂, temperature, humidity, pm2_5, vocIndex, noxIndex) < 0.05

If true, route to an outlier topic:

outlierTopic: /building/level1/room-201/outliers

Regression + residual approach (alternative):

selector: abs(rf.predict(room201_rf, CO₂, temperature, humidity, pm2_5, vocIndex, noxIndex) - AQI) > 25

(25 AQI points is just an example threshold; choose based on baseline variance.)

Note: Function names above (rf.train, rf.classifyprob, rf.predict) are indicative of the RF operation family exposed to JMS selectors. Use the exact operation names configured in your Maps build.

Thresholds & tuning​

  • Start conservative (e.g., P(normal) < 0.01 or residual > 3× baseline std-dev).
  • Adjust per room; ventilation cycles and occupancy patterns differ.
  • Relearn on season change or after HVAC modifications (use retrainThreshold or schedule retraining).

Operations & outputs​

  • Pass/Drop/Route: A selector condition routes the event (e.g., to /outliers) without blocking other processing.
  • Metadata enrichment: Attach probabilities/residuals as headers for downstream analysis.
  • Fleet rollout: Train per-room models, or train a generic model and specialize thresholds by room.

Operational notes​

  • Smile post-install: run {MAPS_HOME}/bin/download-smile.sh or download-smile.bat once per node.
  • Backends: file/S3/Nexus/Maps are supported; the runtime execution path is identical.
  • Security: use TLS and scoped credentials; for the maps backend, scope API keys per namespace.