JWeBot/docs/telemetry.md

# Runtime Telemetry And ACK Semantics

Status: local instrumentation contract for `alpha 0.6.x`.

## What ACK Means

ACK is the short acknowledgement sent immediately after Edge accepts a request.
It is not the final answer and it does not mean the expensive work has
completed.

The ordered path is:

1. OCR/responder submits one normalized event to Edge.
2. Edge durably records the event and queues the requested work.
3. Edge returns the short ACK (`喵......` or the cloud acknowledgement).
4. The recorder sends that ACK through the guarded GUI path.
5. The recorder calls Edge with `fast_ack_sent` or `fast_ack_failed`.
6. Expensive work may run concurrently, but its final reply cannot be delivered
   until ACK delivery is confirmed.

This separates perceived response latency from completion latency while
preserving message order. A failed ACK prevents the later final reply from
appearing without its expected acknowledgement.

## Privacy Boundary

Telemetry never records message bodies, sender names, raw group identifiers,
reply text, secrets, environment values, or service payloads.

Event and group identifiers are represented only by 12-character SHA-256
prefixes. Plugin IDs, route names, queue states, operation names, counts,
bounded error class names, and durations are allowed.

Every structured log uses schema version `1`:

```json
{
  "schema_version": 1,
  "name": "edge.ack.confirmed",
  "status": "ok",
  "component": "edge",
  "event_hash": "0123456789ab",
  "elapsed_ms": 83.417,
  "ack_kind": "chat",
  "delivered": true
}
```

## Instrumented Stages

| Stage | Events |
|---|---|
| Capture and OCR | `ocr.warmup`, `ocr.capture`, `ocr.frame.wait`, `ocr.scan`, `ocr.pipeline` |
| OCR queue | `ocr.queue.wait`, `ocr.queue.process` |
| Recorder to Edge | `recorder.edge.request` |
| ACK GUI and confirmation | `recorder.ack.gui`, `recorder.ack.end_to_end`, `recorder.ack.confirm`, `edge.ack.prepared`, `edge.ack.confirmed` |
| Fast ACK dedicated view | `recorder.fast_ack.gui`, `recorder.fast_ack.end_to_end`, `recorder.fast_ack.confirm`, `edge.fast_ack.prepared`, `edge.fast_ack.confirmed` |
| Edge routing | `edge.route.classified`, `edge.local.complete`, `edge.alias.complete` |
| Extension startup and execution | `edge.extensions.startup`, `edge.extension.execute` |
| Host process and IPC | `host.process.spawn`, `host.runtime.load`, `host.request.received`, `host.service.request`, `host.service.wait`, `host.service.complete`, `host.invoke`, `host.process.complete` |
| Kernel broker | `edge.service.call` |
| Model and cloud providers | `edge.provider.complete` |
| Durable queues | `edge.queue.wait`, `edge.queue.claim`, `edge.queue.work` |
| Final GUI delivery | `edge.delivery.process`, `edge.delivery.complete`, `edge.delivery.end_to_end`, `delivery.gui.send` |
| Guarded GUI send internals | `gui.send.step`, `gui.bridge.step`, `gui.send.total` |

`gui.send.step` is emitted by the parent OCR process around safety boundaries:
GUI lock acquisition, pre-draft screenshot, pre-draft group check, bridge
process calls, post-draft screenshot, post-draft group check, callbacks, and
total completion. `gui.bridge.step` is emitted from the child bridge timing
metadata and covers VNC focus clicks, fixed settle waits, screenshot polling,
draft confirmation, send-button confirmation, and Enter fallback.

The Kernel also retains process-local low-cardinality counts and
average/maximum latency summaries. They are exposed through Edge health output
for diagnosis and reset when the process restarts.

`edge.route.classified` records one bounded `execution_path` value:
`local-command`, `deepseek`, or `openclaw`. OCR scan events record cache hits,
misses, and periodic refresh without recording image or message content.
Direct-request queue events use `queue=direct_generation` for parallel provider
or Steam work and `queue=direct_delivery` for the single ordered GUI worker.
Their bounded `worker` field distinguishes `generation-0`, `generation-1`, and
`delivery-0` without exposing event content.

## Error Rules

- Record only the exception class, never exception text from plugin or user
  data boundaries.
- Distinguish controlled errors, retries, uncertain delivery, startup fallback,
  timeouts, and final failures.
- Do not replay an active Host invocation after a service call may have
  committed state.
- Keep existing durable queue state as the source of truth; telemetry is
  observational and must never control delivery.
- Telemetry observer and logger failures are swallowed so instrumentation
  cannot break the reply path.

## Performance Analysis

The current points support separate measurement of:

- screen capture delay;
- OCR recognition cost and bubble count;
- stable-frame/debounce delay;
- recorder-to-Edge socket latency;
- ACK GUI latency and ACK confirmation latency;
- Host process startup and plugin load cost;
- service broker round trips;
- model/provider execution;
- queue waiting and retry time;
- final guarded GUI delivery.

Future optimization can then trade compute for latency deliberately, for
example:

- narrower OCR regions or separate sender/body recognizers;
- more frequent capture with bounded coalescing;
- selective high-cost OCR only for likely incoming bubbles;
- warm or persistent Extension Hosts;
- concurrent ACK delivery and expensive computation;
- queue prioritization by route and age;
- preloaded models and cached layout/title fingerprints.

No such optimization should be accepted without comparing end-to-end ACK and
final-delivery percentiles and checking duplicate/ordering regressions.

Run `tools/performance-baseline.py` for repository-only microbenchmarks or pass
one or more private `--telemetry` logs to calculate stage distributions.