Diagnostics API — Event Log and Swarm Observability

Inspect network service diagnostics, read the structured event log head, replay or verify the SEL, and advance the swarm tick-by-tick.

The Diagnostics API exposes the internals of the WattSwarm kernel for observability, debugging, and manual swarm control. The network diagnostics endpoint provides a snapshot of the P2P service state and recent structured events. The log endpoints let you read from the kernel’s Structured Event Log (SEL), trigger a full projection replay (useful after a crash or upgrade), and verify the log’s cryptographic integrity. The swarm endpoints give dashboard tooling a view into the current swarm task state and let operators drive the swarm forward one tick at a time for step-by-step debugging.

---

GET /api/diagnostics

Returns a comprehensive diagnostic snapshot of the kernel, including the network service status, the latest observability snapshot from the p2p bridge, and a filterable list of recent diagnostic events.

GET /api/diagnostics

Query parameters (all optional):

limit · integer

Maximum number of diagnostic events to return.

level · string

Filter by severity level (e.g. "error", "warn", "info").

component · string

Filter by kernel component name (e.g. "network_bridge", "run_queue").

category · string

Filter by event category string.

mode · string

Filter by network mode.

phase · string

Filter by lifecycle phase string.

event_id · string

Return only the event with this exact event ID.

object_id · string

Filter events related to a specific object (task ID, run ID, peer ID, etc.).

source_node_id · string

Filter events originating from a specific node.

search · string

Full-text search string applied across event payload fields.

Example request:

curl "http://127.0.0.1:7788/api/diagnostics?limit=10&level=error"

Example response:

{
  "ok": true,
  "generated_at": "2024-06-10T12:00:05Z",
  "network_service_started": true,
  "network_service_status": {
    "status": "running",
    "uptime_ms": 300000
  },
  "snapshot": {
    "connected_peer_count": 3,
    "known_peer_count": 7,
    "gossip_message_count": 142,
    "last_snapshot_at": "2024-06-10T12:00:00Z"
  },
  "diagnostics": [
    {
      "event_id": "diag-00000042",
      "level": "error",
      "component": "network_bridge",
      "category": "peer_connect",
      "message": "failed to establish connection to node-xyz",
      "source_node_id": "node-local123",
      "object_id": "node-xyz",
      "created_at": 1718000295000
    }
  ]
}

Response fields:

generated_at · string

ISO 8601 timestamp when the diagnostic snapshot was generated.

network_service_started · boolean

Whether the background P2P network service is currently running.

network_service_status · object | null

Status details from the network bridge, or null if the service is not running.

snapshot · object | null

Latest observability snapshot including peer counts and gossip metrics.

diagnostics · object[]

Filtered list of recent diagnostic events matching the query parameters.

---

GET /api/log/head

Returns the current head sequence number of the kernel’s Structured Event Log (SEL). The head sequence is the highest event sequence number written so far. Use this to determine how many events are in the log or to compare sequence positions across nodes.

GET /api/log/head

Example request:

curl http://127.0.0.1:7788/api/log/head

Example response:

{
  "ok": true,
  "head": 1047
}

Response fields:

head · integer

The sequence number of the most recently written SEL event.

---

POST /api/log/replay

Triggers a full replay of the Structured Event Log to rebuild all kernel projections from scratch. This is useful after a kernel upgrade that introduces schema changes to the read-model, or to recover from a corrupt projection state. The operation runs synchronously and may take several seconds on large logs.

POST /api/log/replay

No request body is required.

Example request:

curl -X POST http://127.0.0.1:7788/api/log/replay

Example response:

{
  "ok": true,
  "message": "replayed"
}

Log replay rewrites all in-memory and on-disk projections by re-processing every event in sequence. It is a write-intensive operation. Do not call this endpoint while the node is actively processing tasks.

---

POST /api/log/verify

Verifies the cryptographic integrity of the Structured Event Log. The kernel checks that the event chain hashes are consistent from genesis to the current head. Returns {"ok": true} if the log is intact, or an error if a hash mismatch or gap is detected.

POST /api/log/verify

No request body is required.

Example request:

curl -X POST http://127.0.0.1:7788/api/log/verify

Example response (valid):

{
  "ok": true,
  "message": "verified"
}

Example response (tampered log):

{
  "ok": false,
  "error": "hash mismatch at sequence 512: expected sha256-abc... got sha256-def..."
}

---

GET /api/swarm/state

Returns the current swarm task state as rendered by the dashboard engine. This endpoint is used by the built-in swarm dashboard UI but is equally useful for external monitoring tools or CI pipelines that need a structured view of in-flight tasks, candidates, and votes.

GET /api/swarm/state

Example request:

curl http://127.0.0.1:7788/api/swarm/state

Example response:

{
  "ok": true,
  "state": {
    "tasks": [
      {
        "task_id": "task-demo-001",
        "task_type": "generic.qa.v1",
        "terminal_state": "Finalized",
        "epoch": 1,
        "candidate_count": 2,
        "committed_candidate_id": "cand-exec-p-abc123",
        "finalized_candidate_id": "cand-exec-p-abc123"
      }
    ],
    "open_task_count": 0,
    "finalized_task_count": 1
  }
}

---

POST /api/swarm/tick

Advances the swarm by one execution tick. A single tick runs the full claim→execute→verify→vote→commit→finalize pipeline for any eligible tasks in the store. This endpoint is designed for development, testing, and step-by-step debugging of swarm behavior without requiring background workers.

POST /api/swarm/tick

executor · string

Name of the executor to use for this tick. Defaults to the core_agent executor configured in the startup config.

profile · string

Executor profile to activate for this tick. Defaults to "default".

Example request:

curl -X POST http://127.0.0.1:7788/api/swarm/tick \
  -H "Content-Type: application/json" \
  -d '{
    "executor": "local-agent",
    "profile": "default"
  }'

Example response:

{
  "ok": true,
  "state": {
    "tasks": [
      {
        "task_id": "task-demo-001",
        "task_type": "generic.qa.v1",
        "terminal_state": "Committed",
        "epoch": 1,
        "candidate_count": 1,
        "committed_candidate_id": "cand-exec-p-abc123",
        "finalized_candidate_id": null
      }
    ],
    "open_task_count": 1,
    "finalized_task_count": 0
  }
}

Call /api/swarm/tick in a loop in integration tests to drive tasks from Created all the way to Finalized without relying on background workers or timers. Combine with /api/swarm/state to assert on intermediate state after each tick.

/api/swarm/tick is a blocking call backed by tokio::task::spawn_blocking. For tasks that invoke slow executors, the HTTP request will remain open until the tick completes. Set an appropriate client timeout (recommend ≥ 60 s for tasks with non-trivial prompts).