> ## Documentation Index
> Fetch the complete documentation index at: https://mx-6c34bcc6.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# WattSwarm Diagnostics: API, CLI, and Console Tools

> Use the WattSwarm diagnostics endpoint, CLI log commands, and the kernel console to inspect P2P status, event flow, backfill gaps, and executor health.

When a node is misbehaving — events not flowing, peers not syncing, tasks stalling — the diagnostics API and kernel console give you a structured view into what the network bridge is actually doing. Rather than grepping raw logs, you can query the running node directly and get a snapshot of gossip topics, backfill cursors, transport events, and peer state all in one place.

## GET /api/diagnostics

The primary diagnostics endpoint reads the node's `diagnostics/wattswarm_node.jsonl` log and combines it with a live bridge observability snapshot.

```bash theme={null}
curl http://127.0.0.1:7788/api/diagnostics | jq .
```

You can filter the event list with query parameters:

| Parameter        | Description                                          |
| ---------------- | ---------------------------------------------------- |
| `limit`          | Maximum number of diagnostic events to return        |
| `level`          | Filter by log level (e.g. `error`, `warn`, `info`)   |
| `component`      | Filter by component name (e.g. `gossip`, `backfill`) |
| `category`       | Filter by event category                             |
| `mode`           | Filter by mode field                                 |
| `phase`          | Filter by lifecycle phase                            |
| `event_id`       | Filter by specific SEL event ID                      |
| `object_id`      | Filter by task, run, or object ID                    |
| `source_node_id` | Filter events from a specific remote node            |
| `search`         | Free-text search across event payloads               |

### What the response includes

The response object contains the following fields:

* **`network_service_started`** — `true` if the P2P bridge started successfully during this node session.
* **`network_service_status`** — current bridge status string (e.g. `running`, `stopped`).
* **`snapshot`** — the latest bridge observability snapshot, which includes:
  * **`p2p_foundation`** — `"iroh"` on the standard state-dir startup path; indicates which transport stack is active.
  * **`local_iroh_endpoint_id`** — the Iroh `NodeId` / `EndpointId` for this node's active endpoint.
  * **Iroh gossip topic IDs joined** — the set of deterministic gossip topic IDs derived from `network_id + scope + gossip_kind`.
  * **Known Iroh contact count** — number of peers for which persisted Iroh contact material is available.
  * **`legacy_transport_active`** — `false` on normal state-dir startup. If `p2p_foundation` is `"iroh"`, legacy transport fields in the snapshot are compatibility placeholders only.
* **`diagnostics`** — array of structured transport, gossip, backfill, and agent-callback event entries from the JSONL log.

## Accessing diagnostics via the UI

The kernel console includes a formatted diagnostics view. Open `http://127.0.0.1:7788/diagnostics` in your browser to see the same data rendered with filtering controls. Use this when you want to scan events visually rather than parsing JSON.

<Tip>
  Open the **`/swarm` dashboard** at `http://127.0.0.1:7788/swarm` to watch real-time task state transitions driven by live kernel calls. The dashboard calls `/api/swarm/state` and `/api/swarm/tick` against actual executor runtimes, so the panels reflect true SEL and projection state — not a simulation.
</Tip>

## CLI log commands

The `wattswarm log` subcommands give you direct access to the append-only Structured Event Log (SEL) stored in PostgreSQL.

```bash theme={null}
# Show the latest events in the SEL (most recent head sequence number and entries)
wattswarm log head --pg-url postgres://postgres:postgres@127.0.0.1:55432/wattswarm

# Replay all events and rebuild projections from scratch
wattswarm log replay --pg-url postgres://postgres:postgres@127.0.0.1:55432/wattswarm

# Verify log integrity — checks sequence continuity and hash chain
wattswarm log verify --pg-url postgres://postgres:postgres@127.0.0.1:55432/wattswarm
```

Use `log replay` after a crash or manual schema change to rebuild the node's projection tables from the raw event history. Use `log verify` to confirm the log has not been truncated or tampered with.

## Node status

To check the node's current identity, network membership, and P2P info:

```bash theme={null}
curl http://127.0.0.1:7788/api/node/status | jq .
```

The response includes `node_id`, `running`, `mode`, `local_protocol_version`, and `peer_protocol_distribution` — a map of protocol version strings to the count of peers seen at each version.

## Peers list

To see all discovered peers and how they were found:

```bash theme={null}
curl http://127.0.0.1:7788/api/peers/list | jq .

# or via CLI
wattswarm peers list --pg-url postgres://postgres:postgres@127.0.0.1:55432/wattswarm
```

Each peer entry includes a `source_kind` that tells you how the node learned about it:

| `source_kind`     | Meaning                                            |
| ----------------- | -------------------------------------------------- |
| `udp`             | Discovered via UDP multicast or broadcast announce |
| `bootstrap`       | Loaded from startup `bootstrap_contacts`           |
| `connected`       | Established an active Iroh session                 |
| `identify`        | Learned via P2P handshake/identify                 |
| `bootstrap_index` | Found via bootstrap index lookup                   |
| `local_discovery` | Discovered via mDNS (legacy compatibility path)    |

## Checking executor health from the kernel

To trigger an executor health check from within the running kernel (rather than from the CLI), send a POST request directly:

```bash theme={null}
curl -X POST http://127.0.0.1:7788/api/executors/check \
  -H "Content-Type: application/json" \
  -d '{"name": "rt"}'
```

The kernel calls the executor's `/health` and `/capabilities` endpoints and returns a structured result. This is the same check the worker performs before dispatching a step.

## Common investigation patterns

### Missing events between nodes

If you expect events from a remote node but they are not appearing in the local SEL, check the backfill cursor state in the diagnostics log:

```bash theme={null}
curl 'http://127.0.0.1:7788/api/diagnostics?component=backfill&limit=50' | jq '.diagnostics'
```

Look for backfill events where `phase` is `request` but there is no corresponding `response`, or where `error` payloads appear. A persistent gap in cursors means the peer is either unreachable or returning empty backfill responses. Check that the remote node is `UP` and that its `head_seq` is advancing with `wattswarm log head`.

### Slow finalization

If tasks are taking longer than expected to finalize, check the resolution path breakdown in the run result:

```bash theme={null}
curl 'http://127.0.0.1:7788/api/run/result?run_id=<run-id>' | jq '.result.aggregation.resolution_paths'
```

The `resolution_paths` field shows which aggregation steps were taken (e.g. `TIE`, `REEXPLORE`, `STOCHASTIC`). Cross-reference with `aggregation.null_resolution` to see if null paths were triggered. For latency profiling, the runtime metrics endpoint exposes true p95 latency computed from collected samples.

### Sync issues after reconnect

If a node reconnects after a partition and events are still missing, look for gossip backfill events with error payloads in the diagnostics log:

```bash theme={null}
curl 'http://127.0.0.1:7788/api/diagnostics?component=gossip&level=error' | jq '.diagnostics'
```

Backfill is lane-aware and persists cursor state across reconnects, so a successfully reconnected peer should resume from where it left off. Error payloads here indicate the remote peer rejected or could not serve the backfill range — check whether the remote node's event log covers the requested sequence range.