> ## Documentation Index
> Fetch the complete documentation index at: https://artemiscity.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring

> Prometheus, Grafana, and the redis-exporter sidecar for scraping governance metrics, firing alerts on quarantines and trust collapses, and visualizing agent state.

Artemis City ships a bootable monitoring stack in `docker-compose.yaml`: the FastAPI kernel exposes a Prometheus-format `/metrics` endpoint covering governance, ATP, and memory-bus instrumentation; a `redis-exporter` sidecar covers Redis health; Prometheus scrapes both and evaluates a governance alert-rules file; Grafana provisions a pre-built **Artemis City Governance** dashboard on top of that data.

## When to use this

* You are running Artemis City from compose and want operator-facing visibility into agent trust, quarantines, and violations.
* You need alerts to page when governance stores go unreadable, an agent lands in quarantine, or Redis drops offline.
* You are extending governance and want to know which Prometheus series are already exported so you can reuse them in dashboards and rules.

If you only run unit tests or a single-process kernel locally, you can skip this page — the `/metrics` endpoint still responds, but nothing scrapes it.

## Bringing the stack up

The monitoring services are part of the same compose file as the kernel. From the repo root:

```bash theme={null}
docker compose up -d kernel redis redis-exporter prometheus grafana
```

Once the containers are healthy:

* Kernel metrics: `http://localhost:8000/metrics`
* Prometheus UI: `http://localhost:9090`
* Grafana UI: `http://localhost:3000` (admin password from `GRAFANA_PASSWORD`)

Prometheus stores its TSDB in the `prometheus-data` compose volume with a **30-day retention window**. Grafana state lives in the `grafana-data` volume.

`GRAFANA_PASSWORD` and `REDIS_PASSWORD` must be set before starting the stack; both services fail fast if they are missing. See [Secrets setup](/Documentation/operations/secrets-setup).

## The `/metrics` endpoint

`GET /metrics` on the FastAPI kernel serves the Prometheus default registry. That covers three families:

* **Governance metrics** — read at scrape time from the governance SQLite stores (`agent_registry.db`, `hebbian_weights.db`) so exported values match what the registry itself reports.
* **ATP metrics** — existing agent-task-protocol instrumentation.
* **Memory-bus metrics** — existing hybrid memory-bus instrumentation.
* **Process metrics** — the Prometheus client's default Python process collectors.

The endpoint is **intentionally unauthenticated**, matching `/health`. The payload only exposes aggregate governance state — no task content, credentials, or per-request payloads — so the scraper needs no API key. If you expose the kernel outside a trusted network, front `/metrics` with a network policy or reverse-proxy rule instead of adding auth to the endpoint itself.

## Governance metric catalog

All governance series use the `artemis_` prefix.

| Metric                                      | Labels               | Meaning                                                                                                                                    |
| ------------------------------------------- | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `artemis_agent_trust_score`                 | `agent`, `tier`      | Current governance trust score per agent.                                                                                                  |
| `artemis_agent_violations`                  | `agent`              | Cumulative recorded governance violations.                                                                                                 |
| `artemis_agents`                            | `status`             | Agent count grouped by `active`, `suspended`, or `quarantined`.                                                                            |
| `artemis_hebbian_sentinel_alert`            | `agent`, `task_type` | `1` while a Hebbian Sentinel oscillation alert is active for this scope, otherwise `0`.                                                    |
| `artemis_hebbian_sentinel_oscillation_rate` | `agent`, `task_type` | Rolling sign-change rate the Sentinel uses to decide whether routing weights are oscillating.                                              |
| `artemis_governance_scrape_ok`              | `store`              | `1` if the kernel could read the named governance store during the scrape, `0` if not. A `0` means governance decisions are running blind. |

The collector runs once per scrape, so timing and cardinality are bounded by the Prometheus `scrape_interval` (30s by default in the shipped config).

## Alert rules

Prometheus loads `config/alerts.yml` as its rule file. Two rule groups ship out of the box.

### `artemis-governance`

| Alert                       | Expression (summary)                          | Severity | Fires after |
| --------------------------- | --------------------------------------------- | -------- | ----------- |
| `AgentQuarantined`          | `artemis_agents{status="quarantined"} > 0`    | critical | 1m          |
| `AgentSuspended`            | `artemis_agents{status="suspended"} > 0`      | warning  | 5m          |
| `AgentTrustCollapse`        | `artemis_agent_trust_score < 0.3`             | warning  | 10m         |
| `GovernanceViolationSpike`  | `increase(artemis_agent_violations[15m]) > 3` | warning  | immediately |
| `HebbianSentinelAlert`      | `artemis_hebbian_sentinel_alert > 0`          | warning  | 1m          |
| `GovernanceStoreUnreadable` | `artemis_governance_scrape_ok == 0`           | critical | 2m          |

`GovernanceViolationSpike` intentionally trips past the sandbox's 3-strike quarantine threshold — by the time it pages, the agent is already at or beyond the point where the registry would quarantine it. `GovernanceStoreUnreadable` and `AgentQuarantined` are the two critical alerts and should route to whatever paging channel handles operator response.

### `artemis-availability`

| Alert               | Expression                              | Severity | Fires after |
| ------------------- | --------------------------------------- | -------- | ----------- |
| `KernelMetricsDown` | `up{job="artemis-kernel"} == 0`         | critical | 2m          |
| `RedisDown`         | `redis_up == 0 or up{job="redis"} == 0` | critical | 2m          |

`RedisDown` uses `or up{job="redis"} == 0` on purpose: when the exporter itself is unreachable, `redis_up` is absent rather than `0`, and the scrape-target `up{}` series covers that gap.

To route these alerts to a receiver, add an Alertmanager service and point Prometheus at it — the shipped stack ships the rules and their evaluation, but not a notification pipeline.

## Redis exporter

A `redis-exporter` sidecar (`oliver006/redis_exporter:v1.66.0-alpine`, pinned) translates Redis `INFO` into Prometheus metrics. It reuses the same `REDIS_PASSWORD` contract the Redis service enforces, so no separate credential is required:

```yaml theme={null}
redis-exporter:
  image: oliver006/redis_exporter:v1.66.0-alpine
  environment:
    - REDIS_ADDR=redis://redis:6379
    - REDIS_PASSWORD=${REDIS_PASSWORD:?REDIS_PASSWORD must be set}
```

Prometheus scrapes it as the `redis` job. Standard `redis_*` series (memory, clients, uptime, replication) are available for dashboards and additional rules.

## Grafana dashboard

Grafana is provisioned via two mounted directories:

* `config/grafana/datasources/prometheus.yml` — declares the Prometheus datasource pointing at `http://prometheus:9090`.
* `config/grafana/dashboards/provider.yml` — registers the dashboards folder so Grafana auto-loads every JSON dashboard from `config/grafana/dashboards/`.

The **Artemis City Governance** dashboard (`config/grafana/dashboards/governance.json`) ships with:

* Stat tiles for quarantined agents, suspended agents, active Sentinel alerts, governance-store health, and Redis reachability.
* Timeseries for `artemis_agent_trust_score` and `artemis_hebbian_sentinel_oscillation_rate`.
* Violations per agent, and Redis memory and client-connection panels.

Both the datasource and the dashboard load automatically on first Grafana boot from the compose volumes — no manual import step is required.

## Extending the stack

* **Adding a metric** — export it through the Prometheus client's default registry from the kernel process. Follow the same import-guarded, reimport-safe pattern as the existing governance and ATP collectors so hot reloads do not register a series twice.
* **Adding an alert** — append a rule to `config/alerts.yml`; Prometheus reloads it on restart. Keep governance rules in the `artemis-governance` group and availability rules in `artemis-availability`.
* **Adding a dashboard** — drop the JSON into `config/grafana/dashboards/`. The provisioned provider picks it up on next Grafana start.