Monitoring

Prometheus, Grafana, and the redis-exporter sidecar for scraping governance metrics, firing alerts on quarantines and trust collapses, and visualizing agent state.

Artemis City ships a bootable monitoring stack in docker-compose.yaml: the FastAPI kernel exposes a Prometheus-format /metrics endpoint covering governance, ATP, and memory-bus instrumentation; a redis-exporter sidecar covers Redis health; Prometheus scrapes both and evaluates a governance alert-rules file; Grafana provisions a pre-built Artemis City Governance dashboard on top of that data.

When to use this

• You are running Artemis City from compose and want operator-facing visibility into agent trust, quarantines, and violations.
• You need alerts to page when governance stores go unreadable, an agent lands in quarantine, or Redis drops offline.
• You are extending governance and want to know which Prometheus series are already exported so you can reuse them in dashboards and rules.

If you only run unit tests or a single-process kernel locally, you can skip this page — the /metrics endpoint still responds, but nothing scrapes it.

Bringing the stack up

The monitoring services are part of the same compose file as the kernel. From the repo root:

docker compose up -d kernel redis redis-exporter prometheus grafana

Once the containers are healthy:

• Kernel metrics: http://localhost:8000/metrics
• Prometheus UI: http://localhost:9090
• Grafana UI: http://localhost:3000 (admin password from GRAFANA_PASSWORD)

Prometheus stores its TSDB in the prometheus-data compose volume with a 30-day retention window. Grafana state lives in the grafana-data volume.

GRAFANA_PASSWORD and REDIS_PASSWORD must be set before starting the stack; both services fail fast if they are missing. See Secrets setup.

The /metrics endpoint

GET /metrics on the FastAPI kernel serves the Prometheus default registry. That covers three families:

• Governance metrics — read at scrape time from the governance SQLite stores (agent_registry.db, hebbian_weights.db) so exported values match what the registry itself reports.
• ATP metrics — existing agent-task-protocol instrumentation.
• Memory-bus metrics — existing hybrid memory-bus instrumentation.
• Process metrics — the Prometheus client’s default Python process collectors.

The endpoint is intentionally unauthenticated, matching /health. The payload only exposes aggregate governance state — no task content, credentials, or per-request payloads — so the scraper needs no API key. If you expose the kernel outside a trusted network, front /metrics with a network policy or reverse-proxy rule instead of adding auth to the endpoint itself.

Governance metric catalog

All governance series use the artemis_ prefix.

Metric	Labels	Meaning
artemis_agent_trust_score	agent, tier	Current governance trust score per agent.
artemis_agent_violations	agent	Cumulative recorded governance violations.
artemis_agents	status	Agent count grouped by active, suspended, or quarantined.
artemis_hebbian_sentinel_alert	agent, task_type	1 while a Hebbian Sentinel oscillation alert is active for this scope, otherwise 0.
artemis_hebbian_sentinel_oscillation_rate	agent, task_type	Rolling sign-change rate the Sentinel uses to decide whether routing weights are oscillating.
artemis_governance_scrape_ok	store	1 if the kernel could read the named governance store during the scrape, 0 if not. A 0 means governance decisions are running blind.

The collector runs once per scrape, so timing and cardinality are bounded by the Prometheus scrape_interval (30s by default in the shipped config).

Alert rules

Prometheus loads config/alerts.yml as its rule file. Two rule groups ship out of the box.

artemis-governance

Alert	Expression (summary)	Severity	Fires after
AgentQuarantined	artemis_agents{status="quarantined"} > 0	critical	1m
AgentSuspended	artemis_agents{status="suspended"} > 0	warning	5m
AgentTrustCollapse	artemis_agent_trust_score < 0.3	warning	10m
GovernanceViolationSpike	increase(artemis_agent_violations[15m]) > 3	warning	immediately
HebbianSentinelAlert	artemis_hebbian_sentinel_alert > 0	warning	1m
GovernanceStoreUnreadable	artemis_governance_scrape_ok == 0	critical	2m

GovernanceViolationSpike intentionally trips past the sandbox’s 3-strike quarantine threshold — by the time it pages, the agent is already at or beyond the point where the registry would quarantine it. GovernanceStoreUnreadable and AgentQuarantined are the two critical alerts and should route to whatever paging channel handles operator response.

artemis-availability

Alert	Expression	Severity	Fires after
KernelMetricsDown	up{job="artemis-kernel"} == 0	critical	2m
RedisDown	redis_up == 0 or up{job="redis"} == 0	critical	2m

RedisDown uses or up{job="redis"} == 0 on purpose: when the exporter itself is unreachable, redis_up is absent rather than 0, and the scrape-target up{} series covers that gap.

To route these alerts to a receiver, add an Alertmanager service and point Prometheus at it — the shipped stack ships the rules and their evaluation, but not a notification pipeline.

Redis exporter

A redis-exporter sidecar (oliver006/redis_exporter:v1.66.0-alpine, pinned) translates Redis INFO into Prometheus metrics. It reuses the same REDIS_PASSWORD contract the Redis service enforces, so no separate credential is required:

redis-exporter:
  image: oliver006/redis_exporter:v1.66.0-alpine
  environment:
    - REDIS_ADDR=redis://redis:6379
    - REDIS_PASSWORD=${REDIS_PASSWORD:?REDIS_PASSWORD must be set}

Prometheus scrapes it as the redis job. Standard redis_* series (memory, clients, uptime, replication) are available for dashboards and additional rules.

Grafana dashboard

Grafana is provisioned via two mounted directories:

• config/grafana/datasources/prometheus.yml — declares the Prometheus datasource pointing at http://prometheus:9090.
• config/grafana/dashboards/provider.yml — registers the dashboards folder so Grafana auto-loads every JSON dashboard from config/grafana/dashboards/.

The Artemis City Governance dashboard (config/grafana/dashboards/governance.json) ships with:

• Stat tiles for quarantined agents, suspended agents, active Sentinel alerts, governance-store health, and Redis reachability.
• Timeseries for artemis_agent_trust_score and artemis_hebbian_sentinel_oscillation_rate.
• Violations per agent, and Redis memory and client-connection panels.

Both the datasource and the dashboard load automatically on first Grafana boot from the compose volumes — no manual import step is required.

Extending the stack

• Adding a metric — export it through the Prometheus client’s default registry from the kernel process. Follow the same import-guarded, reimport-safe pattern as the existing governance and ATP collectors so hot reloads do not register a series twice.
• Adding an alert — append a rule to config/alerts.yml; Prometheus reloads it on restart. Keep governance rules in the artemis-governance group and availability rules in artemis-availability.
• Adding a dashboard — drop the JSON into config/grafana/dashboards/. The provisioned provider picks it up on next Grafana start.