Monitoring

Dugite provides two complementary monitoring tools: a terminal dashboard (dugite-monitor) for quick at-a-glance status, and a Prometheus-compatible metrics endpoint for production alerting and dashboards.

Terminal Dashboard (dugite-monitor)

dugite-monitor is a standalone binary that renders a real-time status dashboard in the terminal by polling the node's Prometheus endpoint. It requires no external infrastructure and works over SSH.

# Auto-discover a running dugite-node and attach
dugite-monitor

# Monitor a specific endpoint (skips discovery)
dugite-monitor --metrics-url http://192.168.1.100:12796/metrics

# Pin the epoch length instead of auto-detecting it from dugite_network_magic
dugite-monitor --network-magic 2

# Show disk usage for the node's database volume in the Resources panel
dugite-monitor --db-path ./db-preview

When --metrics-url is omitted, dugite-monitor enumerates running dugite-node processes (via sysinfo + netstat2) and probes their /metrics endpoints. One node found: it attaches silently. Multiple: a selection dialog appears. None: it falls back to http://localhost:12798/metrics.

Careful: that fallback (12798) is not the node's default metrics port (12796 — see Which port? below). The fallback only matters when discovery finds nothing, in which case there is usually no node to monitor anyway. If the monitor reports no data, pass --metrics-url explicitly with the port your config actually pins.

The dashboard displays five panels:

• Node — role, network, version, era, uptime
• Chain — epoch progress bar, block/slot/tip metrics, density, forks, tx counts
• Connections — P2P state, inbound/outbound, cold/warm/hot, uni/bi/duplex counts
• Resources — CPU %, live memory, RSS memory (plus disk when --db-path is set)
• Peers — RTT bands (0-50 ms, 50-100 ms, 100-200 ms, 200 ms+), min/avg/max RTT

Metrics are polled once per second. The interval is a compile-time constant — there is no flag to change it.

Prometheus Metrics Endpoint

Dugite exposes a Prometheus-compatible metrics endpoint for monitoring node health and sync progress.

Metrics Endpoint

The metrics server responds to any unrecognised HTTP path with Prometheus exposition format metrics:

http://localhost:12796/metrics

Which port?

The built-in fallback is 12796, deliberately offset from cardano-node's 12798 so a dugite node and a Haskell node can co-exist on one host. The shipped per-network configs each pin a port explicitly, so in practice the port comes from the config file:

Resolution order, highest priority first:

1. --no-metrics → server disabled.
2. --metrics-port <PORT> → explicit operator override (wins even over TurnOnLogMetrics=false).
3. TurnOnLogMetrics: false in the config JSON → server disabled (master off-switch, matching cardano-node).
4. MetricsPort in the config JSON.
5. Built-in default 12796.

Pass --require-metrics to make a bind failure a fatal startup error instead of a logged warning.

The examples below use 12796. Substitute your network's port from the table above.

Example response:

# HELP dugite_blocks_received_total Total blocks received from peers
# TYPE dugite_blocks_received_total counter
dugite_blocks_received_total 1523847

# HELP dugite_blocks_applied_total Total blocks applied to ledger
# TYPE dugite_blocks_applied_total counter
dugite_blocks_applied_total 1523845

# HELP dugite_slot_number Current slot number
# TYPE dugite_slot_number gauge
dugite_slot_number 142857392

# HELP dugite_block_number Current block number
# TYPE dugite_block_number gauge
dugite_block_number 11283746

# HELP dugite_epoch_number Current epoch number
# TYPE dugite_epoch_number gauge
dugite_epoch_number 512

# HELP dugite_sync_progress_percent Chain sync progress (0-10000, divide by 100 for %)
# TYPE dugite_sync_progress_percent gauge
dugite_sync_progress_percent 9542

# HELP dugite_utxo_count Number of entries in the UTxO set
# TYPE dugite_utxo_count gauge
dugite_utxo_count 15234892

# HELP dugite_mempool_tx_count Number of transactions in the mempool
# TYPE dugite_mempool_tx_count gauge
dugite_mempool_tx_count 42

# HELP dugite_peers_connected Number of connected peers
# TYPE dugite_peers_connected gauge
dugite_peers_connected 8

Health Endpoint

The metrics server exposes a /health endpoint for monitoring node status:

GET http://localhost:12796/health

Always returns 200 OK. The status field carries the verdict:

• healthy — sync progress >= 99.9%
• syncing — actively catching up to chain tip
• stalled — no blocks received for > 5 minutes AND sync < 99%

{
  "status": "healthy",
  "uptime_seconds": 3421,
  "slot_number": 142857392,
  "block_number": 11283746,
  "epoch_number": 512,
  "sync_progress": 99.95,
  "peers_connected": 8,
  "last_block_received_at": "2026-03-14T12:34:56.789Z"
}

last_block_received_at is null until the first block arrives.

Readiness Endpoint

For Kubernetes readiness probes:

GET http://localhost:12796/ready

Returns 200 OK when sync_progress >= 99.9%, 503 Service Unavailable otherwise:

{"ready": true}

or:

{"ready": false, "sync_progress": 75.42}

Liveness Endpoint

For Kubernetes liveness probes — this is the one that should restart a wedged pod:

GET http://localhost:12796/live

Returns 200 OK when a block has been applied within --liveness-threshold-secs (default 600), or when no block has arrived yet but the node has been up for less than that window (warm-up grace). Otherwise 503:

{"alive": true, "threshold_secs": 600}

{"alive": false, "threshold_secs": 600, "last_block_received_at": "2026-03-14T12:34:56.789Z"}

Set --liveness-threshold-secs 0 to make /live always return 200.

/ready tracks sync progress; /live tracks forward progress. A node that is 40% synced but applying blocks is correctly not-ready and alive. Wiring a liveness probe to /ready will restart-loop a node that is merely still syncing.

EKG Compatibility Endpoint

GET http://localhost:12796/ekg

Returns the nested-object JSON layout that cardano-node's EKG (System.Remote.Monitoring) exposes, so legacy gLiveView / CNTools dashboards that poll port 12788 can be pointed at dugite unmodified.

Available Metrics

Counters

Gauges

Chain and sync

Peers and connections

Mempool and transactions

Ledger and governance

Process and host

Histograms

Histograms expose _bucket, _count, and _sum suffixes for standard Prometheus histogram queries.

Note: the block-fetch histogram is named dugite_peer_block_fetch_range_ms, not dugite_peer_block_fetch_ms. The bundled Grafana dashboard still queries the old name — see the caveat under Grafana Dashboard.

Prometheus Configuration

Add the Dugite node as a scrape target in your prometheus.yml:

scrape_configs:
  - job_name: 'dugite'
    scrape_interval: 15s
    static_configs:
      - targets: ['localhost:12800']    # mainnet; see the port table above
        labels:
          network: 'mainnet'
          node: 'relay-1'

A ready-made config is committed at config/monitoring/prometheus.yml. It scrapes every well-known dugite port so the dashboard works regardless of which just run-{bp,relay} <network> recipe is active — down targets simply show as DOWN in /targets:

Alert rules live alongside it in config/monitoring/prometheus-alerts.yml, covering forge failures, rollback rate, tip age, hot-peer starvation, and RSS growth.

Grafana Dashboard

Dugite ships with a pre-built Grafana dashboard at config/monitoring/grafana-dashboard.json. The dashboard covers all node metrics organized into nine sections:

• Overview — Sync progress gauge, block height, epoch, slot, connected peers, blocks forged
• Node Health — Uptime, disk available (stat + time series)
• Sync & Throughput — Sync progress over time, block apply/receive rate (blk/s), block height, rollbacks
• Peers — Connected peer count over time, peer state breakdown (hot/warm/cold stacked)
• Mempool & Transactions — Mempool tx count, mempool size (bytes), transaction rate (received/validated/rejected)
• Ledger State — UTxO set size, stake delegations, treasury balance (ADA), registered stake pools
• Governance — Registered DReps, active governance proposals
• Block Production — Total blocks forged, block forge rate (blk/h)
• Network Latency — Handshake RTT and block fetch latency percentiles (p50/p95/p99), request counts
• Validation Errors — Error breakdown by type (stacked bars), error totals (bar chart)

Known stale panel: the block-fetch latency panels query dugite_peer_block_fetch_ms_bucket, but the node emits dugite_peer_block_fetch_range_ms_bucket. Those panels render empty until the dashboard JSON is updated. Every other metric referenced by the dashboard matches a metric the node actually emits.

Quick Start (Docker)

The fastest way to start a local monitoring stack is with the included script:

# Start Prometheus + Grafana
just monitor-start         # or: ./scripts/monitoring/start.sh

# Open the dashboard (admin/admin)
open http://localhost:3000/d/dugite-node/dugite-node

# Check status
just monitor-status        # or: ./scripts/monitoring/start.sh status

# Stop
just monitor-stop          # or: ./scripts/monitoring/start.sh stop

The script starts Prometheus (port 9090) and Grafana (port 3000) as Docker containers, auto-configures the Prometheus datasource, and imports the Dugite dashboard. Prometheus data is persisted in .monitoring-data/ so metrics survive restarts.

Environment variables for port customization:

Importing the Dashboard

1. Open Grafana and go to Dashboards > Import
2. Click Upload JSON file and select config/monitoring/grafana-dashboard.json
3. Select your Prometheus data source when prompted
4. Click Import

The dashboard includes an instance template variable so you can monitor multiple Dugite nodes (relays + block producer) from a single dashboard. It auto-refreshes every 30 seconds.

Provisioning

To auto-provision the dashboard, copy it into your Grafana provisioning directory:

cp config/monitoring/grafana-dashboard.json /etc/grafana/provisioning/dashboards/dugite.json

Add a dashboard provider in /etc/grafana/provisioning/dashboards/dugite.yaml:

apiVersion: 1
providers:
  - name: Dugite
    folder: Cardano
    type: file
    options:
      path: /etc/grafana/provisioning/dashboards
      foldersFromFilesStructure: false

Quick Start (macOS)

To quickly preview the dashboard locally with Homebrew:

# Install Prometheus and Grafana
brew install prometheus grafana

# Configure Prometheus to scrape Dugite
cat > /opt/homebrew/etc/prometheus.yml << 'EOF'
global:
  scrape_interval: 5s

scrape_configs:
  - job_name: dugite
    static_configs:
      - targets: ['localhost:12796']    # match your config's MetricsPort
EOF

# Provision the datasource
cat > "$(brew --prefix)/opt/grafana/share/grafana/conf/provisioning/datasources/dugite.yaml" << 'EOF'
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://localhost:9090
    isDefault: true
    uid: DS_PROMETHEUS
EOF

# Provision the dashboard
cat > "$(brew --prefix)/opt/grafana/share/grafana/conf/provisioning/dashboards/dugite.yaml" << 'EOF'
apiVersion: 1
providers:
  - name: Dugite
    folder: Cardano
    type: file
    options:
      path: /opt/homebrew/var/lib/grafana/dashboards
EOF

mkdir -p /opt/homebrew/var/lib/grafana/dashboards
sed 's/${DS_PROMETHEUS}/DS_PROMETHEUS/g' config/monitoring/grafana-dashboard.json \
  > /opt/homebrew/var/lib/grafana/dashboards/dugite.json

# Start services
brew services start prometheus
brew services start grafana

# Open the dashboard (default login: admin/admin)
open "http://localhost:3000/d/dugite-node/dugite-node"

To stop:

brew services stop prometheus grafana

Key Queries

Console Logging

In addition to the Prometheus endpoint, Dugite logs sync progress to the console every 5 seconds — but only while catching up. Once the node is following the tip (remaining would be 0) the line stops being emitted, so silence here means "synced", not "stalled". The fields are:

Example log line:

2026-03-12T12:34:56.789Z  INFO dugite_node::node: Syncing progress="95.42%" epoch=512 block=11283746 tip=11300000 remaining=16254 speed="312 blk/s" utxos=15234892

Log output can be directed to stdout, file, or systemd journal. See Logging for full details on output targets, file rotation, and log level configuration.