Docs/Authority/Sidecar and Operations

Sidecar and Operations

predicate-authorityd runs as a local sidecar for policy reload, revocation, identity task issuance, queue flushing, and health/status checks.

For a detailed walkthrough of how the sidecar integrates with Identity Providers (Okta, Entra, OIDC) and Local IDP mode, see How the Sidecar Works.

Installing the Sidecar

The predicate-authorityd sidecar is a Rust binary. Choose your preferred installation method:

Option A: Install via SDK (Recommended)

Python:

# Use quotes for zsh compatibility
pip install "predicate-authority[sidecar]"

# IMPORTANT: The binary is NOT downloaded automatically during pip install.
# You must manually download it:
predicate-download-sidecar

TypeScript/Node.js:

# Binary is automatically included for your platform
npm install @predicatesystems/authorityd

Option B: Download Binary Directly

Download the binary for your platform from GitHub Releases:

Platform	Binary
macOS (Apple Silicon)	`predicate-authorityd-darwin-arm64.tar.gz`
macOS (Intel)	`predicate-authorityd-darwin-x64.tar.gz`
Linux (x64)	`predicate-authorityd-linux-x64.tar.gz`
Linux (ARM64)	`predicate-authorityd-linux-arm64.tar.gz`
Windows (x64)	`predicate-authorityd-windows-x64.zip`

# Example for macOS Apple Silicon
curl -LO https://github.com/PredicateSystems/predicate-authority-sidecar/releases/latest/download/predicate-authorityd-darwin-arm64.tar.gz
tar -xzf predicate-authorityd-darwin-arm64.tar.gz
chmod +x predicate-authorityd

Option C: Use the Download Helper

If you have the Python SDK installed:

predicate-download-sidecar
# Or with a specific version:
predicate-download-sidecar --version v0.3.8

Binary location after download:

macOS: ~/Library/Application Support/predicate-authority/bin/predicate-authorityd
Linux: ~/.local/share/predicate-authority/bin/predicate-authorityd
Windows: %LOCALAPPDATA%/predicate-authority/bin/predicate-authorityd.exe

Sidecar CLI Reference

IMPORTANT: CLI arguments must be placed before the run subcommand.

GLOBAL OPTIONS (use before 'run'):
  -c, --config <FILE>           Path to TOML config file [env: PREDICATE_CONFIG]
      --host <HOST>             Host to bind to [env: PREDICATE_HOST] [default: 127.0.0.1]
      --port <PORT>             Port to bind to [env: PREDICATE_PORT] [default: 8787]
      --mode <MODE>             local_only or cloud_connected [env: PREDICATE_MODE]
      --policy-file <PATH>      Path to policy JSON [env: PREDICATE_POLICY_FILE]
      --identity-file <PATH>    Path to local identity registry [env: PREDICATE_IDENTITY_FILE]
      --log-level <LEVEL>       trace, debug, info, warn, error [env: PREDICATE_LOG_LEVEL]
      --control-plane-url <URL> Control-plane URL [env: PREDICATE_CONTROL_PLANE_URL]
      --tenant-id <ID>          Tenant ID [env: PREDICATE_TENANT_ID]
      --project-id <ID>         Project ID [env: PREDICATE_PROJECT_ID]
      --predicate-api-key <KEY> API key [env: PREDICATE_API_KEY]
      --sync-enabled            Enable control-plane sync [env: PREDICATE_SYNC_ENABLED]
      --fail-open               Fail open if control-plane unreachable [env: PREDICATE_FAIL_OPEN]

IDENTITY PROVIDER OPTIONS:
      --identity-mode <MODE>    local, local-idp, oidc, entra, or okta [env: PREDICATE_IDENTITY_MODE]
      --allow-local-fallback    Allow local/local-idp in cloud_connected mode
      --idp-token-ttl-s <SECS>  IdP token TTL seconds [default: 300]
      --mandate-ttl-s <SECS>    Mandate TTL seconds [default: 300]

LOCAL IDP OPTIONS (for identity-mode=local-idp):
      --local-idp-issuer <URL>  Issuer URL [env: LOCAL_IDP_ISSUER]
      --local-idp-audience <AUD> Audience [env: LOCAL_IDP_AUDIENCE]
      --local-idp-signing-key-env <VAR> Env var for signing key [default: LOCAL_IDP_SIGNING_KEY]

OIDC OPTIONS (for identity-mode=oidc):
      --oidc-issuer <URL>       Issuer URL [env: OIDC_ISSUER]
      --oidc-client-id <ID>     Client ID [env: OIDC_CLIENT_ID]
      --oidc-audience <AUD>     Audience [env: OIDC_AUDIENCE]

ENTRA OPTIONS (for identity-mode=entra):
      --entra-tenant-id <ID>    Tenant ID [env: ENTRA_TENANT_ID]
      --entra-client-id <ID>    Client ID [env: ENTRA_CLIENT_ID]
      --entra-audience <AUD>    Audience [env: ENTRA_AUDIENCE]

OKTA OPTIONS (for identity-mode=okta):
      --okta-issuer <URL>       Okta issuer URL [env: OKTA_ISSUER]
      --okta-client-id <ID>     Okta client ID [env: OKTA_CLIENT_ID]
      --okta-audience <AUD>     Okta audience [env: OKTA_AUDIENCE]
      --okta-required-claims    Required claims (comma-separated)
      --okta-required-scopes    Required scopes (comma-separated)
      --okta-required-roles     Required roles/groups (comma-separated)
      --okta-allowed-tenants    Allowed tenant IDs (comma-separated)

COMMANDS:
  run          Start the daemon (default)
  dashboard    Start with interactive TUI
  init-config  Generate example config file
  check-config Validate config file
  version      Show version info

Terminal Dashboard

The sidecar includes an interactive terminal dashboard for real-time monitoring of authorization decisions:

./predicate-authorityd --policy-file policy.json dashboard

Dashboard Layout

┌────────────────────────────────────────────────────────────────────────────┐
│  PREDICATE AUTHORITY v0.5.0    MODE: strict  [LIVE]  UPTIME: 2h 34m  [?]  │
│  Policy: loaded                Rules: 12 active      [Q:quit P:pause]     │
├─────────────────────────────────────────┬──────────────────────────────────┤
│  LIVE AUTHORITY GATE                    │  METRICS                         │
│                                         │                                  │
│  [ ✓ ALLOW ] agent:web                  │  Total Requests:    1,870        │
│    browser.navigate → github.com        │  ├─ Allowed:        1,847 (98.8%)│
│    m_7f3a2b1c | 0.4ms                   │  └─ Blocked:           23  (1.2%)│
│                                         │                                  │
│  [ ✗ DENY  ] agent:scraper              │  Throughput:        12.3 req/s   │
│    fs.write → ~/.ssh/config             │  Avg Latency:       0.8ms        │
│    EXPLICIT_DENY | 0.2ms                │                                  │
│                                         │  TOKEN CONTEXT SAVED             │
│  [ ✓ ALLOW ] agent:worker               │  Blocked early:     23 actions   │
│    browser.click → button#checkout      │  Est. tokens saved: ~4,140       │
│    m_9c2d4e5f | 0.6ms                   │                                  │
├─────────────────────────────────────────┴──────────────────────────────────┤
│  Generated 47 proofs this session. Run `predicate login` to sync to vault.│
└────────────────────────────────────────────────────────────────────────────┘

Dashboard Demo: Real-time ALLOW/DENY decisions as authorization requests flow through the sidecar.

Keyboard Shortcuts

Key	Action
`j/k` or `↑/↓`	Scroll through events
`f`	Cycle filter: ALL → DENY → agent input
`/`	Enter agent filter mode directly
`c`	Clear filter (show all events)
`P`	Pause/resume live updates
`?`	Toggle help overlay
`Q` or `Esc`	Quit dashboard

Live Filtering

Filter authorization events in real-time to focus on specific patterns:

Filter by Denied Events Only

Press f once to show only denied events:

┌─ LIVE AUTHORITY GATE ──────────────────────────────────────────────────────┐
│  [FILTER: DENY ONLY]                                                       │
│                                                                            │
│  [ ✗ DENY  ] agent:scraper                                                 │
│    fs.write → ~/.ssh/config                                                │
│    EXPLICIT_DENY | 0.2ms                                                   │
│                                                                            │
│  [ ✗ DENY  ] agent:web                                                     │
│    admin.delete → /users/123                                               │
│    NO_MATCHING_ALLOW | 0.3ms                                               │
└────────────────────────────────────────────────────────────────────────────┘

Filter by Agent ID

Press f again (or / directly) to enter agent filter mode, then type the agent ID:

┌─ LIVE AUTHORITY GATE ──────────────────────────────────────────────────────┐
│  [FILTER: agent:web]                                                       │
│                                                                            │
│  [ ✓ ALLOW ] agent:web                                                     │
│    browser.navigate → github.com                                           │
│    m_7f3a2b1c | 0.4ms                                                      │
│                                                                            │
│  [ ✓ ALLOW ] agent:web                                                     │
│    browser.click → button#submit                                           │
│    m_8e4b3c2d | 0.5ms                                                      │
├────────────────────────────────────────────────────────────────────────────┤
│  Filter agent: web█                                          [Enter/Esc]   │
└────────────────────────────────────────────────────────────────────────────┘

Filter supports partial matching—typing web matches agent:web, agent:web-scraper, etc.

Audit Mode (Dry-Run)

Audit mode allows you to test policy changes without blocking actions. Denied requests are logged but allowed through, displayed as [ ⚠ WOULD DENY ] in yellow.

Enable Audit Mode

Option 1: CLI flag

./predicate-authorityd --policy-file policy.json --audit-mode dashboard

Option 2: Auto-detection from policy filename

Policy files containing audit, dry-run, or dryrun in the filename automatically enable audit mode:

# These all auto-enable audit mode:
./predicate-authorityd --policy-file policy-audit.json dashboard
./predicate-authorityd --policy-file dry-run-policy.json dashboard
./predicate-authorityd --policy-file new-rules-dryrun.json dashboard

Audit Mode Display

When audit mode is active:

Header shows [AUDIT] status indicator
Denied events display as [ ⚠ WOULD DENY ] in yellow instead of [ ✗ DENY ] in red
Border color changes to yellow

┌────────────────────────────────────────────────────────────────────────────┐
│  PREDICATE AUTHORITY v0.5.0    MODE: strict  [AUDIT]  UPTIME: 15m    [?]  │
│  Policy: policy-audit.json     Rules: 8 active       [Q:quit P:pause]     │
├─────────────────────────────────────────┬──────────────────────────────────┤
│  LIVE AUTHORITY GATE                    │  METRICS                         │
│                                         │                                  │
│  [ ✓ ALLOW ] agent:web                  │  Total Requests:      127        │
│    browser.navigate → github.com        │  ├─ Allowed:          124 (97.6%)│
│    m_7f3a2b1c | 0.4ms                   │  └─ Would Block:        3  (2.4%)│
│                                         │                                  │
│  [ ⚠ WOULD DENY ] agent:scraper         │  Throughput:         8.2 req/s   │
│    fs.write → ~/.ssh/config             │  Avg Latency:        0.6ms       │
│    EXPLICIT_DENY | 0.2ms                │                                  │
└─────────────────────────────────────────┴──────────────────────────────────┘

Use Cases

Policy testing: Validate new rules before enforcement
Migration: Test policy changes in production without risk
Debugging: Understand what would be blocked without affecting agents
Compliance review: Generate audit logs showing policy effectiveness

Start sidecar in local mode

# Arguments BEFORE 'run' subcommand
./predicate-authorityd \
  --host 127.0.0.1 \
  --port 8787 \
  --mode local_only \
  --policy-file policy.json \
  run

Using environment variables:

export PREDICATE_HOST=127.0.0.1
export PREDICATE_PORT=8787
export PREDICATE_MODE=local_only
export PREDICATE_POLICY_FILE=policy.json

./predicate-authorityd run

Using a config file:

# Generate example config
./predicate-authorityd init-config --output config.toml

# Run with config
./predicate-authorityd --config config.toml run

Enable local identity registry

./predicate-authorityd \
  --host 127.0.0.1 \
  --port 8787 \
  --mode local_only \
  --policy-file policy.json \
  --identity-file ./local-identities.json \
  run

Key operations endpoints

GET /health
GET /status
GET /metrics
POST /ledger/flush-now
GET /ledger/dead-letter
POST /ledger/requeue

Cloud-connected mode (control-plane sync)

Connect to Predicate Authority control-plane for policy sync, revocation push, and audit forwarding:

export PREDICATE_API_KEY="your-api-key"

./predicate-authorityd \
  --host 127.0.0.1 \
  --port 8787 \
  --mode cloud_connected \
  --policy-file policy.json \
  --control-plane-url https://api.predicatesystems.dev \
  --tenant-id your-tenant \
  --project-id your-project \
  --predicate-api-key "$PREDICATE_API_KEY" \
  --sync-enabled \
  run

Identity Provider Modes

The sidecar supports multiple identity modes for token validation on authorization requests:

local (default): No token validation. Suitable for development and trusted environments.
local-idp: Self-issued JWT tokens for ephemeral task identities.
oidc: Generic OIDC provider integration.
entra: Microsoft Entra ID (Azure AD) integration.
okta: Enterprise Okta integration with JWKS validation.

Local IDP mode

Use --identity-mode local-idp for self-issued JWT tokens with ephemeral task identities:

export LOCAL_IDP_SIGNING_KEY="your-signing-key"

./predicate-authorityd \
  --host 127.0.0.1 \
  --port 8787 \
  --mode local_only \
  --policy-file policy.json \
  --identity-mode local-idp \
  --local-idp-issuer "http://localhost/predicate-local-idp" \
  --local-idp-audience "api://predicate-authority" \
  run

OIDC identity mode

Use --identity-mode oidc for generic OIDC provider integration:

./predicate-authorityd \
  --host 127.0.0.1 \
  --port 8787 \
  --mode cloud_connected \
  --policy-file policy.json \
  --identity-mode oidc \
  --oidc-issuer "https://your-oidc-provider/.well-known/openid-configuration" \
  --oidc-client-id "your-client-id" \
  --oidc-audience "api://predicate-authority" \
  run

Entra identity mode

Use --identity-mode entra for Microsoft Entra ID (Azure AD) integration:

./predicate-authorityd \
  --host 127.0.0.1 \
  --port 8787 \
  --mode cloud_connected \
  --policy-file policy.json \
  --identity-mode entra \
  --entra-tenant-id "your-tenant-id" \
  --entra-client-id "your-client-id" \
  --entra-audience "api://predicate-authority" \
  run

Okta identity mode

Use --identity-mode okta for enterprise Okta integration with JWKS validation:

export OKTA_ISSUER="https://<org>.okta.com/oauth2/default"
export OKTA_CLIENT_ID="<okta-client-id>"
export OKTA_AUDIENCE="api://predicate-authority"

./predicate-authorityd \
  --host 127.0.0.1 \
  --port 8787 \
  --mode cloud_connected \
  --policy-file policy.json \
  --identity-mode okta \
  --okta-issuer "$OKTA_ISSUER" \
  --okta-client-id "$OKTA_CLIENT_ID" \
  --okta-audience "$OKTA_AUDIENCE" \
  --okta-required-claims "sub,tenant_id" \
  --okta-required-scopes "authority:check" \
  --okta-allowed-tenants "tenant-a,tenant-b" \
  --idp-token-ttl-s 300 \
  --mandate-ttl-s 300 \
  run

Safety notes:

idp-token-ttl-s must be >= mandate-ttl-s (enforced at startup)
In cloud_connected mode, local or local-idp identity requires --allow-local-fallback

Okta token exchange/OBO compatibility (capability-gated)

Run this in AgentIdentity repo to determine if your tenant supports IdP token exchange:

export OKTA_OBO_COMPAT_CHECK_ENABLED=1

# Tenant supports token exchange:
export OKTA_SUPPORTS_TOKEN_EXCHANGE=true
python3 -m pytest tests/test_okta_obo_compatibility.py -k "live_check_when_enabled"

# Tenant does not support token exchange:
export OKTA_SUPPORTS_TOKEN_EXCHANGE=false
python3 -m pytest tests/test_okta_obo_compatibility.py -k "live_check_when_enabled"

If token exchange is not supported, delegation should use authority mandate delegation fallback.

Entra OBO compatibility (capability-gated)

Run this in AgentIdentity repo to validate Entra OBO capability:

export ENTRA_OBO_COMPAT_CHECK_ENABLED=1

# Tenant supports OBO and user assertion is available:
export ENTRA_SUPPORTS_OBO=true
export ENTRA_USER_ASSERTION="<user-assertion-jwt>"
python3 -m pytest tests/test_entra_obo_compatibility.py -k "live_check_when_enabled"

# Tenant does not support OBO (or grant not enabled):
export ENTRA_SUPPORTS_OBO=false
python3 -m pytest tests/test_entra_obo_compatibility.py -k "live_check_when_enabled"

Demo script:

python examples/delegation/entra_obo_compat_demo.py \
  --tenant-id "$ENTRA_TENANT_ID" \
  --client-id "$ENTRA_CLIENT_ID" \
  --client-secret "$ENTRA_CLIENT_SECRET" \
  --scope "$ENTRA_SCOPE"

If OBO is unavailable, delegation should use authority mandate delegation fallback.

OIDC token exchange compatibility (capability-gated)

Run this in AgentIdentity repo when using a generic OIDC provider:

export OIDC_COMPAT_CHECK_ENABLED=1

# Provider supports token exchange:
export OIDC_SUPPORTS_TOKEN_EXCHANGE=true
export OIDC_SUBJECT_TOKEN="<subject-access-token>"
python3 -m pytest tests/test_oidc_compatibility.py -k "live_check_when_enabled"

# Provider does not support token exchange:
export OIDC_SUPPORTS_TOKEN_EXCHANGE=false
python3 -m pytest tests/test_oidc_compatibility.py -k "live_check_when_enabled"

Demo script:

python examples/delegation/oidc_compat_demo.py \
  --issuer "$OIDC_ISSUER" \
  --client-id "$OIDC_CLIENT_ID" \
  --client-secret "$OIDC_CLIENT_SECRET" \
  --audience "$OIDC_AUDIENCE" \
  --scope "${OIDC_SCOPE:-authority:check}"

If token exchange is unavailable, delegation should use authority mandate delegation fallback.

Production safety patterns

Keep protected actions fail-closed.
Quarantine repeatedly failing queue items.
Monitor /status counters and queue depth.
Keep runbooks for manual flush and dead-letter requeue.

Control-plane connection

The sidecar can connect to the Predicate Authority control-plane for:

Policy sync: Pull policy updates via long-poll
Revocation push: Receive revocation events in near-real-time
Audit forwarding: Ship audit events to durable storage
Fleet management: Enroll sidecars and track fleet health

The default control-plane URL is https://api.predicatesystems.dev.

Basic control-plane connection

export PREDICATE_API_KEY="<api-key>"

./predicate-authorityd \
  --host 127.0.0.1 \
  --port 8787 \
  --mode cloud_connected \
  --policy-file policy.json \
  --control-plane-url https://api.predicatesystems.dev \
  --tenant-id "your-tenant" \
  --project-id "your-project" \
  --predicate-api-key "$PREDICATE_API_KEY" \
  run

Note: --control-plane-url defaults to https://api.predicatesystems.dev if not specified.

Control-plane long-poll sync (policy + revocation push)

Use this in cloud_connected mode when you want active authority updates from control-plane:

export PREDICATE_API_KEY="<api-key>"

./predicate-authorityd \
  --host 127.0.0.1 \
  --port 8787 \
  --mode cloud_connected \
  --policy-file policy.json \
  --control-plane-url https://api.predicatesystems.dev \
  --tenant-id "your-tenant" \
  --project-id "your-project" \
  --predicate-api-key "$PREDICATE_API_KEY" \
  --sync-enabled \
  run

Quick checks:

curl -s http://127.0.0.1:8787/status | jq '.control_plane_sync_poll_count, .control_plane_sync_update_count, .control_plane_sync_error_count, .control_plane_last_sync_error'
curl -s http://127.0.0.1:8787/metrics | rg "predicate_authority_control_plane_sync_total"

Sidecar enrollment and fleet sync

Use this flow when running many sidecars (e.g., OpenClaw bot fleets) and you want:

Short-lived sidecar identity tokens
Control-plane fleet visibility
Low-latency revocation push (SSE) for kill-switch scenarios

1. Enroll sidecar to get a short-lived token

curl -s -X POST "https://api.predicatesystems.dev/v1/sidecars/enroll" \
  -H "Content-Type: application/json" \
  -d '{
    "tenant_id":"your-tenant",
    "sidecar_id":"sidecar-bot-01",
    "version":"1.0.0",
    "api_key":"<enrollment-key>"
  }' | jq

Response includes:

sidecar_token: Bearer token for sidecar-authenticated sync endpoints
expires_at: Token expiry timestamp
sidecar: Sidecar metadata (id/version/last_seen/active)

2. Send periodic heartbeat

curl -s -X POST "https://api.predicatesystems.dev/v1/sidecars/heartbeat" \
  -H "Authorization: Bearer $SIDECAR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"version":"1.0.1"}' | jq

Recommended cadence: every 15-30 seconds.

3. Sidecar sync pull (long-poll)

curl -s "https://api.predicatesystems.dev/v1/sync/authority-updates/sidecar?tenant_id=your-tenant&wait_timeout_s=15" \
  -H "Authorization: Bearer $SIDECAR_TOKEN" | jq

Returns policy + revocation snapshot and a sync_token.

4. Revocation push stream (SSE)

Subscribe for near-real-time revocation events:

curl -N "https://api.predicatesystems.dev/v1/sync/authority-updates/stream?tenant_id=your-tenant&last_event_id=0" \
  -H "Authorization: Bearer $SIDECAR_TOKEN"

When operators trigger revocations (including global kill-switch), stream events are pushed with event: revocation.

5. Fleet visibility endpoint

curl -s "https://api.predicatesystems.dev/v1/sidecars?tenant_id=your-tenant&active_only=true" \
  -H "Authorization: Bearer $TOKEN" | jq

Use this to track active sidecar count, versions, and last-seen status.

Failure handling

If SSE connection drops, reconnect and pass the last processed event_id
If sidecar token expires, re-enroll and rotate token in sidecar runtime
If control-plane is unreachable, continue local fail-closed guard behavior and retry sync/stream with backoff

Audit logging

Local sidecar logs are ephemeral by design (24h TTL, redacted payloads). For production audit requirements, connect to the control-plane.

See Audit Logging & Provenance for:

Local vs. cloud audit strategies
Compliance considerations for distributed fleets
Integrity proofs and audit exports

Audit integrity endpoints (Merkle proofs)

Use control-plane integrity endpoints to prove an event is included in a tenant audit set:

curl -s "http://127.0.0.1:8080/v1/audit/integrity/root?tenant_id=<tenant>" \
  -H "Authorization: Bearer $TOKEN" | jq

curl -s "http://127.0.0.1:8080/v1/audit/integrity/proof/<event_id>?tenant_id=<tenant>" \
  -H "Authorization: Bearer $TOKEN" | jq

Circuit breaker and Kafka streaming ops notes

During upstream DB distress, control-plane can fail fast with 503 store_circuit_open:<operation>.
If Kafka streaming is enabled with fail-closed, stream outages can return 503 event_stream_unavailable:<topic>.
With Kafka fail-open, core authority paths continue while event emission degrades.

Example recovery flow

# trigger immediate flush
curl -s -X POST http://127.0.0.1:8787/ledger/flush-now \
  -H "Content-Type: application/json" \
  -d '{"max_items":50}' | jq

# inspect quarantined events
curl -s http://127.0.0.1:8787/ledger/dead-letter | jq

# requeue one item
curl -s -X POST http://127.0.0.1:8787/ledger/requeue \
  -H "Content-Type: application/json" \
  -d '{"queue_item_id":"q_abc123"}' | jq

Fastest Local Validation

How the Sidecar Works