Audit Logging & Provenance
The predicate-authorityd sidecar generates a cryptographic proof event for every authorization mandate it issues or denies. How you handle these events depends on your environment.
Looking for Fleet-Wide Visibility?
If you need centralized logging across all your agents, MCPs, CLIs, and skills in one place, see the Centralized Fleet Logging section below or explore the full Predicate Vault documentation for fleet management, SIEM integrations, and compliance features.
Local Sidecar Logging (Default)
By default, the sidecar writes verification events and mandate receipts to a local queue file. This mode is designed for local development, single-agent testing, and CI/CD pipeline debugging.
predicate-authorityd \
--local-identity-enabled \
--local-identity-registry-file ./.predicate-authorityd/local-identities.jsonLocal queue endpoints:
GET /ledger/flush-queue- List pending audit eventsGET /ledger/dead-letter- List quarantined eventsPOST /ledger/flush-now- Trigger immediate flushPOST /ledger/requeue- Retry a quarantined event
Ephemeral by Design
Local sidecar logs are deliberately ephemeral:
- 24-hour TTL: Queue items auto-expire after 24 hours by default
- Payload redaction: Sensitive fields (
principal_id,action,resource) are redacted in API responses - No aggregation API: Each sidecar is isolated; no cross-node query capability
# Configure TTL (default: 86400 seconds / 24 hours)
--queue-item-ttl-seconds 86400This design reflects the operational reality of distributed systems, not an artificial limitation.
Constraints in Distributed Environments
Relying on isolated sidecar logs in a production, multi-agent fleet introduces strict operational constraints:
| Constraint | Impact |
|---|---|
| Ephemeral data loss | If a container crashes or scales down, local logs are destroyed. You lose the cryptographic proof of what the agent did. |
| Lack of immutability | Local log files are mutable. A compromised host can alter or delete logs to erase an agent's tracks. |
| Fleet blindness | No native way to query a single intent_hash or principal_id across 50+ distributed sidecars. |
| Compliance gaps | SOC2, ISO 27001, and similar frameworks require Write-Once-Read-Many (WORM) retention and strict access controls. |
Building a compliant audit pipeline from ephemeral sidecar logs requires:
- Log forwarding infrastructure (FluentBit, Logstash, Vector)
- Centralized storage with WORM guarantees (S3 Object Lock, Azure Immutable Blob)
- Custom indexing for
principal_id,intent_hash,mandate_idqueries - Merkle tree implementation for tamper-evident integrity proofs
- Retention policy enforcement and legal hold workflows
Production Governance: Predicate Audit Vault
For production workloads, connect sidecars to the Predicate Control Plane. Instead of writing to local disk, the sidecar streams signed proof events directly into the Audit Vault.
predicate-authorityd \
--mode cloud_connected \
--control-plane-enabled \
--control-plane-tenant-id "your-tenant" \
--control-plane-project-id "your-project"Capabilities
| Capability | Description |
|---|---|
| Immutable retention | Append-only event storage with configurable retention periods |
| Global search | Query across your entire fleet by principal_id, intent_hash, or mandate_id |
| Integrity proofs | Merkle tree inclusion proofs for any audit event |
| SIEM streaming | Webhook delivery to Datadog, Splunk, or any HTTP endpoint |
| Legal hold | Preserve audit exports beyond normal retention for investigations |
Integrity verification
Verify any audit event is included in the tenant's tamper-evident log:
# Get current Merkle root
curl -s "https://api.predicatesystems.dev/v1/audit/integrity/root?tenant_id=your-tenant" \
-H "Authorization: Bearer $TOKEN" | jq
# Get inclusion proof for specific event
curl -s "https://api.predicatesystems.dev/v1/audit/integrity/proof/<event_id>?tenant_id=your-tenant" \
-H "Authorization: Bearer $TOKEN" | jqAudit exports
Export audit events for compliance reporting or external analysis:
# Create export job
curl -s -X POST "https://api.predicatesystems.dev/v1/audit/exports" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{
"tenant_id": "your-tenant",
"start_time": "2024-01-01T00:00:00Z",
"end_time": "2024-01-31T23:59:59Z",
"format": "jsonl"
}' | jq
# Download completed export
curl -s "https://api.predicatesystems.dev/v1/audit/exports/<export_id>/download" \
-H "Authorization: Bearer $TOKEN" -o audit-export.jsonl.gzCentralized Fleet Logging
When running multiple AI agents across different environmentsβMCP servers, CLI tools, custom skills, browser automationβyou need a single place to see what every agent did.
The question: "How do I get audit and governance for all my agent tools in one centralized view?"
The answer: Connect your sidecars to the Predicate Control Plane. Every authorization decision from every sidecar streams to a unified audit log.
Architecture
βββββββββββββββββββ βββββββββββββββββββ βββββββββββββββββββ
β Agent + MCP β β Agent + CLI β β Agent + Skills β
β (Sidecar) β β (Sidecar) β β (Sidecar) β
ββββββββββ¬βββββββββ ββββββββββ¬βββββββββ ββββββββββ¬βββββββββ
β β β
β POST /v1/audit/events:batch β
β β β
βββββββββββββββββββββββΌββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββ
β Control Plane β
β βββββββββββββββββ β
β ProofEvents Table β
β Merkle Verifier β
β Retention Manager β
β Kafka/SSE Stream β
ββββββββββββ¬βββββββββββ
β
βΌ
βββββββββββββββββββββββ
β Vault UI β
β βββββββββββββββββ β
β Fleet-wide view β
β Audit timeline β
β Integrity checks β
βββββββββββββββββββββββWhat You Get
| Capability | Description |
|---|---|
| Fleet-wide aggregation | All sidecars report to a single control planeβsee every ALLOW/DENY across your entire infrastructure |
| Real-time streaming | Events stream via SSE or Kafka for live dashboards and alerting |
| Cross-agent queries | Search by principal_id, action, resource, or time range across all agents |
| Tamper-proof audit | Merkle tree integrity verification proves logs weren't modified |
| Compliance retention | Hot (30 days), Warm (1 year), Cold WORM (7 years) retention tiers |
Quick Setup
# Connect sidecar to control plane
predicate-authorityd \
--mode cloud_connected \
--control-plane-enabled \
--control-plane-tenant-id "your-tenant" \
--control-plane-project-id "your-project"Once connected, open the Predicate Vault to see your fleet-wide audit timeline.
Full Fleet Management
For policy management, global kill-switches, SIEM integrations, and more, see the complete Predicate Vault documentation.
Choosing Your Audit Strategy
| Environment | Recommended Approach |
|---|---|
| Local development | Local sidecar logging (default) |
| CI/CD pipelines | Local sidecar logging with test assertions |
| Staging / Pre-prod | Control-plane connected for integration testing |
| Production | Control-plane connected with SIEM streaming |
| Regulated industries | Control-plane with legal hold and retention attestation |