Skip to main content

Auditing and Forensics CLI Reference

This guide covers running chain verification, pinpointing corrupted records, and replaying projections using the m audit CLI commands implemented in libs/framework-m-standard/src/framework_m_standard/cli/audit.py.

1. Verifying Hash Chains (verify-chain)​

Scans records in chronological order and checks:

  1. prev_hash[n] == hash[n-1].
  2. hash[n] == sha256(prev_hash[n] : JCS(payload[n])).

Command Syntax​

# Verify a single DocType across all streams
m audit verify-chain --doctype <DocTypeName>

# Verify a specific stream partition
m audit verify-chain --doctype <DocTypeName> --stream <stream_id>

# Verify all registered DocTypes with Meta.tamper_proof = True
m audit verify-chain --all

Example​

m audit verify-chain --doctype AuditEntry

Output if intact:

✓ AuditEntry (stream=all): 150 records verified. Chain INTACT.

Output if broken:

✗ AuditEntry (stream=all): TAMPER DETECTED at record 'AUDI-0089'! Checked 89 records.

2. Pinpointing Corruption (forensic-trace)​

When verify-chain reports that an integrity break occurred, run forensic-trace to identify the first record where the cryptographic continuity broke.

Command Syntax​

m audit forensic-trace --doctype <DocTypeName>

Example Output​

m audit forensic-trace --doctype AuditEntry
🚨 FORENSIC ANOMALY DETECTED in 'AuditEntry':
  First tampered / broken link: AUDI-0089
  Records scanned prior to break: 88
Evidence indicates direct database alteration without valid private key seal.

If the table is entirely authentic:

✓ All 150 records in 'AuditEntry' are authentic. No break points found.

3. Rebuilding Read Projections (rehydrate)​

Replays events from the immutable stream to rebuild read projections via any provider implementing ProjectionRehydrationProtocol.

Command Syntax​

m audit rehydrate --doctype <DocTypeName> [--stream <stream_id>]

Provider Implementation Contract​

In Python application code:

from framework_m_core.interfaces.rehydration import ProjectionRehydrationProtocol


class InvoiceRehydrationProvider(ProjectionRehydrationProtocol):
    async def rehydrate_doctype(
        self, doctype_name: str, stream_id: str | None = None
    ) -> int:
        # Re-derive read balances or search indices from the immutable rows
        count = await sync_projection(doctype_name, stream_id)
        return count

When invoked:

m audit rehydrate --doctype Invoice --stream pos-01

Output:

✓ Rehydrated 42 records for Invoice successfully.

4. Disaster Recovery & Compromise Incident Response​

When forensic-trace or health metrics detect an integrity anomaly, follow this recovery playbook.

Scenario A: Single Edge Node Compromised​

If a single edge node (e.g. node-munich-04) is compromised or its local key is exfiltrated:

  1. Isolate & Revoke Key in O(1): Immediately revoke the node's public key fingerprint in the central registry:

    m crypto revoke-key --fingerprint <HEX_FINGERPRINT> --reason "Suspected physical compromise of store node 04"

    IngressVerificationService instantly drops any subsequent envelopes signed with this key without consuming cryptographic CPU cycles.

  2. Quarantine Ingested Envelopes: Any payloads delivered by the compromised node during the incident window are routed to _quarantine_dlq. Inspect quarantined messages:

    SELECT * FROM _quarantine_dlq WHERE node_id = 'node-munich-04';

  3. Re-key and Re-provision Node:

    • Re-image or wipe the edge node.
    • Generate a new node keypair: 
      m crypto generate-node-key
    • Register the new public key on the central cluster with an active status.

  4. Replay Authentic Edge Transactions: Replay verified operator transactions signed via client WebAuthn credentials that occurred before the break point.

Scenario B: Central Cluster / All Nodes Compromised ("Catastrophic Recovery")​

If a rogue root administrator compromised the central database directly and corrupted historical rows:

  1. Locate the First Point of Tampering: Run forensic-trace across all tamper-proof tables to find the exact boundary of integrity:

    m audit forensic-trace --doctype GeneralLedgerEntry

    Note the first corrupted record identifier (T_break) and scan count N.

  2. Cross-Verify with S3 WORM Merkle Roots: Retrieve the latest untampered epoch root published by EpochAnchorService from AWS S3 Object Lock (Compliance Mode):

    aws s3 cp s3://immutable-ledger-anchors/epochs/latest_epoch.json .

    Compare the Merkle root against local records to verify the last indisputably valid state.

  3. Restore Database via Point-in-Time Recovery (PITR): Restore the database instance up to timestamp T_break - 1 (the last valid cryptographic state).

  4. Re-ingest Sealed Outbox Envelopes: Because edge nodes maintain append-only outbox_envelopes independently, trigger edge outbox resynchronization from timestamp T_break. Each edge node will resubmit its gapless, dual-signed envelopes through IngressVerificationService.

  5. Rehydrate Read Projections: Once the golden hash-chained records are re-ingested:

    m audit rehydrate --doctype GeneralLedgerEntry