PayOps

The problem

Payment incidents leave evidence across infrastructure, deployments, and business telemetry. Investigation needs to connect those signals without permitting unauthorized changes.

System architecture

An agentic investigation correlates Kubernetes state, logs, traces, and payment signals. Structured traces, incident memory, deterministic approval gates, and durable Pub/Sub processing keep the investigation auditable.

Evaluation

The harness contains 24 reproducible failure scenarios, including OOMKills, bad rollouts, scheduler saturation, outages, misleading telemetry, and payment-slice degradations.

Evaluation boundary

Root-cause recall is measured in predicate-assisted development replay. Rejection of unauthorized remediation is measured across 120 component-evaluation attempts.

What I built

• Built an agentic incident-response system correlating Kubernetes health, logs, traces, deployment changes, and payment telemetry across 24 reproducible failure scenarios, achieving 91.7% root-cause Recall@1 and Recall@3 in predicate-assisted development replay.
• Developed a failure-oriented evaluation harness covering OOMKills, bad rollouts, scheduler saturation, dependency outages, misleading telemetry, and payment-slice degradations, rejecting 100% of 120 unauthorized remediation attempts in component evaluations.
• Added structured investigation traces, incident memory, deterministic approval gates, GraphQL, Slack notifications, and durable Pub/Sub processing, achieving 8.41-second median diagnostic replay latency, reducing median investigation time from 11.8 min to 2.9 min, and maintaining $0.07 average LLM cost per incident.

Explore the implementation

Source code, setup instructions, and project documentation are available in the repository.