FinServe

The problem

Inference infrastructure must balance request latency, throughput, GPU utilization, and repeatable model-release decisions.

System architecture

GPU-aware routing, continuous batching, and prefix caching serve inference requests. An 18-task Airflow pipeline coordinates model evaluation, deployment, and rollback with MLflow evidence tracking.

Evaluation

The serving comparison covers 6,000+ inference requests. TTFT and p95 latency figures come from a separate quantized 3B release benchmark, with a 32-case correctness regression suite.

Release validation

The release benchmark completed 512/512 requests, followed by 60 post-deployment probes and 1→2→1 GPU engine scaling validation.

What I built

• Built a distributed text and multimodal inference platform with GPU-aware routing, continuous batching, prefix caching, and speculative-decoding evaluation, increasing sustained throughput from 13.0 to 34.6 requests/s, token throughput by 2.64×, and average GPU utilization from 29% to 58%.
• Benchmarked 6,000+ inference requests across baseline and optimized serving configurations; in a separate quantized 3B release benchmark, reduced median client TTFT from 247 ms to 116 ms and p95 latency from 2.24 s to 0.86 s, achieving 100% correctness on a 32-case release regression suite.
• Automated model evaluation, deployment, and rollback orchestration with CI performance gates, an 18-task Airflow pipeline, and MLflow evidence tracking, completing 512/512 release benchmark requests and validating 60 post-deployment probes and 1→2→1 GPU engine scaling.

Explore the implementation

Source code, setup instructions, and project documentation are available in the repository.