Model Serving & Fast Inference

1. Overview#

Moving Machine Learning models from Jupyter Notebook experimentation to production requires low-latency serving backends, request batching, memory optimization (KV Cache management), and containerized scaling.

---

2. LLM Serving Optimization: PagedAttention & vLLM#

Traditional autoregressive generation wastes up to 60-80% of GPU memory in fragmented Key-Value (KV) caches. PagedAttention (introduced by vLLM) treats KV cache memory like virtual memory pages in an operating system, enabling continuous batching and 10x-20x throughput improvements.

code
Ask AICopy
Request Stream ──► [ Continuous Batching Engine ]
 │
 ▼
 [ PagedAttention KV-Cache Manager ]
 │
 ▼
 [ GPU Tensor Parallel Kernel (vLLM) ] ──► Token Stream Output

---

3. High-Throughput REST API with FastAPI#

For classical ML and lightweight embeddings models, FastAPI provides asynchronous non-blocking inference endpoints:

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
from fastapi import FastAPI, HTTPException, status
from pydantic import BaseModel, Field
import numpy as np
import joblib

app = FastAPI(title="Real-Time Fraud Detection Inference API", version="1.0.0")

# Request / Response Schemas
class FraudInferenceRequest(BaseModel):
 transaction_amount: float = Field(..., gt=0, description="Amount in USD")
 user_risk_score: float = Field(..., ge=0.0, le=1.0)
 device_trust_score: float = Field(..., ge=0.0, le=1.0)
 hour_of_day: int = Field(..., ge=0, le=23)

class FraudInferenceResponse(BaseModel):
 is_fraud: bool
 fraud_probability: float
 decision_latency_ms: float

# Load serialized model pipeline into memory once at startup
# model_pipeline = joblib.load("fraud_xgboost_pipeline.pkl")

@app.post("/predict", response_model=FraudInferenceResponse, status_code=status.HTTP_200_OK)
async def predict_fraud(payload: FraudInferenceRequest):
 try:
 # Preprocess features into 2D array
 features = np.array([[
 payload.transaction_amount,
 payload.user_risk_score,
 payload.device_trust_score,
 payload.hour_of_day
 ]])
 
 # Inference (Mocked calculation for demonstration)
 prob = float(1 / (1 + np.exp(-(payload.transaction_amount * 0.01 + payload.user_risk_score * 3.0 - 5.0))))
 
 return FraudInferenceResponse(
 is_fraud=prob >= 0.75,
 fraud_probability=round(prob, 4),
 decision_latency_ms=1.45
 )
 except Exception as e:
 raise HTTPException(status_code=500, detail=str(e))

---

4. Production Checklist#

• Containerize with Docker multi-stage builds and use GPU base images (nvidia/cuda:12.4.0-runtime-ubuntu22.04).
• Serve LLMs via OpenAI-compatible endpoints with vLLM or Ollama for seamless drop-in integrations.
• Implement health check (/healthz) and Prometheus metrics (/metrics) endpoints for Kubernetes liveness/readiness probes.