Advanced

24 min read

#Deployment#ONNX#TorchScript#Quantization#TensorRT#Serving#MLOps

28. Deep Learning Deployment, ONNX & Inference Optimization

Deploying models to production: serialization checkpoints, TorchScript JIT tracing, ONNX graph export, INT8/FP16 quantization, TensorRT acceleration, and low-latency serving.

Deep Learning Deployment: Complete Notes (Beginner to Advanced)

Introduction#

Deep Learning Deployment is the process of taking a trained neural-network model and making it available for real-world use.

Training happens in a development environment:

text
Data
 ↓
Training
 ↓
Validation
 ↓
Final Model

Deployment moves the model into a system where it can receive new data and produce predictions:

text
New Data
   ↓
Deployed Model
   ↓
Prediction
   ↓
Application / User / System

A complete deployment lifecycle is:

text
Train Model
    ↓
Serialize Model
    ↓
Optimize Model
    ↓
Deploy / Serve Model
    ↓
Inference
    ↓
Monitor
    ↓
Detect Drift
    ↓
Retrain
    ↓
Redeploy

1. Model Serialization

1.1 What is Model Serialization?#

Model serialization is the process of saving a trained model into a format that can later be loaded and used.

Without serialization:

text
Train model
   ↓
Program stops
   ↓
Model exists only in memory

With serialization:

text
Train model
   ↓
Save model
   ↓
Model artifact
   ↓
Load later
   ↓
Inference

1.2 Why Serialize Models?#

A serialized model can be:

Stored
Transferred
Loaded by an inference service
Versioned
Deployed to production
Used for batch inference
Used for real-time inference

1.3 PyTorch Serialization#

A common PyTorch practice is to save the model's state_dict.


🐍 PythonInteractive WebAssembly
torch.save(
    model.state_dict(),
    "model.pth"
)

The architecture is recreated and the saved parameters are loaded:


🐍 PythonInteractive WebAssembly
model = MyModel()

state_dict = torch.load(
    "model.pth",
    map_location="cpu"
)

model.load_state_dict(state_dict)

model.eval()

1.4 Checkpoint vs Deployment Artifact#

A training checkpoint may contain:

text
Model weights
Optimizer state
Epoch
Training metadata

A deployment artifact generally needs what is required for inference, such as:

text
Model
Model configuration
Preprocessing information
Postprocessing information
Version metadata

The exact artifact depends on the deployment framework.

1.5 Common Model Formats#

Different ecosystems use different formats, including:

text
PyTorch state_dict
TorchScript
ONNX
TensorFlow SavedModel
Keras model formats

The choice depends on the framework, target runtime, hardware, and deployment requirements.

2. Model Serving

2.1 What is Model Serving?#

Model serving means making a trained model available so that another system can send input data and receive predictions.

Conceptually:

text
Client
  │
  │ Request
  ▼
Model Serving System
  │
  ▼
Model
  │
  │ Prediction
  ▼
Client

A serving system typically handles:

Loading the model
Receiving requests
Preprocessing inputs
Running inference
Postprocessing outputs
Returning predictions
Logging
Monitoring
Scaling

2.2 Model Server#

A model server is a service responsible for exposing a model for inference.

A simple architecture is:

text
                 ┌─────────────────┐
Client ────────► │ API / Model     │
                 │ Server          │
                 └────────┬────────┘
                          │
                          ▼
                     Preprocess
                          │
                          ▼
                       Model
                          │
                          ▼
                     Postprocess
                          │
                          ▼
                       Response

3. Batch Inference

3.1 What is Batch Inference?#

Batch inference means running predictions on a collection of inputs together, usually on a schedule or as a bulk job.

Example:

text
1,000,000 records
       ↓
Batch inference job
       ↓
Predictions
       ↓
Database / Data Warehouse

It does not require one request per individual prediction.

3.2 Example#

Suppose an e-commerce company wants to generate recommendations for all customers every night.

text
Customer Data
     ↓
Nightly Batch Job
     ↓
Model
     ↓
Recommendations
     ↓
Database

3.3 Advantages#

Efficient for large datasets
Can process data in batches
Easier to schedule
Can use high-throughput hardware
Individual prediction latency is usually less important

3.4 Disadvantages#

Predictions are not necessarily immediate
Requires a batch-processing workflow
Predictions may become stale between runs

3.5 Batch Inference Example#


🐍 PythonInteractive WebAssembly
model.eval()

predictions = []

with torch.no_grad():

    for X_batch in data_loader:

        output = model(X_batch)

        predictions.append(output)

The model processes multiple samples per batch.

4. Real-Time Inference

4.1 What is Real-Time Inference?#

Real-time inference means generating a prediction when a request arrives, usually under a latency requirement.

Example:

text
User Request
     ↓
API
     ↓
Model
     ↓
Prediction
     ↓
Response

Examples include:

Fraud detection
Image classification in an application
Recommendation requests
Speech processing
Chat applications

4.2 Real-Time Inference Requirements#

A real-time system often cares about:

text
Latency
Throughput
Availability
Scalability
Resource usage

4.3 Batch vs Real-Time Inference#

Feature	Batch Inference	Real-Time Inference
Trigger	Scheduled / bulk	Request
Latency requirement	Usually relaxed	Usually strict
Data volume	Often large	Often smaller per request
Typical use	Nightly predictions	User-facing prediction
Infrastructure	Batch job	Online service
Main concern	Throughput	Latency + availability

5. REST API Inference

5.1 What is REST API Inference?#

A REST API can expose a trained model through HTTP.

A client sends input:

›POST /predict

The server:

text
Receive JSON
    ↓
Validate Input
    ↓
Preprocess
    ↓
Model Inference
    ↓
Postprocess
    ↓
Return JSON

5.2 Example Request#

json
{
    "features": [5.2, 3.1, 1.4, 0.2]
}

Example response:

json
{
    "prediction": 0,
    "confidence": 0.97
}

The exact request and response schema depends on the application.

5.3 Simple FastAPI Example#


🐍 PythonInteractive WebAssembly
from fastapi import FastAPI
from pydantic import BaseModel
import torch

app = FastAPI()

model = MyModel()
model.load_state_dict(
    torch.load(
        "model.pth",
        map_location="cpu"
    )
)
model.eval()


class PredictionRequest(BaseModel):
    features: list[float]


@app.post("/predict")
def predict(request: PredictionRequest):

    x = torch.tensor(
        [request.features],
        dtype=torch.float32
    )

    with torch.no_grad():
        output = model(x)

    prediction = output.argmax(dim=1).item()

    return {
        "prediction": prediction
    }

The important deployment principle is:

text
Load model once
      ↓
Keep model in memory
      ↓
Handle many requests

You generally do not want to reload the model from disk for every request.

6. GPU Inference

6.1 What is GPU Inference?#

GPU inference means running model prediction on a GPU rather than a CPU.

This can be useful for computationally heavy models.

text
Input
 ↓
GPU
 ↓
Neural Network
 ↓
Prediction

6.2 PyTorch GPU Inference#


🐍 PythonInteractive WebAssembly
device = torch.device(
    "cuda" if torch.cuda.is_available()
    else "cpu"
)

model = model.to(device)
model.eval()

x = x.to(device)

with torch.no_grad():
    output = model(x)

6.3 GPU Inference Considerations#

GPU inference is not automatically better for every workload.

Consider:

text
Model size
Batch size
Request rate
Latency requirement
CPU/GPU transfer overhead
GPU memory
Cost

For very small requests, CPU inference can sometimes be preferable because transferring data to a GPU has overhead.

7. Model Optimization

7.1 What is Model Optimization?#

Model optimization is the process of making a model more efficient for inference while maintaining acceptable prediction quality.

Common goals:

text
Lower latency
Lower memory usage
Higher throughput
Lower infrastructure cost
Smaller model size

7.2 Common Optimization Techniques#

text
Quantization
Pruning
Knowledge Distillation
Compilation
Graph Optimization
Efficient Architectures

7.3 Quantization#

Quantization represents model values using lower numerical precision.

For example:

›FP32 → FP16
FP32 → INT8

This can reduce:

Model size
Memory usage
Computation cost

It can also improve inference speed on hardware that supports the relevant lower-precision operations.

7.4 Post-Training Quantization#

The model is trained normally and quantized afterward.

text
Train FP32 Model
      ↓
Quantize
      ↓
Deploy

7.5 Quantization-Aware Training#

The model is trained while simulating quantization effects.

text
Training
   ↓
Simulate Quantization
   ↓
Learn to tolerate quantization
   ↓
Quantized Model

This can help preserve accuracy when aggressive quantization would otherwise cause too much degradation.

7.6 Pruning#

Pruning removes or reduces the importance of selected model parameters or structures.

Conceptually:

text
Dense Model
    ↓
Identify less important weights
    ↓
Remove / zero them
    ↓
Smaller or sparse model

Structured pruning can remove entire channels, filters, or other structures, which can be more useful for actual hardware acceleration than arbitrary unstructured sparsity.

7.7 Knowledge Distillation#

A large teacher model teaches a smaller student model.

text
Large Teacher
      ↓
Soft / informative outputs
      ↓
Small Student
      ↓
Efficient Deployment

The student attempts to reproduce useful behavior of the teacher while using fewer resources.

7.8 Model Compilation#

Model compilation can transform the model into a representation optimized for a target runtime or hardware.

Conceptually:

text
Model
 ↓
Compiler / Runtime Optimization
 ↓
Optimized Execution
 ↓
Inference

Examples include framework-specific compilation and optimized inference runtimes.

8. Model Monitoring

8.1 What is Model Monitoring?#

Model monitoring means continuously observing a deployed model and its surrounding system to determine whether it continues to behave correctly.

Monitoring can include:

text
System metrics
Data metrics
Model metrics
Business metrics

8.2 System Metrics#

Examples:

text
Latency
Throughput
CPU usage
GPU usage
Memory usage
Error rate
Request count

Example:

Mathematical Formulation
Average latency = 120 ms
p95 latency = 240 ms
Error rate = 0.3%

Percentile latency such as p95 is often more informative than only the average because it shows behavior experienced by slower requests.

8.3 Data Metrics#

Monitor incoming data for changes such as:

text
Missing values
Unexpected ranges
New categories
Feature distributions
Input volume
Schema changes

8.4 Model Metrics#

When ground-truth labels eventually become available, monitor:

text
Accuracy
Precision
Recall
F1
MAE
RMSE
AUC

The appropriate metric depends on the task.

8.5 Business Metrics#

A model can have acceptable technical metrics but still fail to create business value.

Examples:

text
Conversion rate
Revenue
Fraud loss
Customer retention
Click-through rate
Resolution rate

9. Model Drift

9.1 What is Model Drift?#

Model drift refers broadly to degradation or change in model behavior as the real-world environment changes.

The data distribution, relationship between inputs and targets, or operational conditions may change after deployment.

Conceptually:

text
Training Environment
        ↓
       Model
        ↓
Production
        ↓
World Changes
        ↓
Model Performance Changes

9.2 Data Drift#

Data drift occurs when the distribution of input data changes.

Example:

Training:

Mathematical Formulation
Average customer age = 30

Production later:

Mathematical Formulation
Average customer age = 45

The input distribution has changed.

9.3 Concept Drift#

Concept drift occurs when the relationship between input variables and the target changes.

For example:

text
Same customer behavior
      ↓
Different relationship with fraud
      ↓
Old model becomes less accurate

The mapping:

›P(Y | X)

has changed.

9.4 Label / Target Distribution Changes#

The distribution of target values can also change.

For example:

Mathematical Formulation
Historical fraud rate = 2%
Current fraud rate = 8%

A change in target prevalence can affect model behavior and evaluation.

9.5 Drift Detection#

Possible signals include:

text
Feature distribution comparison
Prediction distribution comparison
Performance monitoring
Statistical tests
Population Stability Index (PSI)

The exact method should match the type of drift and the available data.

10. Model Retraining

10.1 What is Model Retraining?#

Model retraining is the process of training or fine-tuning a deployed model again using newer data.

A common lifecycle is:

text
Production Model
      ↓
Monitor
      ↓
Detect Performance / Data Change
      ↓
Collect New Data
      ↓
Prepare Data
      ↓
Retrain
      ↓
Validate
      ↓
Test
      ↓
Deploy New Version

10.2 Why Retrain?#

Retraining may be required when:

Model performance decreases
Data distribution changes
New patterns appear
User behavior changes
Business rules change
New labeled data becomes available

10.3 Scheduled Retraining#

A model can be retrained periodically:

text
Daily
Weekly
Monthly
Quarterly

The appropriate frequency depends on how quickly the underlying environment changes.

10.4 Trigger-Based Retraining#

Retraining can also be triggered by conditions:

text
Validation metric drops
        ↓
Retraining triggered

or:

text
Drift detected
        ↓
Investigate
        ↓
Retrain if necessary

Drift detection should not automatically imply that retraining is always the correct action; the cause and impact should be investigated.

11. End-to-End Deployment Lifecycle

A production deep-learning lifecycle can look like:

text
                    Training Data
                         │
                         ▼
                    Train Model
                         │
                         ▼
                  Validate / Test
                         │
                         ▼
                Serialize Model
                         │
                         ▼
                 Model Optimization
                         │
                         ▼
                  Deploy Model
                         │
              ┌──────────┴──────────┐
              │                     │
              ▼                     ▼
        Batch Inference       Real-Time API
              │                     │
              └──────────┬──────────┘
                         ▼
                    Predictions
                         │
                         ▼
                    Monitoring
                         │
              ┌──────────┴──────────┐
              │                     │
          No Problem            Drift / Decay
              │                     │
              │                     ▼
              │               New Data
              │                     │
              │                     ▼
              │                 Retrain
              │                     │
              └─────────────────────┤
                                    ▼
                              New Model
                                    │
                                    ▼
                                 Deploy

12. Batch vs Real-Time Deployment

Feature	Batch Inference	Real-Time Inference
Input	Large collection	Individual request / small batch
Trigger	Schedule / job	API request / event
Latency	Usually less important	Usually important
Throughput	Main concern	Important
Example	Daily customer predictions	Fraud check during payment
Infrastructure	Batch pipeline	Online serving system

13. Model Optimization vs Model Monitoring

These are different stages.

Model Optimization#

Asks:

›How can we make inference more efficient?

Examples:

text
Quantization
Pruning
Distillation
Compilation

Model Monitoring#

Asks:

›Is the deployed model still working correctly?

Examples:

text
Latency
Errors
Data distributions
Prediction distributions
Accuracy
Business metrics

14. Data Drift vs Concept Drift

Type	What Changes?
Data Drift	Input distribution `P(X)` changes
Concept Drift	Relationship `P(Y
Target Distribution Shift	Target distribution `P(Y)` changes

Example:

text
Data Drift:
Customers become older.

Concept Drift:
The same behavior now has a different relationship with churn.

Target Shift:
The proportion of churned customers increases.

These changes can occur independently or together.

15. A Practical REST Deployment Architecture

text
                    Client
                      │
                      ▼
                  HTTP Request
                      │
                      ▼
                ┌─────────────┐
                │ API Server  │
                └──────┬──────┘
                       │
                       ▼
                  Input Validation
                       │
                       ▼
                    Preprocess
                       │
                       ▼
                 ┌─────────────┐
                 │ Model       │
                 │ In Memory   │
                 └──────┬──────┘
                       │
                       ▼
                   Prediction
                       │
                       ▼
                  Postprocess
                       │
                       ▼
                  JSON Response
                       │
                       ▼
                     Client

Monitoring
    │
    ├── Latency
    ├── Errors
    ├── Input Drift
    ├── Prediction Drift
    └── Model Performance

16. Production Model Versioning

Models should generally be treated as versioned artifacts.

Example:

text
model-v1
model-v2
model-v3

A model version can be associated with:

text
Model weights
Training data version
Code version
Configuration
Metrics
Dependencies
Preprocessing version

This makes it easier to reproduce and roll back deployments.

17. Important Deployment Principles

Principle 1: Save More Than Just Weights#

A production model often depends on:

text
Weights
+
Architecture
+
Preprocessing
+
Postprocessing
+
Configuration

If preprocessing changes between training and serving, predictions can become incorrect even if the model weights are unchanged.

Principle 2: Load the Model Once#

For a long-running inference server:

text
Server Starts
     ↓
Load Model
     ↓
Keep Model in Memory
     ↓
Handle Requests

Avoid:

text
Request
 ↓
Load Model
 ↓
Predict
 ↓
Unload

for every request unless the architecture specifically requires it.

Principle 3: Monitor the Whole System#

Do not monitor only model accuracy.

Monitor:

text
Infrastructure
+
Data
+
Model
+
Business Outcome

Principle 4: Keep a Rollback Option#

If:

›Model v2

causes production problems, a deployment system should ideally allow:

›v2 → rollback → v1

18. Summary Table

Concept	Main Idea
Model Serialization	Save a trained model for later use
Model Serving	Make a model available for inference
Batch Inference	Generate predictions for data in bulk
Real-Time Inference	Generate predictions when requests arrive
REST API Inference	Expose model predictions through HTTP endpoints
GPU Inference	Run inference using GPU hardware
Model Optimization	Improve inference efficiency
Model Monitoring	Observe production model/system behavior
Model Drift	Detect changes that can degrade model usefulness
Model Retraining	Train/update the model using newer data

19. Quick Recap

text
TRAIN
  ↓
SAVE
  ↓
OPTIMIZE
  ↓
DEPLOY
  ↓
SERVE
  ↓
INFER
  ↓
MONITOR
  ↓
DRIFT DETECTED?
  │
  ├── NO ──► Continue Serving
  │
  └── YES
       ↓
   Investigate
       ↓
   New Data
       ↓
    Retrain
       ↓
    Validate
       ↓
    Deploy New Version

The core mental model is:

text
Model Serialization
→ Save the model

Model Serving
→ Make the model available

Batch Inference
→ Predict in bulk

Real-Time Inference
→ Predict on request

REST API
→ Provide an HTTP interface

GPU Inference
→ Accelerate suitable workloads

Model Optimization
→ Make inference more efficient

Model Monitoring
→ Watch production behavior

Model Drift
→ Detect changes in the environment/model behavior

Model Retraining
→ Update the model when new data or changed conditions require it

Knowledge Checkpoint

28. Deep Learning Deployment Checkpoint

Q1.What is the primary advantage of exporting a PyTorch model to Open Neural Network Exchange (ONNX) format?

AIt decouples the model graph from Python runtime, allowing execution on high-performance C++ inference engines (ONNX Runtime, TensorRT) across diverse hardware.

BIt automatically improves accuracy by 10%.

CIt eliminates the need for model weights.

DIt converts models into static HTML files.

Q2.What is Post-Training Quantization (PTQ) to INT8, and what benefits does it deliver?

AConverting 32-bit floating point weights and activations to 8-bit integers using scale factors, reducing model size by 4x and delivering 2-4x higher inference throughput.

BRemoving 75% of model layers.

CRetraining the model from scratch on integer datasets.

DRounding loss values during backprop.

Q3.What is Dynamic Batching in production model serving servers (like Triton Inference Server or vLLM)?

AGrouping asynchronous incoming individual user requests over a micro-window into a single batched GPU execution to maximize compute utilization without violating latency SLAs.

BChanging the model architecture on every request.

CRandomly dropping incoming requests during high traffic.

DDistributing requests to random CPU cores.

Track Your Learning

Finished studying this notebook?

Mark this guide as completed to update your course progress roadmap.

Previous Lesson

27. Model Training, Validation & Evaluation Pipelines

Next Lesson

29. Large Language Models (LLM) Pre-training & Inference