Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 14 hours
Course Outline
Deployment Options
- Trade-offs between hosted APIs and self-hosting
- On-premises, edge, and air-gapped deployment patterns
- Single model vs. model portfolio strategies
- Cost-per-decision analysis
Hardware Sizing and Capacity
- Memory footprint relative to parameter count
- CPU-only vs. GPU acceleration options
- Planning for concurrency and throughput
- Allowing headroom for task length growth
Serving Stacks
- vLLM and SGLang options
- llama.cpp and GGUF execution
- Ollama for local development
- Selecting tools based on decision workload characteristics
Quantization Without Calibration Loss
- Precision formats and memory efficiency
- Measuring quality and calibration degradation
- Per-tensor and importance-aware quantization variants
- Acceptance testing post-quantization
Latency Engineering
- Median vs. tail latency considerations
- Latency impact of context length
- Warm starts and model caching
- Batching multiple decision queries
Edge and Constrained Environments
- Running small decision models on CPUs
- Deployment on Apple Silicon and embedded systems
- Offline operation and update strategies
- Distributing weights in air-gapped settings
Containerization and Scaling
- Designing images for model serving
- Managing volumes and caches
- Horizontal scaling and traffic routing
- Resource limits and GPU sharing
Observability
- Tracking request, latency, and error metrics
- Monitoring confidence distributions
- Logging decision outcomes and escalations
- Alerting on calibration drift
Versioning and Rollout
- Versioning models and adapters
- Shadow and canary deployment strategies
- Pinning model files for audit compliance
- Rollback procedures
Security and Multi-Tenancy
- Authentication and rate limiting
- Isolating prompts and logs per tenant
- Network isolation and egress control
- Auditing decision logs
Requirements
- Linux administration and containerization experience.
- Proficiency with model inference servers and GPU management.
- Fundamental understanding of REST APIs, load balancing, and monitoring.
Audience
- MLOps and platform engineers managing inference infrastructure
- SRE teams supporting AI services
- AI engineers deploying decision models on premises
- Architects designing air-gapped or regulated AI deployments
