Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 14 hours
Course Outline
AI Sovereignty and Local LLM Deployment
- Challenges associated with cloud LLMs: data retention policies, input training usage, and foreign jurisdiction risks.
- Ollama architecture: examining the model server, registry, and OpenAI-compatible API components.
- Comparative analysis with vLLM, llama.cpp, and Text Generation Inference.
- Licensing considerations for Llama, Mistral, Qwen, and Gemma models.
Installation and Hardware Configuration
- Installing Ollama on Linux systems with CUDA and ROCm capabilities.
- CPU-only fallback strategies and AVX/AVX2 instruction set optimization.
- Deployment via Docker and management of persistent volume mappings.
- Configuring multi-GPU setups and establishing VRAM allocation strategies.
Model Management
- Retrieving models from the Ollama registry using commands like ollama pull llama3.
- Importing GGUF formats from HuggingFace and TheBloke repositories.
- Evaluating quantization levels such as Q4_K_M, Q5_K_M, and Q8_0 for optimal performance.
- Managing model switching and understanding concurrent loading limitations.
Custom Modelfiles
- Syntax guide for Modelfiles: utilizing FROM, PARAMETER, SYSTEM, and TEMPLATE directives.
- Adjusting temperature, top_p, and repeat_penalty parameters.
- Engineering system prompts to achieve role-specific model behavior.
- Publishing custom models to local registries.
API Integration
- Utilizing the OpenAI-compatible /v1/chat/completions endpoint.
- Handling streaming responses and enabling JSON mode.
- Integration with frameworks like LangChain, LlamaIndex, and bespoke applications.
- Implementing authentication and rate limiting via reverse proxies.
Performance Optimization
- Optimizing context window sizes and managing KV caches.
- Handling batch inference and parallel requests efficiently.
- Allocating CPU threads with NUMA topology awareness.
- Monitoring GPU utilization and memory pressure metrics.
Security and Compliance
- Isolating network environments for model serving endpoints.
- Implementing input filtering and output moderation pipelines.
- Maintaining audit logs for all prompts and completions.
- Verifying model provenance and hash integrity.
Requirements
- Proficiency in intermediate Linux administration and container management.
- A foundational understanding of machine learning concepts and transformer architectures.
- Working knowledge of REST APIs and JSON data structures.
Target Audience
- AI engineers and developers looking to transition away from cloud LLM APIs.
- Organizations with strict data sensitivity requirements that preclude the use of cloud models.
- Government and defense sectors necessitating air-gapped language model deployments.
