Learn: Inference and GPUs

1. Mastering LLM techniques: inference optimizationNVIDIA · Where the time and memory go when a model answers.Mark done
2. Optimizing inferenceHugging Face · Caching, compilation and attention tricks, in practice.Mark done
3. QuantizationHugging Face · Smaller numbers, smaller bills: the methods compared.Mark done
4. vLLM quickstartvLLM · Serving an open model yourself, in minutes.Mark done

• Inference
• GPU
• CUDA
• Quantization
• Latency
• Mixture of experts (MoE)
• HBM

1. Architecting for Hardware Diversity: Meta's End-to-End Co-Design for High-Performance AI Training on AMD GPUs
2. Efficient KV Cache Migration for Geo-Distributed LLM Inference in Collaborative Edge Computing
3. Exact Makespan Bounds and the GPipe Homogeneity Bias for Heterogeneous LLM Inference Pipelines
4. LOAD: Enabling Low-Bit Group Quantization for Vision Transformers via Adaptive Data Formats
5. When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill

More in Systems and databases →