Keyboard shortcuts Press ← or → to navigate between chapters Press S or / to search in the book Press ? to show this help Press Esc to hide this help A Flexible Framework for Experiencing Cutting-edge LLM Inference/Fine-tune Optimizations KTransformers is a research project focused on efficient inference and fine-tuning of large language models through CPU-GPU heterogeneous computing. The project now exposes two user-facing capabilities from the kt-kernel source tree: Inference and SFT . Oct 15, 2026 : DeepSeek-V4-Flash LoRA fine-tuning supports native MXFP4 routed experts. quickstart . Oct 8, 2026 : We release the author preprint of KTransformers-FineTune (KT-FT) , presenting layout-aware and schedule-optimized CPU-GPU co-computation for MoE fine-tuning. Paper and citation · PDF . Sep 13, 2026 : Kimi K2.5 / K2.6 LoRA fine-tuning is supported with native RAWINT4 routed experts. Aug 26, 2026 : Added native support for GLM-5.3-flash , bringing 1M-token context and multimodal input to consumer GPUs. ( Tutorial ) Aug 25, 2026 : Uploaded a new easy-to-use KTransformers × LlamaFactory MoE Fine-Tuning Cookbook , covering hardware checks, installation, BF16/FP8/INT8 recipes, LoRA and full fine-tuning, resource planning, and troubleshooting. Aug 17, 2026 : LoRA fine-tuning now supports compatible AVX512 x86 CPUs, including AMD servers, without requiring AMX. ( v0.7.0 Release Notes ) Aug 16, 2026 : DeepSeek-V4-Flash on a single Ascend NPU with CPU expert offload. ( Tutorial ) Aug 5, 2026 : Introduced native block-FP8 LoRA fine-tuning ( PR #2141 ) July 23, 2026 : Added end-to-end BF16 full-parameter fine-tuning for MoE models, including complete checkpoint saving. ( PR #2094 ) June 21, 2026 : MiniMax-M3 Day0 Support! ( Tutorial ) June 17, 2026 : GLM-5.2 Day0 Support! ( Tutorial ) May 6, 2026 : KTransformers at GOSIM Paris 2026 — “Agentic AI on Edge” track. We’ll present KT’s inference performance on consumer hardware. May 02, 2026 : DeepSeek-V4-Flash Support! ( Tutorial ) Apr 30, 2026 : KTransformers v0.6.1 refreshes kt-kernel inference and SFT docs with separate Inference and SFT Quick Start entry points. Mar 26, 2026 : Support AVX2-only CPU backend for KT-Kernel inference. ( Tutorial ) Feb 13, 2026 : MiniMax-M2.5 Day0 Support! ( Tutorial ) Feb 12, 2026 : GLM-5 Day0 Support! ( Tutorial ) Jan 27, 2026 : Kimi-K2.5 Day0 Support! ( Tutorial ) ( SFT Tutorial ) Jan 22, 2026 : Support CPU-GPU Expert Scheduling , Native BF16 and FP8 per channel Precision and AutoDL unified fine-tuning and inference Dec 24, 2025 : Support Native MiniMax-M2.1 inference. ( Tutorial ) Dec 22, 2025 : Support RL-DPO fine-tuning with LLaMA-Factory. ( Tutorial ) Dec 5, 2025 : Support Native Kimi-K2-Thinking inference ( Tutorial ) Nov 6, 2025 : Support Kimi-K2-Thinking inference ( Tutorial ) and fine-tune ( Tutorial ) Nov 4, 2025 : KTransformers Fine-Tuning × LLaMA-Factory Integration. ( Tutorial ) Oct 27, 2025 : Support Ascend NPU. ( Tutorial ) Oct 10, 2025 : Integrating into SGLang. ( Roadmap , Blog ) Sept 11, 2025 : Support Qwen3-Next. ( Tutorial ) Sept 05, 2025 : Support Kimi-K2-0905. ( Tutorial ) July 26, 2025 : Support SmallThinker and GLM4-MoE. ( Tutorial ) July 11, 2025 : Support Kimi-K2. ( Tutorial ) June 30, 2025 : Support 3-layer (GPU-CPU-Disk) prefix cache reuse. May 14, 2025 : Support Intel Arc GPU ( Tutorial ). Apr 29, 2025 : Support AMX-Int8、 AMX-BF16 and Qwen3MoE ( Tutorial ) Apr 9, 2025 : Experimental support for LLaMA 4 models ( Tutorial ). Apr 2, 2025 : Support Multi-concurrency. ( Tutorial ). Mar 15, 2025 : Support ROCm on AMD GPU ( Tutorial ). Mar 5, 2025 : Support unsloth 1.58/2.51 bits weights and IQ1_S/FP8 hybrid weights. Support 139K Longer Context for DeepSeek-V3 and R1 in 24GB VRAM. Feb 25, 2025 : Support FP8 GPU kernel for DeepSeek-V3 and R1; Longer Context .

Feb 15, 2025 : Longer Context (from 4K to 8K for 24GB VRAM) & Slightly Faster Speed （+15%, up to 16 Tokens/s), update docs and online books . Feb 10, 2025 : Support Deepseek-R1 and V3 on single (24GB VRAM)/multi gpu and 382G DRAM, up to 3~28x speedup. For detailed show case and reproduction tutorial, see here . Aug 28, 2024 : Decrease DeepseekV2’s required VRAM from 21G to 11G. Aug 15, 2024 : Update detailed tutorial for injection and multi-GPU. Aug 14, 2024 : Support llamfile as linear backend. Aug 12, 2024 : Support multiple GPU; Support new model: mixtral 8*7B and 8*22B; Support q2k, q3k, q5k dequant on gpu. Aug 9, 2024 : Support windows native. 🚀 Inference - High-Performance kt-kernel Serving CPU-optimized kernel operations for heterogeneous LLM inference. AMX/AVX Acceleration : Intel AMX and AVX512/AVX2 optimized kernels for INT4/INT8 quantized inference MoE Optimization : Efficient Mixture-of-Experts inference with NUMA-aware memory management Quantization Support : CPU-side INT4/INT8 quantized weights, GPU-side GPTQ support Easy Integration : Clean Python API for SGLang and other frameworks CPU-GPU hybrid inference for large MoE models Integration with SGLang for production serving Heterogeneous expert placement (hot experts on GPU, cold experts on CPU) Performance Examples: 👉 Full Documentation → 🎓 SFT - Fine-Tuning with LlamaFactory Research paper: KTransformers-FineTune (KT-FT) · PDF . KTransformers × LlamaFactory integration for ultra-large MoE model fine-tuning. The new Cookbook provides an easy-to-use path from hardware checks and installation to BF16/FP8/INT8 configuration, LoRA/full training, resource planning, and troubleshooting. Kimi K2.5 / K2.6 LoRA fine-tuning is supported with native RAWINT4 routed experts. Follow the PyPI installation, training, resume and SGLang serving guide (中文) . End-to-end validation uses Kimi K2.5; K2.6 follows the same model-architecture path. Multi-Backend Support : CPU/GPU hybrid fine-tuning with INT8/INT4 quantization Ultra-Large MoE Support : Fine-tune models like DeepSeek-V3/R1 on limited GPU memory Faster than ZeRO-Offload : 6-12x training speedup in benchmarked MoE SFT workloads Lower CPU Memory : About half the CPU memory of the previous KT SFT path in the benchmarked setup LlamaFactory Integration : Seamless integration with the popular fine-tuning framework 👉 Easy-to-Use Fine-Tuning Cookbook → 👉 Quick Start → 👉 Full Documentation → If you use KTransformers in your research, please cite the relevant paper. Inference paper Fine-tuning paper KTransformers-FineTune (KT-FT) · Full BibTeX 👥 Contributors & Team Developed and maintained by: MADSys Lab @ Tsinghua University Community contributors We welcome contributions! Please feel free to submit issues and pull requests. 💬 Community & Support GitHub Issues : Report bugs or request features WeChat Group : See archive/WeChatGroup.png 📦 KT original Code The original integrated KTransformers framework has been archived to the archive/ directory for reference. The project now organizes the two capabilities above from the kt-kernel source tree for clearer documentation and maintenance. For the original documentation with full quick-start guides and examples, see: archive/README.md (English) archive/README_ZH.md (中文)