Keyboard shortcuts Press ← or → to navigate between chapters Press S or / to search in the book Press ? to show this help Press Esc to hide this help A Flexible Framework for Experiencing Cutting-edge LLM Inference/Fine-tune Optimizations KTransformers is a research project focused on efficient inference and fine-tuning of large language models through CPU-GPU heterogeneous computing. The project now exposes two user-facing capabilities from the kt-kernel source tree: Inference and SFT . Oct 15, 2026 : DeepSeek-V4-Flash LoRA fine-tuning supports native MXFP4 routed experts. quickstart . Oct 8, 2026 : We release the author preprint of KTransformers-FineTune (KT-FT) , presenting layout-aware and schedule-optimized CPU-GPU co-computation for MoE fine-tuning. Paper and citation · PDF . Sep 13, 2026 : Kimi K2.5 / K2.6 LoRA fine-tuning is supported with native RAWINT4 routed experts. Aug 26, 2026 : Added native support for GLM-5.3-flash , bringing 1M-token context and multimodal input to consumer GPUs. ( Tutorial ) Aug 25, 2026 : Uploaded a new easy-to-use KTransformers × LlamaFactory MoE Fine-Tuning Cookbook , covering hardware checks, installation, BF16/FP8/INT8 recipes, LoRA and full fine-tuning, resource planning, and troubleshooting. Aug 17, 2026 : LoRA fine-tuning now supports compatible AVX512 x86 CPUs, including AMD servers, without requiring AMX. ( v0.7.0 Release Notes ) Aug 16, 2026 : DeepSeek-V4-Flash on a single Ascend NPU with CPU expert offload. ( Tutorial ) Aug 5, 2026 : Introduced native block-FP8 LoRA fine-tuning ( PR #2141 ) July 23, 2026 : Added end-to-end BF16 full-parameter fine-tuning for MoE models, including complete checkpoint saving. ( PR #2094 ) June 21, 2026 : MiniMax-M3 Day0 Support! ( Tutorial ) June 17, 2026 : GLM-5.2 Day0 Support! ( Tutorial ) May 6, 2026 : KTransformers at GOSIM Paris 2026 — “Agentic AI on Edge” track. We’ll present KT’s inference performance on consumer hardware. May 02, 2026 : DeepSeek-V4-Flash Support! ( Tutorial ) Apr 30, 2026 : KTransformers v0.6.1 refreshes kt-kernel inference and SFT docs with separate Inference and SFT Quick Start entry points. Mar 26, 2026 : Support AVX2-only CPU backend for KT-Kernel inference. ( Tutorial ) Feb 13, 2026 : MiniMax-M2.5 Day0 Support! ( Tutorial ) Feb 12, 2026 : GLM-5 Day0 Support! ( Tutorial ) Jan 27, 2026 : Kimi-K2.5 Day0 Support! ( Tutorial ) ( SFT Tutorial ) Jan 22, 2026 : Support CPU-GPU Expert Scheduling , Native BF16 and FP8 per channel Precision and AutoDL unified fine-tuning and inference Dec 24, 2025 : Support Native MiniMax-M2.1 inference. ( Tutorial ) Dec 22, 2025 : Support RL-DPO fine-tuning with LLaMA-Factory. ( Tutorial ) Dec 5, 2025 : Support Native Kimi-K2-Thinking inference ( Tutorial ) Nov 6, 2025 : Support Kimi-K2-Thinking inference ( Tutorial ) and fine-tune ( Tutorial ) Nov 4, 2025 : KTransformers Fine-Tuning × LLaMA-Factory Integration. ( Tutorial ) Oct 27, 2025 : Support Ascend NPU. ( Tutorial ) Oct 10, 2025 : Integrating into SGLang. ( Roadmap , Blog ) Sept 11, 2025 : Support Qwen3-Next. ( Tutorial ) Sept 05, 2025 : Support Kimi-K2-0905. ( Tutorial ) July 26, 2025 : Support SmallThinker and GLM4-MoE. ( Tutorial ) July 11, 2025 : Support Kimi-K2. ( Tutorial ) June 30, 2025 : Support 3-layer (GPU-CPU-Disk) prefix cache reuse. May 14, 2025 : Support Intel Arc GPU ( Tutorial ). Apr 29, 2025 : Support AMX-Int8、 AMX-BF16 and Qwen3MoE ( Tutorial ) Apr 9, 2025 : Experimental support for LLaMA 4 models ( Tutorial ). Apr 2, 2025 : Support Multi-concurrency. ( Tutorial ). Mar 15, 2025 : Support ROCm on AMD GPU ( Tutorial ). Mar 5, 2025 : Support unsloth 1.58/2.51 bits weights and IQ1_S/FP8 hybrid weights. Support 139K Longer Context for DeepSeek-V3 and R1 in 24GB VRAM. Feb 25, 2025 : Support FP8 GPU kernel for DeepSeek-V3 and R1; Longer Context . Feb 15, 2025 : Longer Context (from 4K to 8K for 24GB VRAM) & Slightly Faster Speed （+15%, up to 16 Tokens/s), update docs and online books . Feb 10, 2025 : Support Deepseek-R1 and V3 on single (24GB VRAM)/multi gpu and 382G DRAM, up to 3~28x speedup. For detailed show case and reproduction tutorial, see here . Aug 28, 2024 : Decrease DeepseekV2’s required VRAM from 21G to 11G. Aug 15, 2024 : Update detailed tutorial for injection and multi-GPU. Aug 14, 2024 : Support llamfile as linear backend. Aug 12, 2024 : Support multiple GPU; Support new model: mixtral 8*7B and 8*22B; Support q2k, q3k, q5k dequant on gpu. Aug 9, 2024 : Support windows native. 🚀 Inference - High-Performance kt-kernel Serving CPU-optimized kernel operations for heterogeneous LLM inference. AMX/AVX Acceleration : Intel AMX and AVX512/AVX2 optimized kernels for INT4/INT8 quantized inference MoE Optimization : Efficient Mixture-of-Experts inference with NUMA-aware memory management Quantization Support : CPU-side INT4/INT8 quantized weights, GPU-side GPTQ support Easy Integration : Clean Python API for SGLang and other frameworks CPU-GPU hybrid inference for large MoE models Integration with SGLang for production serving Heterogeneous expert placement (hot experts on GPU, cold experts on CPU) Performance Examples: 👉 Full Documentation → 🎓 SFT - Fine-Tuning with LlamaFactory Research paper: KTransformers-FineTune (KT-FT) · PDF . KTransformers × LlamaFactory integration for ultra-large MoE model fine-tuning. The new Cookbook provides an easy-to-use path from hardware checks and installation to BF16/FP8/INT8 configuration, LoRA/full training, resource planning, and troubleshooting. Kimi K2.5 / K2.6 LoRA fine-tuning is supported with native RAWINT4 routed experts. Follow the PyPI installation, training, resume and SGLang serving guide (中文) . End-to-end validation uses Kimi K2.5; K2.6 follows the same model-architecture path. Multi-Backend Support : CPU/GPU hybrid fine-tuning with INT8/INT4 quantization Ultra-Large MoE Support : Fine-tune models like DeepSeek-V3/R1 on limited GPU memory Faster than ZeRO-Offload : 6-12x training speedup in benchmarked MoE SFT workloads Lower CPU Memory : About half the CPU memory of the previous KT SFT path in the benchmarked setup LlamaFactory Integration : Seamless integration with the popular fine-tuning framework 👉 Easy-to-Use Fine-Tuning Cookbook → 👉 Quick Start → 👉 Full Documentation → If you use KTransformers in your research, please cite the relevant paper. Inference paper Fine-tuning paper KTransformers-FineTune (KT-FT) · Full BibTeX 👥 Contributors & Team Developed and maintained by: MADSys Lab @ Tsinghua University Community contributors We welcome contributions! Please feel free to submit issues and pull requests. 💬 Community & Support GitHub Issues : Report bugs or request features WeChat Group : See archive/WeChatGroup.png 📦 KT original Code The original integrated KTransformers framework has been archived to the archive/ directory for reference. The project now organizes the two capabilities above from the kt-kernel source tree for clearer documentation and maintenance. For the original documentation with full quick-start guides and examples, see: archive/README.md (English) archive/README_ZH.md (中文) High-performance kernel operations for KTransformers, featuring CPU-optimized MoE inference with AMX, AVX, KML and blis (amd library) support. Installation Option 1: Install from PyPI (Recommended for Most Users) Option 2: Install from Source (For Local Use or Custom Builds) Option 1: Install from PyPI (Recommended for Most Users) Option 2: Install from Source (For Local Use or Custom Builds) KT CLI Overview Integration with SGLang Installation Steps Complete Example: Qwen3-30B-A3B KT-Kernel Parameters Installation Steps Complete Example: Qwen3-30B-A3B KT-Kernel Parameters Direct Python API Usage Advanced Options Manual Configuration (Advanced) Advanced Options Manual Configuration (Advanced) Build Configuration Manual Installation (Without install.sh) Manual Installation (Without install.sh) Error Troubleshooting CUDA Not Found hwloc Not Found hwloc Not Found Weight Quantization Current Support Status: ✅ Native Precision with AVX512/AMX : Supported with AVX512 CPUs in FP8 , BF16 and RAWINT4 format - Guide ✅ Intel CPUs with AMX : Fully supported (using weights converted to INT4/INT8 format) ✅ Universal CPU (llamafile backend) : Supported (using GGUF-format weights) ✅ AMD CPUs with BLIS : Supported (for int8 prefill & decode) - Guide We are developing a simpler way to use KTransformers. Check out the KT-CLI Guide for more details. CPU-Optimized MoE Kernels : High-throughput MoE expert kernels optimized for instruction sets. AVX512 Native Precision Backend : FP8 / BF16 / INT4 native MoE backend for AVX512-capable servers. AMX INT4/INT8 Backend : INT4 / INT8 quantized expert inference backend for AMX-capable servers. Llamafile CPU Backend : AVX2/AVX512-based MoE backend built on Llamafile for universal CPU deployment. NUMA-Aware Execution : Thread pool and memory layout designed for multi-socket / multi-NUMA machines. Option 1: Install from PyPI (Recommended for Most Users) Install the latest version with a single command: Note : Check the latest version on PyPI ✅ Automatic CPU detection : Detects your CPU and loads the optimal kernel variant ✅ CPU multi-variant support : Includes AMX, AVX512 (Base/VNNI/VBMI/BF16), and AVX2 variants ✅ CUDA support included : GPU acceleration for NVIDIA GPUs (SM 80, 86, 89, 90) ✅ No compilation needed : Pre-built wheels for Python 3.10, 3.11, 3.12 ✅ Static CUDA runtime : No CUDA toolkit installation required ✅ Works on CPU-only systems : CUDA features automatically disabled when GPU not available Python 3.10, 3.11, or 3.12 Linux x86-64 (manylinux_2_17 compatible) CPU with AVX2 support (Intel Haswell 2013+, AMD Zen+) Optional: NVIDIA GPU with compute capability 8.0+ for CUDA features CUDA Installation (GPU Acceleration) For NVIDIA GPU-accelerated inference: ✅ Multi-architecture support : Single wheel supports SM 80/86/89/90 (Ampere, Ada, Hopper) ✅ Static CUDA runtime : No CUDA toolkit installation required ✅ Broad compatibility : Works with CUDA 11.8+ and 12.x drivers ✅ PyTorch compatible : Works with any PyTorch CUDA variant (cu118, cu121, cu124) Python 3.10, 3.11, or 3.12 Linux x86-64 (manylinux_2_17 compatible) NVIDIA GPU with compute capability 8.0+ (Ampere or newer) ✅ Supported: A100, RTX 3000/4000 series, H100 ❌ Not supported: V100, P100, GTX 1000/2000 series (too old) ✅ Supported: A100, RTX 3000/4000 series, H100 ❌ Not supported: V100, P100, GTX 1000/2000 series (too old) NVIDIA driver with CUDA 11.8+ or 12.x support (no CUDA toolkit needed) GPU Compatibility Matrix: CUDA Driver Compatibility (for GPU features): CUDA 11.8, 11.9, 12.0-12.6+: Full support CUDA 11.0-11.7: Not supported (upgrade driver or use CPU-only) CPU Variants Included: The wheel includes 6 optimized variants that are automatically selected at runtime based on your CPU: Verify installation: Environment Variables: Option 2: Install from Source (For Local Use or Custom Builds) Build from source for local installation or when you need AMD (BLIS), ARM (KML), or custom CUDA versions. First, initialize git submodules and create a conda environment: Quick Installation (Recommended) Simply run the install script - it will auto-detect your CPU and optimize for best performance: What happens automatically: Auto-detects CPU capabilities (AMX, AVX512_VNNI, AVX512_BF16) Installs system dependencies ( cmake , libhwloc-dev , pkg-config ) Builds optimized binary for your CPU only (using -march=native ) Software fallbacks : Automatically enabled for CPUs without VNNI/BF16 Optional: Two-step installation CPU Requirements by Backend: Software Fallback Support (AVX512 backends): ✅ VNNI fallback: Uses AVX512BW instructions ✅ BF16 fallback: Uses AVX512F instructions ✅ Older AVX512 CPUs (Skylake-X, Cascade Lake) can run RAWINT4 with fallbacks ⚠️ Portability Note: The default build is optimized for your specific CPU and may not work on different/older CPUs. For portable builds or binary distribution, see Manual Configuration below. ⚠️ AMD BLIS backend users: See installation guide for AMD-specific setup. After installation, verify that the CLI is working: Expected output: You can also verify the Python module directly: KT CLI Overview The kt command-line tool provides a unified interface for running and managing KTransformers models: Quick Start Example: Run kt --help for more options, or kt <command> --help for command-specific help. Integration with SGLang KT-Kernel can be used standalone via Direct Python API or integrated with SGLang for production deployment. This section describes SGLang integration to enable CPU-GPU heterogeneous inference, where “hot” experts run on GPU and “cold” experts run on CPU for optimal resource utilization. Installation Steps 1. Install SGLang Install the kvcache-ai fork of SGLang (required for kt-kernel support): Important: Use sglang-kt (kvcache-ai fork), not the official sglang package. If you have the official version installed, uninstall it first: pip uninstall sglang -y 2. Prepare Weights You need both GPU weights and CPU-side expert weights for heterogeneous inference. The exact format depends on the backend: GPU Weights (for all backends): Use the model weights required by SGLang for GPU inference (for example, the original or already-quantized model directory from Hugging Face). CPU Weights (AMX backend: AMXINT4 / AMXINT8 ): Quantize weights to AMX-optimized INT4/INT8 format using the provided script: --input-path : Path to GPU-side original weights --input-type : Depends on your GPU weights type ( fp8 , fp16 , or bf16 ) In SGLang integration, --kt-weight-path should point to this converted CPU weights directory. Supported input formats: FP8, FP16, BF16 → INT4/INT8. CPU Weights (LLAMAFILE backend: LLAMAFILE ): LLAMAFILE uses pre-quantized GGUF weights on the CPU side directly, without running convert_cpu_weights.py . You need to: Download a GGUF model directly from the web (e.g., GGUF repos on Hugging Face / Modelscope); In SGLang integration, use that GGUF directory as --kt-weight-path . KT-Kernel supports multiple GGUF quantization formats such as Q4_KM , Q4_K , Q5_K , etc. Choose based on your latency and accuracy requirements. 3. Launch SGLang Server Start the SGLang server with your normal SGLang parameters, and add the following KT-Kernel specific parameters to enable CPU-GPU heterogeneous inference: KT-Kernel Parameters to Add: --kt-method : Backend method (AMXINT4, AMXINT8, or LLAMAFILE) --kt-weight-path : Path to the converted CPU weights --kt-cpuinfer : Number of CPU inference threads (set to physical cores) --kt-threadpool-count : Number of thread pools (set to NUMA node count) --kt-num-gpu-experts : Number of experts to keep on GPU --kt-max-deferred-experts-per-token : Deferred experts for pipelined execution See KT-Kernel Parameters section below for detailed parameter tuning guidelines. Complete Example: Qwen3-30B-A3B This example demonstrates the full workflow from downloading weights to launching the server, showing Native backend , AMX backend and LLAMAFILE backend options. Hardware Configuration: GPU : NVIDIA RTX 4090 24GB CPU : 2x Intel Xeon Gold 6454S (64 physical cores total, 128 threads, 2 NUMA nodes) Model : Qwen3-30B-A3B How to verify your system configuration: Parameter Rationale: --kt-cpuinfer 64 : Set to physical cores (64), not hyperthreads (128) --kt-threadpool-count 2 : 2 NUMA nodes detected (dual-socket system) --kt-num-gpu-experts 32 : With 24GB GPU memory, we can fit ~32 experts on GPU for this model (varies by model architecture and actual memory usage) --kt-max-deferred-experts-per-token 2 : Enable pipelined execution; allows CPU to process next batch while GPU completes current batch --kt-gpu-prefill-token-threshold 2048 : Use layerwise prefill strategy when token count exceeds 2048 (for native backends only) Option A: Native Backend (BF16) For AVX512 CPUs with BF16 support. Step 1: Download model weights Step 2: Launch SGLang server Option B: AMX Backend (AMXINT8) For Intel CPUs with AMX instruction set support. Step 1: Download model weights Step 2: Convert to CPU weights (AMXINT8) Step 3: Launch SGLang server Option C: LLAMAFILE Backend (GGUF) For universal CPUs (no AMX required), using pre-quantized GGUF weights directly. Step 1: Download GPU weights (original model) Step 2: Download CPU weights (GGUF format) Step 3: Launch SGLang server KT-Kernel Parameters Parameter Guidelines: kt-method : Choose based on your CPU and weight format: AMXINT4 : Best performance on AMX CPUs with INT4 quantized weights (May cause huge accuracy drop for some models, e.g., Qwen3-30B-A3B) AMXINT8 : Higher accuracy with INT8 quantized weights on AMX CPUs RAWINT4 : Native INT4 weights shared by CPU and GPU (currently supports Kimi-K2-Thinking model). See Kimi-K2-Thinking Native Tutorial for details. FP8 , FP8_PERCHANNEL : FP8 weights shared by CPU and GPU BF16 : BF16 weights shared by CPU and GPU LLAMAFILE : GGUF-based backend kt-method : Choose based on your CPU and weight format: AMXINT4 : Best performance on AMX CPUs with INT4 quantized weights (May cause huge accuracy drop for some models, e.g., Qwen3-30B-A3B) AMXINT8 : Higher accuracy with INT8 quantized weights on AMX CPUs RAWINT4 : Native INT4 weights shared by CPU and GPU (currently supports Kimi-K2-Thinking model). See Kimi-K2-Thinking Native Tutorial for details. FP8 , FP8_PERCHANNEL : FP8 weights shared by CPU and GPU BF16 : BF16 weights shared by CPU and GPU LLAMAFILE : GGUF-based backend kt-cpuinfer : Set to the number of physical CPU cores (not hyperthreads). Check physical cores: lscpu | grep -E "^CPU\(s\)|Thread\(s\) per core" Physical cores = CPU(s) / Thread(s) per core Example: If CPU(s)=128 and Thread(s) per core=2, then physical cores = 64 Important : Do NOT set to hyperthread count - this will degrade performance kt-cpuinfer : Set to the number of physical CPU cores (not hyperthreads). Check physical cores: lscpu | grep -E "^CPU\(s\)|Thread\(s\) per core" Physical cores = CPU(s) / Thread(s) per core Example: If CPU(s)=128 and Thread(s) per core=2, then physical cores = 64 Important : Do NOT set to hyperthread count - this will degrade performance kt-threadpool-count : Set to the number of NUMA nodes . Check NUMA count: lscpu | grep "NUMA node(s)" Or use: numactl --hardware | grep "available" Note : NUMA node count is NOT necessarily the number of physical CPUs It represents memory domains, which may be divided within a single CPU or across multiple CPUs Use the NUMA node count from lscpu , regardless of physical CPU count Typical values: 1-2 for single-socket, 2-4 for dual-socket systems This enables better memory bandwidth utilization across NUMA domains kt-threadpool-count : Set to the number of NUMA nodes . Check NUMA count: lscpu | grep "NUMA node(s)" Or use: numactl --hardware | grep "available" Note : NUMA node count is NOT necessarily the number of physical CPUs It represents memory domains, which may be divided within a single CPU or across multiple CPUs Use the NUMA node count from lscpu , regardless of physical CPU count It represents memory domains, which may be divided within a single CPU or across multiple CPUs Use the NUMA node count from lscpu , regardless of physical CPU count Typical values: 1-2 for single-socket, 2-4 for dual-socket systems This enables better memory bandwidth utilization across NUMA domains kt-num-gpu-experts : Determine based on GPU memory and profiling: More GPU experts = lower latency but higher GPU memory usage (May cause OOM) kt-num-gpu-experts : Determine based on GPU memory and profiling: More GPU experts = lower latency but higher GPU memory usage (May cause OOM) kt-max-deferred-experts-per-token : Enables pipelined execution: 0 : Synchronous execution (simpler, higher latency) 1-4 : Deferred execution (recommended range; good latency/quality balance, requires tuning) 5-7 : Highest latency reduction but may introduce noticeable accuracy loss; use with care kt-max-deferred-experts-per-token : Enables pipelined execution: 0 : Synchronous execution (simpler, higher latency) 1-4 : Deferred execution (recommended range; good latency/quality balance, requires tuning) 5-7 : Highest latency reduction but may introduce noticeable accuracy loss; use with care kt-gpu-prefill-token-threshold (FP8 and RAWINT4 only): Controls prefill strategy for native FP8 and INT4 inference: ≤ threshold : Uses hybrid CPU+GPU prefill. No extra VRAM needed, but performance degrades slowly as token count increases. > threshold : Uses layerwise GPU prefill. Performance scales better with longer sequences, but requires one MoE layer extra VRAM (e.g., ~9GB+ for Kimi-K2-Thinking and ~3.6GB for MiniMax-M2.1). Only applicable when --kt-method RAWINT4 or --kt-method FP8 is used. kt-gpu-prefill-token-threshold (FP8 and RAWINT4 only): Controls prefill strategy for native FP8 and INT4 inference: ≤ threshold : Uses hybrid CPU+GPU prefill. No extra VRAM needed, but performance degrades slowly as token count increases. > threshold : Uses layerwise GPU prefill. Performance scales better with longer sequences, but requires one MoE layer extra VRAM (e.g., ~9GB+ for Kimi-K2-Thinking and ~3.6GB for MiniMax-M2.1). Only applicable when --kt-method RAWINT4 or --kt-method FP8 is used. kt-enable-dynamic-expert-update : Enables dynamic expert placement updates during inference. During layerwise prefill, the system collects actual routing statistics and redistributes GPU experts accordingly. Requires --kt-gpu-prefill-token-threshold to be set, and prefill length must be ≥ the threshold value. Particularly effective at lower GPU expert ratios (10%-70%), where it can significantly outperform static strategies. See Expert Scheduling Tutorial for benchmarks and details. kt-enable-dynamic-expert-update : Enables dynamic expert placement updates during inference. During layerwise prefill, the system collects actual routing statistics and redistributes GPU experts accordingly. Requires --kt-gpu-prefill-token-threshold to be set, and prefill length must be ≥ the threshold value. Particularly effective at lower GPU expert ratios (10%-70%), where it can significantly outperform static strategies. See Expert Scheduling Tutorial for benchmarks and details. kt-expert-placement-strategy : Determines which experts are placed on GPU at server startup. uniform : Distributes GPU experts evenly across all MoE layers. Default option, no prior statistics needed. frequency : Places the most frequently activated experts on GPU. Best performance when activation statistics are available; requires --init-expert-location pointing to a .pt statistics file. front-loading : Fills GPU experts from the first MoE layer onwards. random : Randomly selects experts with a fixed seed (42). See Expert Scheduling Tutorial for strategy comparison. kt-expert-placement-strategy : Determines which experts are placed on GPU at server startup. uniform : Distributes GPU experts evenly across all MoE layers. Default option, no prior statistics needed. frequency : Places the most frequently activated experts on GPU. Best performance when activation statistics are available; requires --init-expert-location pointing to a .pt statistics file. front-loading : Fills GPU experts from the first MoE layer onwards. random : Randomly selects experts with a fixed seed (42). See Expert Scheduling Tutorial for strategy comparison. Direct Python API Usage For standalone usage without SGLang, you can use KT-Kernel directly via Python API: Advanced Options Manual Configuration (Advanced) For portable builds, binary distribution, or cross-machine deployment, you need to manually specify target instruction sets: Optional: Override VNNI/BF16 detection Optional: oneDNN INT8 BRGEMM on non-AMX AVX512 CPUs See ./install.sh --help for all available options. Build Configuration Manual Installation (Without install.sh) If you prefer manual installation without the install.sh script: 1. Install System Dependencies cmake (recommended: conda install -y cmake ) libhwloc-dev and pkg-config 2. Set Build Configuration Instruction Set Details: Example Configurations: 3. Build and Install Error Troubleshooting Make sure you have the CUDA toolkit installed and nvcc is in your system PATH. Try export CMAKE_ARGS="-D CMAKE_CUDA_COMPILER=$(which nvcc)" and reinstall again. hwloc Not Found Run sudo apt install libhwloc-dev if on a Debian-based system or build from source: https://www.open-mpi.org/projects/hwloc/. Weight Quantization For AMX backends ( AMXINT4 / AMXINT8 ), CPU-side experts must be converted to AMX-friendly INT4/INT8 format using the provided script: Supported formats: FP8, FP16, BF16 → INT4/INT8 For LLAMAFILE backend ( LLAMAFILE ), CPU-side experts are loaded directly from GGUF weights. You do not need to run the AMX conversion script; instead, download a GGUF model from the web (e.g., a GGUF repo on Hugging Face) and point weight_path / SGLang --kt-weight-path (or --model when appropriate) to that GGUF directory. KT-Kernel supports multiple GGUF quantization types such as Q4_KM , Q4_K , Q5_K , etc. For detailed documentation, advanced options, and low-memory mode, see scripts/README.md . Commit messages should follow the Conventional Commits specification: https://www.conventionalcommits.org/ Please format your code before committing: You may need a newer clang-format (at least version 18). In a conda environment: It’s also recommended to install black for Python code formatting: Introduction Fine-Tuning Results (Examples) Fine-Tuning Results (Examples) Quick to Start Environment Setup Core Feature 1: Use KTransformers backend to fine-tune ultra-large MoE models Core Feature 2: Chat with the fine-tuned model (base + LoRA adapter) Core Feature 3: Batch inference + metrics (base + LoRA adapter) Environment Setup Core Feature 1: Use KTransformers backend to fine-tune ultra-large MoE models Core Feature 2: Chat with the fine-tuned model (base + LoRA adapter) Core Feature 3: Batch inference + metrics (base + LoRA adapter) KT Fine-Tuning Speed (User-Side View) End-to-End Performance GPU/CPU Memory Footprint End-to-End Performance GPU/CPU Memory Footprint MadSys Lab, KVCache-AI Team, Approaching AI, LLaMA-Factory Team From DeepSeek-V3/R1 to Qwen3-MoE and Kimi-K2 , each wave of open-sourced large models brings leaps in performance and scale. However, many researchers and developers are constrained by expensive GPUs and models with tens or even hundreds of billions of parameters, making it hard to fine-tune very large models under limited resources . To bridge this gap, we propose a practical approach: combining KTransformers with LLaMA-Factory . With just 2–4 RTX 4090s and a high-memory CPU, you can fine-tune ultra-large MoE models like DeepSeek-671B. Our goal is to give resource-constrained researchers a local path to explore fine-tuning ultra-large models , and also a fast way to customize smaller models (e.g., 14B/30B) for specific scenarios. We validate the setup using stylized dialogue , Westernized translation tone , and medical Q&A as representative tasks, showing that personalized adaptation can be achieved within hours . As shown below, LLaMA-Factory is the unified orchestration/configuration layer for the whole fine-tuning workflow—handling data, training scheduling, LoRA injection, and inference interfaces. KTransformers acts as a pluggable high-performance backend that takes over core operators like Attention/MoE under the same training configs, enabling efficient GPU+CPU heterogeneous cooperation . Within LLaMA-Factory, we compared LoRA fine-tuning with HuggingFace , Unsloth , and KTransformers backends. KTransformers is the only workable 4090-class solution for ultra-large MoE models (e.g., 671B) and also delivers higher throughput and lower GPU memory on smaller MoE models (e.g., DeepSeek-14B). † 1400 GB is a theoretical FP16 full-parameter resident footprint (not runnable). 70 GB is the measured peak with KT strategy (Attention on GPU + layered MoE offload). Fine-Tuning Results (Examples) Stylized Dialogue (CatGirl tone) Dataset: NekoQA-10K . Goal: improve style consistency and recognizability. The figure compares responses from the base vs. fine-tuned models. The fine-tuned model maintains the target tone and address terms more consistently (red boxes), validating the effectiveness of style-transfer fine-tuning . (1) Translational-Style-ChatLLM , which asks for an exaggerated, Westernized translation tone—clear, stylized customization. (2) AfriMed-QA (ACL 2025), a medical dataset for African contexts with strong domain specificity, including multiple-choice and short-answer sub-tasks—well-suited for vertical fine-tuning evaluation. The tables show metrics before vs. after LoRA fine-tuning. We observe large improvements across metrics, verifying fine-tuning effectiveness: Even for ultra-large MoE models, KTransformers-backed fine-tuning achieves strong task performance quickly. This section shows how to install and use LLaMA-Factory + KTransformers for fine-tuning and inference: Environment setup Fine-tune ultra-large MoE models with KTransformers backend Load the fine-tuned model (base + LoRA adapter) for chat/inference Batch inference and metric evaluation Environment Setup According to the following example, install both the KTransformers and LLaMA-Factory environments simultaneously. This time, to simplify the installation process of KTransformers, use the PyPI packages to avoid local compilation. The detailed installation steps are as follows: (Note: Make sure your local Python version , Torch version , and CUDA version are compatible with the installed packages.) Usage tip: In LLaMA-Factory YAML, set use_kt: true and pick a kt_optimize_rule file to have KTransformers handle the core compute. The features below show typical configs. Core Feature 1: Use KTransformers backend to fine-tune ultra-large MoE models Run the command: USE_KT=1 ACCELERATE_USE_KT=true accelerate launch --config_file examples/ktransformers/accelerate/fsdp2_kt_bf16.yaml -m llamafactory.cli train examples/ktransformers/train_lora/deepseek_v3_lora_sft_kt.yaml . Note: You must provide a BF16 model. DeepSeek-V3-671B is released in FP8 by default; convert with DeepSeek-V3/inference/fp8_cast_bf16.py . kt_optimize_rule controls placement strategy . See also ktransformers/optimize_rules . Naming hints ( * = wildcard): Example: DeepSeek-V3-Chat-sft-amx-multi-gpu.yaml = V3-Chat fine-tuning with AMX and 2-GPU model parallel. We recommend AMX acceleration where available ( lscpu | grep amx ). AMX supports BF16/INT8. Example: Outputs go to output_dir in safetensors format plus adapter metadata for later loading. Core Feature 2: Chat with the fine-tuned model (base + LoRA adapter) Run the command: llamafactory-cli chat examples/inference/qwen3_lora_sft.yaml . Use the safetensors adapter trained with KT for inference. We also support GGUF adapters: for safetensors, set the directory ; for GGUF, set the file path in adapter_name_or_path . During loading, LLaMA-Factory maps layer names to KT’s naming. You’ll see logs like Loaded adapter weight: XXX -> XXX : Core Feature 3: Batch inference + metrics (base + LoRA adapter) Run the command: API_PORT=8000 llamafactory-cli api examples/inference/qwen3_lora_sft.yaml . Invoke the KT fine-tuned adapter to provide the API; the usage logic of other APIs is consistent with the native LLaMA-Factory approach. KT Fine-Tuning Speed (User-Side View) End-to-End Performance step_time : wall-clock time for a full optimization step (tensor movement + Attention + MoE + other compute). tokens_per_step = GAS × qlen ; token/s = tokens_per_step / step_time . Settings: GAS=16 , qlen=512 (→ tokens_per_step = 8192 ); LoRA ( r=8, alpha=32, dropout=0.1 ); AMX enabled; GPU: RTX 4090, CPU: Intel Xeon Platinum 8488C. DeepSeek-V3-671B: step_time = 203 s → token/s ≈ 8192 / 203 ≈ 40.35 DeepSeek-V2-Lite-14B: step_time = 36 s → token/s ≈ 8192 / 36 ≈ 227.6 GPU/CPU Memory Footprint DeepSeek-V3 (671B; 61 layers with 58 MoE): ~ 70 GB total GPU VRAM (multi-GPU), ~ 1.2–1.3 TB CPU RAM. DeepSeek-V2-Lite (14B; 27 layers with 26 MoE): ~ 5.5 GB GPU VRAM, ~ 30 GB CPU RAM. By integrating KTransformers LoRA fine-tuning into LLaMA-Factory , we provide a practical guide for efficient training and deployment of MoE LLMs. KT brings cutting-edge optimizations (DeepSeek/Qwen/Kimi support with AMX-accelerated kernels), and LoRA enables customization under very low GPU memory. LLaMA-Factory offers a friendly, unified interface. This integration (akin to Unsloth-style speedups) means even models with tens to hundreds of billions of parameters can be fine-tuned and deployed with low latency on commodity hardware. You get memory savings, speed-ups, and usability together. We encourage you to try LLaMA-Factory + KT for your next MoE project and follow this guide. Feedback is welcome! Kimi K2.5: PyPI Install, LoRA Training, Resume and Serving (中文) KTransformers × LlamaFactory MoE Fine-Tuning Cookbook KTransformers × LlamaFactory MoE Fine-Tuning Cookbook (中文) v0.6.1 Quick Start Fine-Tuning User Guide KT-FT Fine-Tuning and Inference Loop DeepSeek-V4-Flash LoRA quickstart DeepSeek-V4-Flash MXFP4 LoRA training and static deployment Developer Technical Notes Injection Tutorial Research paper: KTransformers-FineTune (KT-FT) · PDF . From Qwen3.5 to DeepSeek-V4, Kimi-K3, and GLM-5.2, each new ultra-large open model brings a major leap in capability and scale. However, the cost of high-end GPUs still prevents many researchers and developers from fine-tuning these models under constrained resources. KTransformers and LlamaFactory provide a practical alternative: with 1–4 RTX 4090 GPUs and a CPU platform with sufficient memory, users can fine-tune trillion-parameter MoE (Mixture of Experts) models such as the DeepSeek-V3/V4 family, Kimi-K2.5, and GLM-5.2. KTransformers integrates with LlamaFactory while preserving its familiar training workflow. LlamaFactory remains the unified configuration and orchestration layer for data processing, training, LoRA injection, and inference integration. The GPU runs Attention and Shared Expert modules, while KTransformers manages Routed Experts in CPU memory for heterogeneous GPU–CPU execution. Technical overview: KTransformers places compute-intensive Attention and shared modules on the GPU, while storing the large, sparsely activated Routed Experts in CPU memory. This avoids repeatedly transferring large Expert weights over PCIe and lets the CPU runtime select an optimized backend according to the weight format and available instruction set. Guide at a glance: For a first run, start with native BF16 + LoRA or full fine-tuning + kt_backend: auto . This path requires no weight conversion and starts with two YAML files. If the model provides a native FP8 checkpoint and the training method is LoRA, KTransformers can load the native FP8 Expert weights directly to reduce host-memory usage. Use converted INT8 or AMXINT4 weights only when host memory remains the limiting resource and the matching converted artifacts are available. Check hardware and install Choose a fine-tuning setup Custom YAML: advanced settings Performance and resource estimates Appendix: complete configurations and Q&A 1. Check hardware and install 1.1 Check hardware resources Check the CPU instruction set first to identify the available acceleration backend: Then check GPU memory, host memory, and disk capacity: Confirm two requirements: Instruction-set compatibility: Native FP8 requires a compatible AMD/x86 CPU with AVX512F, AVX512_BF16, AVX512_VNNI, and AVX512_VBMI. For INT8, auto can select an AMX-INT8 or AVX512-VNNI implementation. AMXINT4 requires AMX and matching converted weights. Native BF16 performs best when AMX is available. Sufficient capacity: Host memory must hold the Expert weights and leave room for activations, gradients, and optimizer states. Disk capacity must also cover checkpoints and temporary files. 1.2 Install the environment Use a clean Python 3.11 environment. Pin PyTorch 2.9.1 before installing LlamaFactory, and install the KT dependencies last so standard transformers or accelerate packages do not overwrite the KT variants: After installation, check dependency consistency and the versions that Python actually imports: The compatible environment should include these versions: 2. Choose a fine-tuning setup Choose the Expert weight format first, then select LoRA or full fine-tuning. After making both choices, use the corresponding training YAML and Accelerate YAML below. 2.1 Choose the Expert weight format Native versus converted precision describes the Expert weight format. AMX (Advanced Matrix Extensions) and AVX512 (Advanced Vector Extensions 512-bit) describe CPU instruction sets and backends; they are a separate dimension. Prefer native BF16 or native FP8 when host memory allows. Use INT8 or AMXINT4 only when matching converted weights are available and reducing Expert memory is necessary. 2.2 Choose LoRA or full fine-tuning Expert weight format and parameter-update scope are separate configuration dimensions, but the supported combinations have clear boundaries. Native BF16 supports both LoRA and full fine-tuning. Native FP8, INT8, and AMXINT4 recipes use LoRA with frozen base Experts. Full fine-tuning uses BF16. 2.3 Responsibilities of the two YAML files Each run uses two YAML files. The training YAML defines the task and owns every KTransformers setting. The Accelerate YAML defines only distributed execution and FSDP2. Current LlamaFactory rejects kt_config in the Accelerate YAML. Check three relationships before launch: The number of GPUs in CUDA_VISIBLE_DEVICES must equal num_processes . If kt_config.kt_model_max_length is set manually in the training YAML, it must cover cutoff_len plus the runtime token margin. Write the LoRA rank only at the top level of the training YAML. LlamaFactory derives the internal KT fields. Remove LoRA-only fields for full fine-tuning. 2.4 Base YAML for four common setups Copy the complete configurations from Appendix A.1 and A.2, then replace the fields shown for the selected setup. Each block labels the training YAML and Accelerate YAML sections. kt_config always belongs in the training YAML. 2.4.1 Native BF16 + LoRA 2.4.2 Native FP8 + LoRA 2.4.3 INT8 / AMXINT4 + LoRA 2.4.4 Native BF16 + full fine-tuning 3. Custom YAML: advanced settings After a base setup launches successfully, tune the following parameters according to the training objective and resource bottleneck. Keep the recommended values unless there is a clear reason to change them, and avoid changing several parameter groups at once. (1) LoRA capacity: lora_rank , lora_alpha , lora_target , lora_dropout Change these four fields only at the top level of the training YAML. LlamaFactory automatically derives the rank, alpha, and dropout used internally by KTransformers. Do not repeat them in kt_config or the Accelerate YAML. Start with lora_rank: 8 , lora_alpha: 16 , lora_target: all , and lora_dropout: 0.0 . If task adaptation is insufficient, increase the rank to 16 and then 32 while keeping alpha at about twice the rank. If a small dataset shows clear overfitting, increase dropout to 0.05 . (2) Sequence length: cutoff_len Set this field in the training YAML according to the effective token length of most samples. Do not use the maximum dataset length solely for a small number of long samples. Longer sequences increase activation and KT buffer usage, which raises both host-memory and GPU-memory requirements. The system derives the KT model-length capacity from cutoff_len . To increase it manually, set kt_model_max_length in the training YAML’s kt_config . It must be greater than cutoff_len and include room for additional runtime tokens. (3) Batch: per_device_train_batch_size , gradient_accumulation_steps Set both fields in the training YAML. For ultra-large MoE models, start with per_device_train_batch_size: 1 . Increase the micro batch only after confirming sufficient GPU and host memory. If GPU memory is insufficient, keep the micro batch at 1 and use gradient accumulation to increase the effective batch. (4) Activation recomputation and CPU Activation Reuse: disable_gradient_checkpointing , kt_cpu_activation In the training YAML, disable_gradient_checkpointing: false enables gradient checkpointing to trade additional computation for lower activation memory. Setting it to true retains more intermediate results and may improve speed, but uses more memory. When host memory is sufficient, add this top-level field to the same training YAML: This option retains CPU Expert activations during checkpoint recomputation, using more host memory to reduce repeated CPU work. When gradient checkpointing is enabled and the field is omitted, CPU and GPU activations are recomputed. kt_cpu_activation controls CPU Expert activation retention, while disable_gradient_checkpointing controls the overall checkpoint-recomputation policy; do not treat them as one switch. (5) CPU Expert reuse and threads: kt_share_backward_bb , kt_num_threads Both fields belong in the training YAML’s kt_config . Keep kt_share_backward_bb: true to reuse the buffer required by CPU Expert backward computation. It is separate from kt_cpu_activation ; do not disable it unless the configuration for the current model explicitly requires that change. Set kt_num_threads to the number of physical CPU cores available to the job, not the total number of logical cores including SMT threads. If data preprocessing shares the same CPUs, reserve cores for the DataLoader and system processes. The system derives kt_threadpool_count from the NUMA topology. Change it manually only after confirming the CPU socket and NUMA layout. (6) Resume from a checkpoint: resume_from_checkpoint Set this field in the training YAML to the target checkpoint-* directory. Omit it for a new run. Keep output_dir pointed at the output directory for the current job, never at the base-model directory, and keep overwrite_output_dir: false to avoid overwriting existing results. 4. Performance and resource estimates The table below uses a 2K context ( sequence length = 2048 ), per-GPU batch size 1, and gradient accumulation 1. It includes only accepted KTransformers BF16 runs that completed real LoRA training. Throughput includes forward, loss, backward, and optimizer steps. The GPU-memory column reports the highest single-GPU peak from the measured run. The host-memory column reports peak CPU RSS for the training process tree, normalized to GiB. These values are minimum references for the measured configurations, not safety margins. Reserve additional capacity for data loading, caches, checkpoint saving, and system processes. Results with different GPU counts should not be used to infer linear scaling directly. Increasing cutoff_len , batch size, or KT cache depth increases resource usage. Full fine-tuning also requires additional capacity for gradients, master weights, optimizer states, and complete checkpoints. A.1 Qwen3.5-397B BF16 LoRA training YAML Save as qwen35_397b_bf16_lora.yaml : A.2 FSDP2 Accelerate YAML Save as qwen35_fsdp2_2gpu.yaml : A.3.1 Why can’t a native FP8 checkpoint use the converted INT8 configuration? The weight formats and loading paths are different. Native FP8 preserves the FP8 weights and scales from the checkpoint and uses the AVX512 native-precision backend. The INT8 recipe reads a mutually matched pair of INT8 Routed Expert weights and a BF16 non-expert cache, then lets auto select an AMX-INT8 or AVX512-VNNI implementation. Do not pass a native FP8 checkpoint through kt_weight_path . A.3.2 auto did not select the expected backend Save the lscpu output and startup log, then confirm that the container or virtualization environment did not hide the required instruction set. Native BF16 can select AMXBF16 when AMX is available. Native FP8 should select AVX512. Do not force a backend that the CPU or weight format does not support. A.3.3 Insufficient host memory First reduce concurrent data loading, sequence length, and cache depth, and confirm that two copies of the Expert weights are not loaded. If memory is still insufficient, prepare converted INT8 or AMXINT4 weights. For full fine-tuning, estimate the additional memory for gradients, master weights, and optimizer states separately. OOM (Out of Memory) means GPU or host memory is insufficient. Reduce cutoff_len , per-GPU batch size, and activation usage first, then increase gradient_accumulation_steps to preserve the global batch size. num_processes , the visible GPU count, and the actual model placement must agree. A.3.5 kt_model_max_length does not match cutoff_len When increasing cutoff_len , increase kt_model_max_length as well and leave room for the effective runtime sequence length. A value that is too small causes buffer or shape errors; an unnecessarily large value increases host-memory usage. 研究论文： KTransformers-FineTune（KT-FT） · PDF 。 从 Qwen3.5 到 DeepSeek-V4、Kimi-K3、GLM-5.2，每一次超大模型的开源都带来性能与规模上的巨大跃升。然而，多数研究者与开发者受限于昂贵的显卡，难以在资源受限条件下微调超大模型。面对这种差距，我们提出了一种更具可行性的方案：通过 KTransformers 与 LlamaFactory 的结合，仅需1~4张RTX 4090与较高内存CPU，便可微调 DeepSeek-V3&V4系列/Kimi-K2.5/GLM-5.2等1T规模的 MoE（Mixture of Experts，混合专家）模型。 为给大家提供便捷高效的使用方式，KTransformers 与 LlamaFactory 合作，保持您的工作流不受影响，最大化支持矩阵。如下图所示，LlamaFactory 是整个微调流程的统一调度与配置框架，负责数据处理、训练调度、LoRA 插入与推理接口管理；GPU 运行 Attention、Shared Expert 等模块，KTransformers 接管位于 CPU 与大内存中的 Routed Experts，实现 GPU+CPU 异构协同。 **技术简介：**如下图，KTransformers 通过将 Attention、共享模块等计算密集部分放在 GPU，将体积大但每次只激活少量的 Routed Experts 放入 CPU 内存。避免了大量专家权重在 PCIe 上面反复传输，通过 GPU+CPU 协同计算的方式，最大化性能。CPU 则根据权重格式和硬件能力自动适配了各种后端。 **指南简介：**首次上手建议直接选择 原精度 BF16（16 位浮点）+ LoRA/全量 + kt_backend: auto ：无需转换权重，修改两份 YAML 配置文件即可启动。模型提供原生 FP8 checkpoint 且选择 LoRA 时，可直接使用原生 FP8 Expert 权重降低主机内存占用。仅在系统内存仍然紧张且已准备好 KT 转换权重时，再考虑 INT8 或 AMXINT4 量化方案。 Custom YAML：高级参数

先检查 CPU 指令集，确认可用的加速后端： 再检查 GPU 显存、系统内存与磁盘空间： 指令集匹配： 原生 FP8 需要具备 AVX512F、AVX512_BF16、AVX512_VNNI、AVX512_VBMI 的兼容 AMD/x86 CPU；INT8 可由 auto 在 AMX-INT8 与 AVX512-VNNI 实现之间选择；AMXINT4 需要 AMX 与匹配的转换权重；原生 BF16 在支持 AMX 时性能最佳。 容量充足： 系统内存要容纳 Expert 权重，同时为 activation（激活值）、梯度和 optimizer state（优化器状态）留出空间；磁盘还要容纳 checkpoint 与临时文件。 推荐使用干净的 Python 3.11 环境。先固定 PyTorch 2.9.1，再安装 LlamaFactory；KT 依赖必须最后安装，避免标准版 transformers 或 accelerate 覆盖 KT 定制版本： 安装完成后检查依赖关系，并核对实际导入版本： 先确定 Expert 权重格式，再决定使用 LoRA 还是全量微调。选定后，直接使用本节对应的训练 YAML 和 Accelerate YAML 基础配置。 2.1 选择 Expert 权重格式 “原精度”与“量化”描述 Expert 权重格式；AMX（Advanced Matrix Extensions）与 AVX512（Advanced Vector Extensions 512-bit）描述 CPU 后端指令集，两者不是同一个维度。 系统内存允许时优先选择原生 BF16 或原生 FP8。只有已经准备好匹配的转换权重，并且确实需要降低 Expert 内存占用时，再选择 INT8 或 AMXINT4。 2.2 选择 LoRA 或全量微调 Expert 权重格式与参数更新范围是两个配置维度，但当前可用组合有明确边界。原生 BF16 支持 LoRA 与全量微调；原生 FP8、INT8 和 AMXINT4 配方用于冻结基座 Expert 的 LoRA。全量微调使用 BF16。 2.3 两份 YAML 的分工 一次训练使用两份 YAML。训练 YAML 描述训练任务并持有全部 KTransformers 设置；Accelerate YAML 只描述分布式与 FSDP2 运行方式。当前 LlamaFactory 会拒绝 Accelerate YAML 中的 kt_config 。 CUDA_VISIBLE_DEVICES 中的 GPU 数必须等于 num_processes ； 手动设置训练 YAML 中的 kt_config.kt_model_max_length 时，它必须覆盖 cutoff_len ，并为运行时额外 token 留出余量； LoRA rank 只写训练 YAML 顶层，LlamaFactory 会派生 KT 内部字段；全量微调删除 LoRA 专用字段。 2.4 四种方案的基础 YAML 先从附录 A.1 和 A.2 复制完整配置，再按所选方案替换下面的字段。每个代码块用注释区分训练 YAML 与 Accelerate YAML； kt_config 始终放在训练 YAML。 2.4.1 原生 BF16 + LoRA 2.4.2 原生 FP8 + LoRA 2.4.3 INT8 / AMXINT4 + LoRA 2.4.4 原生 BF16 + 全量微调 3. Custom YAML：高级参数 基础方案能够启动后，再按训练目标和资源瓶颈调整下面的参数。没有明确需求时保留推荐值，不要同时修改多组参数。 （1）LoRA 容量： lora_rank 、 lora_alpha 、 lora_target 、 lora_dropout 这四项只在训练 YAML 顶层修改，LlamaFactory 会自动派生 KTransformers 内部使用的 rank、alpha 和 dropout；不要在 kt_config 或 Accelerate YAML 中重复填写。建议从 lora_rank: 8 、 lora_alpha: 16 、 lora_target: all 、 lora_dropout: 0.0 开始。 任务适配不足时依次把 rank 提高到 16、32，并让 alpha 保持约为 rank 的两倍。小数据集出现明显过拟合时，再把 dropout 提高到 0.05 。 （2）序列长度： cutoff_len 在训练 YAML 中设置，建议按大多数样本的有效 token 长度选择，不要为了少量超长样本直接采用数据集最大长度。序列越长，中间激活值和 KT 缓冲区占用越高，也就是需要的内存和显存越高。 系统会根据 cutoff_len 生成 KT 的模型长度容量。需要手动扩大时，在训练 YAML 的 kt_config 中设置 kt_model_max_length ，并让它大于 cutoff_len 、保留额外 token 的运行余量。 （3）Batch： per_device_train_batch_size 、 gradient_accumulation_steps 这两项在训练 YAML 中修改。超大 MoE 模型优先使用 per_device_train_batch_size: 1 ；确认显存和内存均仍有余量后再增加 micro batch，显存不足时则保持为 1，通过梯度累积扩大有效 batch。 （4）Activation 重计算与 CPU Activation Reuse： disable_gradient_checkpointing 、 kt_cpu_activation 训练 YAML 中设置 disable_gradient_checkpointing: false 会启用 gradient checkpointing，以额外计算换取更低的 activation 内存；设为 true 会保留更多中间结果，速度可能更高，但内存占用也更大。 主机内存充足时，还可以在同一份训练 YAML 顶层设置： 该选项在 checkpoint 重计算期间保留 CPU Expert activation，以增加主机内存占用换取更少的 CPU 重复计算。启用 gradient checkpointing 且未填写该字段时，CPU 与 GPU activation 默认重计算。 kt_cpu_activation 与 disable_gradient_checkpointing 分别控制 CPU Expert activation 保留和整体 checkpoint 重计算，不要将两者合并成一个开关。 （5）CPU Expert 复用与线程： kt_share_backward_bb 、 kt_num_threads 这两项位于训练 YAML 的 kt_config 。保持 kt_share_backward_bb: true ，复用 CPU Expert 反向计算所需的缓冲；它不等同于 kt_cpu_activation 。除非当前模型的配置说明明确要求，否则不要关闭。 kt_num_threads 填写本次作业实际可用的物理 CPU 核数，不要直接填写包含超线程的逻辑核总数。若数据预处理也使用同一批 CPU，需为 DataLoader 和系统进程预留部分核心。 kt_threadpool_count 与 NUMA 拓扑由系统生成；只有确认 CPU socket 和 NUMA 划分后才手动修改。 （6）断点续训： resume_from_checkpoint 在训练 YAML 中设置，值指向需要恢复的 checkpoint-* 目录；新训练不填写。 output_dir 应继续指向本次任务的输出目录，不得与 base model 目录重叠，并保留 overwrite_output_dir: false ，避免覆盖已有结果。 下表统一采用 2K context（sequence length = 2048）、每卡 batch size 1、梯度累积 1，并只统计完成真实 LoRA 训练的 KTransformers BF16 结果。吞吐量包含 forward、loss、backward 和 optimizer。 显存列取该次运行中占用最高的单卡峰值；系统内存列取训练进程树 CPU RSS 峰值，并统一换算为 GiB。它们是已测配置的最低容量参考，不是安全余量：正式训练还要为数据加载、缓存、保存 checkpoint 和系统进程预留空间。 不同 GPU 数的结果不能直接用来判断线性扩展效率。增大 cutoff_len 、batch size 或 KT cache 深度都会增加资源占用；全量微调还需要额外容纳梯度、master weight、optimizer state 与完整 checkpoint。 A.1 Qwen3.5-397B BF16 LoRA 训练 YAML 保存为 qwen35_397b_bf16_lora.yaml ： A.2 FSDP2 Accelerate YAML 保存为 qwen35_fsdp2_2gpu.yaml ： A.3.1 原生 FP8 为什么不能选择 INT8 转换权重配置 两者的权重格式和加载方式不同。原生 FP8 保留 checkpoint 中的 FP8 权重与 scale，使用 AVX512 原精度后端；INT8 配方读取相互匹配的 Routed Expert INT8 权重与 BF16 non-expert cache，并由 auto 选择 AMX-INT8 或 AVX512-VNNI 实现。不要把原生 FP8 checkpoint 填入 kt_weight_path 。 A.3.2 auto 没有选择预期后端 保存 lscpu 输出和启动日志，确认容器或虚拟化环境没有屏蔽指令集。原生 BF16 可在支持 AMX 时选择 AMXBF16；原生 FP8 应选择 AVX512。不要强制指定本机或权重格式不支持的后端。 A.3.3 CPU 系统内存不足 优先降低并发数据加载、序列长度和 cache 深度，并确认没有重复加载两份 Expert 权重。若仍不足，再考虑准备 INT8 / AMXINT4 转换权重。全量微调还要单独估算梯度、master weight 与 optimizer state。 OOM（Out of Memory）表示显存或系统内存不足。先降低 cutoff_len 、每卡 batch size 和 activation 占用，再增加 gradient_accumulation_steps 维持全局 batch size。 num_processes 、可见 GPU 数与模型实际放置必须一致。 A.3.5 kt_model_max_length 与 cutoff_len 不一致 增大 cutoff_len 时同步提高 kt_model_max_length ，并按分布式运行时的有效序列长度留出余量。值过小会导致 buffer 或 shape 错误，过大会增加系统内存占用。 Last updated: 2026-06-23 This guide documents the current KT-FT loop for Qwen3.5 MoE: train with KT SFT, convert the output once, and serve the fine-tuned result through SGLang with a single merged adapter path. Training-side KT SFT docs remain separate. This page focuses on the bridge from trained LoRA artifacts to online inference. Current supported and validated workflow: Base model: Qwen3.5 MoE, for example Qwen3.5-35B-A3B KTransformers version: v0.6.3 or newer KT expert weights: AMX/BF16 SFT-compatible KT CPU expert path User-facing serving input: one converted merged adapter directory Runtime split: expert LoRA goes to the KT CPU expert path; non-expert LoRA goes to SGLang’s LoRA manager. This split happens automatically at server startup. This workflow is for KT MoE expert LoRA artifacts. Standard dense-model PEFT LoRA adapters usually do not need this converter. 2. Artifacts At Each Stage Raw KT SFT output After LLaMA-Factory + KT training, the output directory contains two LoRA artifacts: Do not pass this raw directory directly to SGLang serving. Converted merged adapter Run the converter once to produce the serving input: This merged directory contains both expert and non-expert LoRA tensors in one PEFT-style adapter. Pass only this directory to --lora-paths . 3. Convert Once The converter reads fused_expert_lora.safetensors and the existing non-expert adapter_model.safetensors , then writes one merged adapter directory. If the raw KT SFT output does not contain an adapter_config.json with lora_alpha , pass --lora-alpha <value> explicitly. The converter does not fold LoRA scaling into the tensors; runtime scaling remains lora_alpha / r . Optional split outputs for debugging: For normal serving, only <MERGED_ADAPTER_DIR> is needed. 4. Launch SGLang Use the KTransformers SGLang fork from this repository and point PYTHONPATH at both kt-kernel/python and third_party/sglang/python . Important points: Pass only one merged adapter through --lora-paths . Do not also pass --kt-expert-lora-path in the normal user workflow. At startup, the server detects the merged KT MoE adapter, splits it internally, and writes runtime cache directories under $TMPDIR/sglang_kt_lora_cache/ (or $SGLANG_KT_LORA_CACHE_DIR if set). Prefer --lora-backend triton for Qwen3.5 full-LoRA generation. Current constraints: single merged KT composite adapter only --kt-num-gpu-experts 0 do not enable --kt-enable-dynamic-expert-update do not use --kt-gpu-prefill-token-threshold --max-running-requests must be at least 2 use an AMX/BF16 SFT-compatible KT method such as AMXINT4 , AMXINT8 , AMXBF16 , or BF16 5. Request Semantics The OpenAI-compatible request model field uses names, not paths. Request behavior in the current single-adapter implementation: The suffix after : must match the left-side name in --lora-paths . If you need a true base-only comparison, launch a separate server without --lora-paths . Startup logs should include lines similar to: 7. Advanced: Manual Split Serving The older split-runtime contract is still available for debugging: This is not the recommended user-facing path. Normal users should pass one merged adapter directory through --lora-paths only. 8. Troubleshooting Got LoRA adapter that has never been loaded: lora0 The adapter name in the request must match the left side of --lora-paths . If you launched with qwen35b_lora=... , request model=qwen3.5-kt-ft:qwen35b_lora , not :lora0 . No visible adapter effect Make sure you are serving the converted merged adapter directory produced by the converter, not the raw KT SFT output directory or a different test adapter. connection refused Check that the server is listening on the port you curl, and remember the example above binds to 127.0.0.1 , not 0.0.0.0 . Server resolves upstream SGLang instead of this checkout The path should come from this repository’s third_party/sglang . 最后更新：2026-06-23 本文档描述当前 Qwen3.5 MoE 的 KT-FT 闭环：用 KT SFT 完成微调，转换一次输出，再通过 SGLang 用单个 merged adapter path 把微调结果服务化。 训练侧 KT SFT 文档仍然独立维护；本文重点说明从已训练 LoRA artifacts 到在线推理的连接部分。 基座模型：Qwen3.5 MoE，例如 Qwen3.5-35B-A3B KTransformers 版本：v0.6.3 或更新版本 KT expert 权重：AMX/BF16 SFT 兼容的 KT CPU expert 路径 用户侧 serving 输入：一个 converted merged adapter 目录 Runtime 内部仍会 split：expert LoRA 走 KT CPU expert path，non-expert LoRA 走 SGLang LoRA manager，但这一步对用户不可见 该工作流面向 KT MoE expert LoRA 产物；普通 dense 模型的标准 PEFT LoRA 通常不需要使用此转换器（converter）。 LLaMA-Factory + KT 训练完成后，输出目录里有两个 LoRA 文件： 不要把 raw 训练目录直接传给 SGLang serving。 Convert 后的 merged adapter converter 一次性生成 serving 输入： 这个 merged 目录同时包含 expert 和 non-expert LoRA。正常 serving 只需要传这一个目录。 converter 会读取 fused_expert_lora.safetensors 和已有的 non-expert adapter_model.safetensors ，写出一个 merged adapter 目录。 如果原始 KT SFT 输出目录没有包含带 lora_alpha 的 adapter_config.json ，需要显式传入 --lora-alpha <value> 。converter 不会把 LoRA scaling 折进 tensor；运行时 scaling 仍然是 lora_alpha / r 。 如需调试，也可以额外输出 split 目录： 正常 serving 只需要 <MERGED_ADAPTER_DIR> 。 请使用本仓库的 KTransformers SGLang fork，并把 PYTHONPATH 指向 kt-kernel/python 和 third_party/sglang/python 。 用户只需要传一个 merged adapter： --lora-paths <name>=<MERGED_ADAPTER_DIR> 正常 workflow 不要再额外传 --kt-expert-lora-path server 启动时会自动识别 merged KT MoE adapter，并在 $TMPDIR/sglang_kt_lora_cache/ （或 $SGLANG_KT_LORA_CACHE_DIR ）下生成 runtime cache Qwen3.5 full LoRA 生成优先使用 --lora-backend triton 只支持单个 merged KT composite adapter --kt-num-gpu-experts 0 不启用 --kt-enable-dynamic-expert-update 不使用 --kt-gpu-prefill-token-threshold --max-running-requests 必须至少为 2 使用 AMX/BF16 SFT 兼容 KT method，例如 AMXINT4 、 AMXINT8 、 AMXBF16 、 BF16 OpenAI-compatible 请求里的 model 字段用 name，不用 path。 当前 single-adapter 实现的请求语义： 冒号后的 adapter 名必须和 --lora-paths 左侧注册名一致。 如果需要 true base-only 对照，请单独启动一个不带 --lora-paths 的服务。 7. 高级：手动 split serving 旧 split runtime 仍可用于调试： 这不是推荐的用户路径。正常用户只需要通过 --lora-paths 传一个 merged adapter 目录。 8. Troubleshooting Got LoRA adapter that has never been loaded: lora0 请求里的 adapter 名必须和 --lora-paths 左侧一致。如果启动时写的是 qwen35b_lora=... ，请求应使用 model=qwen3.5-kt-ft:qwen35b_lora ，而不是 :lora0 。 确认 serving 使用的是 converter 生成的 merged adapter 目录，而不是原始 KT SFT 输出目录或其他测试 adapter。 connection refused 确认 server 监听的端口与 curl 一致；上面的示例绑定的是 127.0.0.1 ，不是 0.0.0.0 。 Server 解析到了上游 SGLang，而不是当前 checkout 路径应来自本仓库的 third_party/sglang 。 Author: Azure-Tang This tutorial will guide you through the process of injecting custom operators into a model using the KTransformers framework. We will use the DeepSeekV2-Chat model as an example to demonstrate how to inject custom operators into the model step by step. The tutorial will cover the following topics: How to Write Injection Rules Understanding Model Structure Matrix Absorption-based MLA Injection Injection of Routed Experts Injection of Linear Layers Injection of Modules with Pre-calculated Buffers Specifying Running Devices for Modules How to Write a New Operator and Inject into the Model How to Write Injection Rules The basic form of the injection rules for the Inject framework is as follows: match: This field marks the matching rules, which can appear in two forms, name and class. These two matching rules can appear together or separately; they only match when both criteria are met. replace: class: Python class that can be imported to replace the target module. If no replacement is desired, set to default. kwargs: List of parameters needed for module initialization. generate_device: The device for this module, can be set to “cpu”, “cuda”, “cuda:1”, etc. class: Python class that can be imported to replace the target module. If no replacement is desired, set to default. kwargs: List of parameters needed for module initialization. generate_device: The device for this module, can be set to “cpu”, “cuda”, “cuda:1”, etc. generate_device: The device for this module, can be set to “cpu”, “cuda”, “cuda:1”, etc. recursive: Whether to recursively inject this module’s submodules, default is True. For the recursive field: Some modules contain multiple submodules, such as the Self-attention module typically includes q/k/v/o four linear modules. If we replace the self-attention module but do not want the internal linear modules to be covered by other rules, set this rule to False. Understanding Model Structure Using deepseek-ai/DeepSeek-V2-Lite-Chat as an example, we can follow the above rules step by step to inject our custom module and run it. KTransformers offers a high degree of flexibility, allowing you to replace/experiment with basic operators. However, it also requires users to clearly understand the structure of the model they are running. Fortunately, knowing the structure of a model is very simple. Open the file list on the deepseek-ai/DeepSeek-V2-Lite homepage, and you can see the following files: From the .saftensors file, we can see the name of each layer’s weights, corresponding to the match.name attribute in the injection rules. From the modeling_deepseek.py file, we can see the specific implementation of each module class, with the class name corresponding to the match.class attribute in the injection rules. The structure of the DeepSeekV2 model from the .saftensors and modeling_deepseek.py files is as follows: Supported operators and their corresponding classes are as follows: Then we start step-by-step injection of custom modules, our targets are: Replace the linear module with custom Marlin linear module. Replace the self-attention module with a custom Absorption-based MLA module. Replace the experts module with a custom Experts module. Replace the MoE module with a custom MoE module. Replace the RoPE module with a custom RoPE module. Set the running device for each module. The full implementation of the injection rules can be found in the here . Matrix Absorption-based MLA Injection For the injection of the Attention module, we only need to use a regular expression to match the module names used in transformers and replace them with our own MLA module implementation. The YAML injection rule is as follows: As you can see, each rule in the YAML file has two parts: match and replace. The match part specifies the module to be replaced, and the replace part specifies the module to be injected into the model along with the initialization keywords. Injection of Routed Experts For Routed Experts (corresponding to the exps in the diagram), the module we inject is CPUInfer, which is wrapped in the wrapper module KTransformersExperts. KTransformersExperts has multiple implementations, and we need to specify keywords to tell the wrapper module which implementation we want to use and how we plan to use it. In the source code of the transformer, MoE is implemented using nn.ModuleList. We do not want KTransformers to traverse all submodules in the list and inject them one by one, so in this rule, we set recursive: False to prevent recursive injection into the submodules of this module. The YAML rule is as follows: If we inject Routed Experts as a custom module, we cannot use the interfaces in the original nn.ModuleList . Therefore, it is necessary to modify the forward function in the FFN module. The simplest method is to implement a new module with a custom forward function and inject it. Injection of Linear Layers For the remaining linear layer modules, we aim to use quantized operators to save storage space while improving performance. Since there is no current research on using MLA and quantization together, we do not want to inject linear into the MLA operator. Therefore, we can modify the regular expression and add a type check in the match part of the rule. Only modules that match both the name and class simultaneously will be injected. We also need to pass some keywords similar to the injection of Routed Experts. The YAML rule is as follows: Injection of Modules with Pre-calculated Buffers To avoid occupying resources when initializing the injected original model, we use torch’s meta device to initialize the original model. The RoPE module pre-calculates some buffers during initialization, but no calculations are performed when using the meta device. Therefore, we need to compensate for the calculation of the buffer when loading the model. Simply, we inject a custom module into the rotary embedding module, which performs pre-calculation during loading. The YAML rule is as follows: Specifying Running Devices for Modules Finally, we set a fallback basic attribute generate_device for all modules: Through these two rules, we place all previously unmatched layers (and their submodules) and lm_head on cuda, and the embedding on cpu. Note that the properties of a module will be determined by the first rule it matches. For example, if you later set a new replace.kwargs.generate_device in an injected module, the device set earlier will take precedence. If your computer has multiple cards, you can also configure the model to multiple cards. If you have multiple GPUs, you can set the device for each module to different GPUs. DeepseekV2-Chat got 60 layers, if we got 2 GPUs, we can allocate 30 layers to each GPU. Complete multi GPU rule examples here . First of all, for multi-GPU, we have to inject an new operator KDeepseekV2Model . And set division of the layers to different GPUs. For our case, we have to set the transfer_map in the KDeepseekV2Model operatoras as follows: And we have to set the device for each module in the model. For example, for routed experts , the yaml for one GPU is: But for two GPUs, we need to set the device for each module in the model. For other modules, we can set the device in the same way. How to Write a New Operator and Inject into the Model In this section, we will explain how to write an operator that can be injected, using the implementation of a new linear as an example. First, all injectable operators need to inherit from the BaseInjectedModule class, which inherits some attributes required by our injection framework. Its initialization function needs to meet the following basic format: If users have other parameters that need to be passed to this class, they can also be included in the init function and re-passed in the kwargs parameter in the yaml file. For example, if our operator wants to pass a parameter my_param , the init function can be written as: Then our injection rule can be written as: For the linear module, it is also necessary to read weights from a gguf file. We provide the KLinearBase class to help users read weights from gguf files. Users only need to inherit and implement the load, unload, and forward functions. Therefore, a fully injectable linear class would look like this: Note that the self.load_weight function is provided by the KLinearBase class to help users load weights from a gguf file into the module. The implementation details of KLinearBase can be found on GITHUB . Overall View of the KT Fine-Tuning Framework Attention (LoRA + KT coexist) MoE (operator encapsulation + backward) Multi-GPU Loading/Training: Placement strategy instead of DataParallel Attention (LoRA + KT coexist) MoE (operator encapsulation + backward) Multi-GPU Loading/Training: Placement strategy instead of DataParallel KT-LoRA Fine-Tuning Evaluation Setup Results Speed Tests Memory Footprint Memory Footprint MadSys Lab, KVCache-AI Team, Approaching AI, LLaMA-Factory Team Recent open-source LLMs—from DeepSeek-V3/R1 to Qwen-MoE and Kimi-K2—have surged in performance and scale. Yet due to compute and memory constraints , it is difficult for typical researchers to fine-tune trillion-parameter-class models. We therefore integrate KTransformers with LLaMA-Factory so that, with 2–4 RTX 4090 GPUs and sufficient CPU memory, one can fine-tune ultra-large Mixture-of-Experts (MoE) models such as DeepSeek-671B. This architecture bridges resource gaps, enabling local fine-tuning of ultra-large models , while also supporting efficient scenario customization at 14B/30B scales. We validate on stylized dialogue, Westernized translation tone, and medical Q&A, achieving rapid adaptation within hours. Architecturally, LLaMA-Factory orchestrates data/config/training, LoRA insertion, and inference; KTransformers is a pluggable, high-performance operator backend that takes over Attention and MoE under the same training code, enabling GPU+CPU heterogeneity to accelerate training and reduce GPU memory. We evaluated LoRA fine-tuning with HuggingFace default, Unsloth, and KTransformers backends (same settings and data). KTransformers is currently the only solution feasible on 2–4×24GB 4090s for 671B-scale MoE , and also shows higher throughput and lower GPU memory for 14B MoEs. † The 1400 GB is the theoretical FP16 full-resident footprint (not runnable). 70 GB is the measured peak with KT (Attention on GPU + layered MoE offload). From the table above, it can be seen that for the 14B model, the KTransformers backend achieves approximately 75% higher throughput than the default HuggingFace solution, while using only about one-fifth of the GPU memory. For the 671B model, both HuggingFace and Unsloth fail to run on a single 4090 GPU, whereas KTransformers is able to perform LoRA fine-tuning at 40 tokens/s, keeping the GPU memory usage within 70 GB. Overall View of the KT Fine-Tuning Framework We detail how KTransformers takes over core operators in LLaMA-Factory’s fine-tuning framework to optimize Attention and MoE. DeepSeek-V3/V2 MoE models comprise a small-parameter dense Attention part and a large-parameter sparse MoE part. For illustration, consider layer 2 of DeepSeek-V2-Lite-Chat (from which each layer includes both Attention and MoE). Attention compute and KV cache mainly reside on the GPU; the heavyweight MoE part is primarily executed on the CPU. We first cover Attention replacement and inheritance , then MoE encapsulation and backend interfacing , and finally multi-GPU placement . Attention (LoRA + KT coexist) KTransformers provides operator injection ( BaseInjectedModule ), and PEFT provides LoRA layer insertion. For fine-tuning, we design KTransformersLinearLora , inheriting from both KTransformersLinear and LoraLayer : Inheritance: KTransformersLinearLora retains KT’s high-performance paths ( prefill_linear / generate_linear ) while accepting LoRA parameters ( lora_A/lora_B ). Replacement: During preparation, we replace original KTransformersLinear layers (Q/K/V/O) with KTransformersLinearLora , preserving KT optimizations while enabling LoRA trainability. After replacement, LoRA is inserted at Q/K/V/O linear transforms (left), and KTransformersLinearLora contains both KT fast paths and LoRA matrices (right). MoE (operator encapsulation + backward) Given large parameters and sparse compute, we encapsulate the expert computation as a differentiable black-box operator —transparent upstream, replaceable downstream. Upstream (PyTorch graph): we register a custom Autograd Function so the MoE layer appears as a single node . In the left figure (red box), only KSFTExpertsCPU is visible; on the right, the unencapsulated graph expands routing, dispatch, and FFN experts. Encapsulation makes the MoE layer behave like a standard nn.Module with gradients. Downstream (backend): inside the Autograd Function, pybind11 calls C++ extensions for forward/backward. Multiple pluggable backends exist (AMX BF16/INT8; llamafile ). The backend can be switched via YAML (e.g., "backend": "AMXBF16" vs. "llamafile" ). MoE backward frequently needs the transposed weights $W^\top$. To avoid repeated runtime transposes, we precompute/cache $W^\top$ at load time (blue box). We also cache necessary intermediate activations (e.g., expert projections, red box) to reuse in backward and reduce recomputation. We provide backward implementations for llamafile and AMX (INT8/BF16) , with NUMA-aware optimizations. Multi-GPU Loading/Training: Placement strategy instead of DataParallel To lower per-GPU memory peaks on 2–4 GPUs, we use model parallelism + explicit placement , not DataParallel (which duplicates the whole model on each GPU). KTrainer: takes over .to(device) to prevent “move whole model to a single GPU”. Using KT’s optimize-rule YAML, each layer declares device: cuda:0/cuda:1/... and is constructed directly on the target GPU (no extra copies). Disable automatic DataParallel: when USE_KT=1 , we disable automatic DP wrappers from LLaMA-Factory/HF Trainer to avoid duplication and keep full control over sharding. Gradient aggregation: gradients are reduced to cuda:0 . Intermediate activations stay local; only necessary tensors are transferred, cutting communication/activation overhead. Thus, we keep KT placement strategies under multi-GPU fine-tuning. Users choose a kt_optimize_rule with multi-gpu . For DeepSeek-671B, DeepSeek-V3-Chat-sft-amx-multi-gpu.yaml is a typical 2-GPU plan: KV/attention parts on each GPU; MoE experts sharded on CPU; both GPUs share the workload. KT-LoRA Fine-Tuning Evaluation LLaMA-Factory orchestration, KTransformers backend, LoRA (rank=8, α=32, dropout=0.1, BF16), GAS=16 , qlen=512 , with the same KT optimize rule as training. We evaluate (a) stylized dialogue transfer and (b) two small-scale representative benchmarks: Translational-Style (generative) and AfriMed-QA (medical vertical; SAQ and MCQ ). AMX is enabled; GPUs: 2×48GB RTX 4090; CPU: Intel Xeon Platinum 8488C. Stylized Dialogue (CatGirl tone) Dataset: NekoQA-10K . The fine-tuned model consistently exhibits the target style (red boxes) versus neutral/rational base (blue). This shows KT-LoRA injects style features into the generation distribution with low GPU cost. Translational-Style benchmark (generative) Dataset: Translational-Style-ChatLLM . Metrics: BLEU-1/2/3/4, ROUGE-1/2/L. As shown by the test results in the tables above, under a unified workflow and placement strategy, both model scales exhibit consistent gains after fine-tuning , supporting the usability and effectiveness of the “KT backend + LoRA fine-tuning” combination for generative style control. At the same time, this indicates that KT’s heterogeneous placement and operator optimizations can stably support small-sample adaptation in the style domain. Medical Vertical Benchmark (AfriMed-SAQ/MCQ) The dataset adopts AfriMed-QA (ACL 2025), a domain-specific dataset for the medical field in Africa with strong scenario customization characteristics, comprising two formats—multiple-choice questions (MCQ) and short-answer questions (SAQ)—which in this case serve as the evaluation for vertical-domain fine-tuning. In terms of evaluation criteria, BLEU/ROUGE are used for SAQ, and Accuracy is used for MCQ. As shown in the tables above, (1) DeepSeek-V3 (671B) after KT-LoRA fine-tuning achieves clearly higher performance than the fine-tuned DeepSeek-V2-Lite (14B) on both MCQ and SAQ, and it also surpasses the V3 base model. Within our small-scale setting, this preliminarily indicates that KT-LoRA fine-tuning of ultra-large-parameter models has practical significance in vertical domains. (2) Across both SAQ/MCQ sub-tasks, KT-LoRA delivers consistent gains, indicating that—with KT’s heterogeneous placement and backend operator support—LoRA fine-tuning can effectively inject the key knowledge points of vertical domains such as medicine into the model. At present, most of our testing is conducted on single datasets and at small scale (≤ 20k examples), with the goal of providing existence evidence of system effectiveness for KT-LoRA fine-tuning , rather than drawing generalized conclusions about algorithmic generalization or scaling laws. Our report primarily presents representative figures; to support stronger algorithmic claims, larger sample sizes, multi-lingual/multi-domain datasets, and multi-seed repeated experiments would be required—these are beyond the scope of this work. We also warmly welcome everyone to join the open-source LLaMA-Factory KT fine-tuning project. If you have additional test results, we especially welcome you to record them in the shared spreadsheet below, and to include the corresponding kt_optimize_rule files, dataset examples, training/evaluation YAMLs, and detailed GPU-memory and CPU configurations for community reference and reproducibility~! End-to-End Performance step_time ：time per optimization step (tensor movement + Attention + MoE + others). tokens_per_step = GAS × qlen ； token/s = tokens_per_step / step_time 。 We use GAS=16 , qlen=512 → tokens_per_step=8192 . MoE Compute (DeepSeek-V3-671B) MoE per-layer, per-token FLOPs (forward+backward) approx.: $$ \text{FLOPs}_{\text{per-layer, per-token}} \approx c \cdot k \cdot H \cdot I $$ ​ with $k = 8$（Top-k），$H = 7168$（hidden size），$I = 2048$（intermediate size），$c\approx16$（≈6 forward + ≈10 backward matmuls）。 Per-step across all MoE layers: $$ \text{FLOPs} {\text{per-step}} \approx c \cdot qlen \cdot k \cdot H \cdot I \cdot L {\text{MoE}} $$ ​ Plugging $c=16, qlen=512, k=8, H=7168, I=2048, L_{MoE}=58$，$\text{FLOPs}_{\text{per-step}} \approx 55.8\ \text{TFLOPs}$. Measured (MoE TFLOPS on CPU) If the MoE-only time per step is t_moe (seconds), $\text{TFLOPS} = \text{FLOPs}_{\text{per-step}} / \text{step_per_second}.$ Use MoE-phase time, not full step_time , to get MoE throughput. Memory Footprint DeepSeek-V3 (671B; 58 MoE layers out of 61): ~ 70 GB total GPU, ~ 1.2–1.3 TB host memory. DeepSeek-V2-Lite (14B; 26 MoE layers out of 27): ~ 5 GB GPU, ~ 30 GB host memory. Integrating KTransformers LoRA with LLaMA-Factory provides a practical path to efficiently train and deploy MoE LLMs. KT contributes placement strategies and operator optimizations (DeepSeek/Qwen/Kimi support with AMX-accelerated kernels), and LoRA enables customization with very low GPU memory; LLaMA-Factory supplies a coherent user-level interface. This means even tens-to-hundreds-of-billion-parameter MoE models can be fine-tuned and served with low latency on ordinary hardware. The approach balances memory savings , speed , and usability , turning ultra-large models into tools that developers can actually wield. This tutorial demonstrates how to use Direct Preference Optimization (DPO) to fine-tune a language model using the LLaMA-Factory framework. DPO is a method for training models based on human preferences, allowing for more aligned and user-centric outputs. Step 1: Create a conda environment and suit it for KTransformers Step 2: Install the LLaMA-Factory environment Step 3: Install KTransformers Option 1: Install the KTransformers wheel that matches your Torch and Python versions, from https://github.com/kvcache-ai/ktransformers/releases/tag/v0.4.4 (Note: The CUDA version can differ from that in the wheel filename.) Option 2: Install KTransformers from source Step 4: Install the Flash-attention wheel that matches your Torch and Python versions, from: https://github.com/Dao-AILab/flash-attention/releases Step 5: (Optional) If you want to use flash_infer (otherwise it defaults to triton) We use deepseek-ai/DeepSeek-V2-Lite as an example here. You can replace it with other models such as Kimi K2. For example, we provide the YAML file as follows: （1）examples/train_lora/deepseek2_lora_dpo_kt.yaml For more details about –kt_optimize_rule, please refer to https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/KTransformers-Fine-Tuning_User-Guide.md Then, you can use the lora adapter saved in saves/Kllama_deepseekV2_DPO for inference the same as the sft training. For example, ⚠️ Note: This feature is currently under active development. Many functionalities are not yet complete and are being improved. Please stay tuned for updates. Design Philosophy KT-CLI is designed to minimize the burden of reading documentation . Instead of requiring users to read lengthy docs, the CLI provides: Interactive Mode : Run commands without arguments to get step-by-step guided prompts Direct Mode : Pass arguments directly for automation and scripting 💡 Tip: The arguments are fully compatible with the previous SGLang + KTransformers approach, so you can migrate seamlessly. 💡 Tip: The arguments are fully compatible with the previous SGLang + KTransformers approach, so you can migrate seamlessly. Simply run a command, and the CLI will interactively guide you through the process! You can check the usage by kt --help KTransformers CLI - A unified command-line interface for KTransformers. 1. 1M Context and Native Multimodality GLM-5.3-flash natively supports a context window of up to 1M tokens . A single request can process a large codebase, a long document, or a long-running agent task without repeatedly splitting the context. The model also supports images, video, reasoning, and tool calling , and can be used directly by coding agents through an OpenAI-compatible API. GLM-5.3-flash has approximately 321B parameters. Its 45 layers comprise 34 Linear Attention layers and 11 DSA layers. 2. Native-Precision KTransformers Support KTransformers (KT) reads the official GLM-5.3-flash FP8 weights directly. No model conversion or additional quantization of expert weights is required. The FP8 model occupies approximately 306 GiB. Reserve at least 350 GB of available system memory. The current implementation supports: NVIDIA SM89 and SM120 GPUs (RTX 40 and 50 series) The AVX-512 FP8 CPU expert kernel Heterogeneous CPU-GPU expert inference and Layerwise Prefill A context window of up to 1M tokens Multimodality: text, multiple images, video 3. Installation Use a clean Python 3.11 environment and run: This command installs compatible versions of KT Kernel, SGLang-KT, and Transformers-KT. Replace /path/to/GLM-5.3-flash with the model directory. The configurations below enable Layerwise Prefill, multimodal input, and decode CUDA Graphs by default. The model supports up to 1M context; the examples use the validated 501025 -token configuration. 4.1 Four-GPU Launch 4.2 Single-GPU Launch When Layerwise Prefill is enabled for GLM-5.3-flash, the current implementation normalizes the resident GPU expert count to zero. The --kt-num-gpu-experts 40 setting in the four-GPU example therefore does not keep 40 experts resident during Layerwise Prefill. The server listens on http://localhost:30000 by default. Check the endpoint after startup: The OpenAI-compatible endpoint is: 5. Multimodal Request Boundaries A request may contain text only, or text with up to eight images. A request may instead contain text with one video. Images and video cannot appear in the same request. Video uses the regular mixed-prefill path rather than Layerwise Prefill. Decode CUDA Graphs remain supported. 1. 1M 长上下文与原生多模态 GLM-5.3-flash 原生支持最高 1M tokens 上下文，可以在一次请求中处理大型代码库、长篇文档和长程 Agent 任务，减少频繁切分上下文带来的信息损失。 模型还原生支持图片、视频、Reasoning 和 Tool Calling，可以直接接入使用 OpenAI 兼容接口的 Coding Agent。GLM-5.3-flash 约 321B 参数；45 层网络中包含 34 层 Linear Attention 和 11 层 DSA。 2. KTransformers 原精度支持 KTransformers（KT）直接读取 GLM-5.3-flash 的官方 FP8 权重，不需要转换模型，也不会对 Expert 权重再次量化。 FP8 模型约占 306 GiB，建议至少预留 350 GB 可用系统内存。当前实现支持： NVIDIA SM89 和 SM120 GPU（RTX 40、50 系列） TP1、TP2、TP4 和 TP8 AVX-512 FP8 CPU Expert Kernel CPU-GPU Expert 异构推理和 Layerwise Prefill 文本、多图、单视频、Reasoning 和 Tool Calling 建议使用全新的 Python 3.11 环境，然后执行： 该命令会自动安装匹配的 KT Kernel、SGLang-KT 和 Transformers-KT，无需克隆源码仓库。 将 /path/to/GLM-5.3-flash 替换为模型目录。下面的配置默认开启 Layerwise Prefill、多模态和 Decode CUDA Graph。模型支持最高 1M 上下文；示例使用已经过验收的 501025 tokens 配置。 当 GLM-5.3-flash 开启 Layerwise Prefill 时，当前实现会将 resident GPU experts 归一为 0；因此四卡示例中的 --kt-num-gpu-experts 40 不会在 Layerwise 阶段常驻 40 个 GPU experts。 服务默认监听 http://localhost:30000 。启动完成后可检查接口： 单个请求可以包含纯文本，或纯文本加最多 8 张图片。 单个请求也可以包含纯文本加 1 个视频。 图片和视频不能出现在同一个请求中。 视频使用普通混合预填充，不走 Layerwise Prefill；解码仍支持 CUDA Graph。 This tutorial explains how to run KTransformers on machines that only support AVX2 (without AVX512 or AMX). Table of Contents Supported Precision Formats Hardware Requirements Starting the Inference Server Example: Qwen3-30B-A3B (BF16) Example: Qwen3.5-35B-A3B-FP8 (FP8) Example: Qwen3-30B-A3B-GPTQ-Int4 (GPTQ_INT4) Example: Kimi-K2.5 (RAWINT4) Sending Requests Example: Qwen3-30B-A3B (BF16) Example: Qwen3.5-35B-A3B-FP8 (FP8) Example: Qwen3-30B-A3B-GPTQ-Int4 (GPTQ_INT4) Example: Kimi-K2.5 (RAWINT4) Sending Requests Performance Tuning Supported Precision Formats Hardware Requirements CPU : x86-64 + AVX2 + FMA (Intel Haswell 2013+ / AMD Zen+) GPU : NVIDIA 24GB+ VRAM (RTX 3090/4090/5090, etc.) Memory : At least the size of the model weights (e.g., Qwen3-30B-A3B BF16 requires 64GB+) Build and install from source (one-click install for kt-kernel + SGLang): On AVX512 or AMX machines, you can also manually force AVX2 compilation: Starting the Inference Server Use --kt-method BF16 , FP8 , GPTQ_INT4 , or RAWINT4 . KT-Kernel will automatically detect the CPU and fall back to the AVX2 backend when AVX512/AMX is unavailable. Example: Qwen3-30B-A3B (BF16) Example: Qwen3.5-35B-A3B-FP8 (FP8) Example: Qwen3-30B-A3B-GPTQ-Int4 (GPTQ_INT4) Example: Kimi-K2.5 (RAWINT4) Note : The following command is optimized for 4x RTX PRO 6000 Blackwell (96GB each) + AMD Threadripper PRO 5995WX (64 cores, 1 NUMA node) + 256GB RAM. Sending Requests Performance Tuning --kt-cpuinfer : set to the number of physical cores --kt-threadpool-count : set to the number of NUMA nodes --kt-num-gpu-experts : higher values reduce CPU load but increase GPU VRAM usage Memory bandwidth is often the bottleneck; high-frequency DDR5 memory helps significantly Reduce --kt-num-gpu-experts , --chunked-prefill-size , --max-total-tokens Lower --mem-fraction-static For more questions, see FAQ . 本教程介绍如何在仅支持 AVX2 的机器上运行 KTransformers（无需 AVX512 或 AMX）。 启动推理服务 示例：Qwen3-30B-A3B (BF16) 示例：Qwen3.5-35B-A3B-FP8 (FP8) 示例：Qwen3-30B-A3B-GPTQ-Int4 (GPTQ_INT4) 示例：Kimi-K2.5 (RAWINT4) 发送请求 示例：Qwen3-30B-A3B (BF16) 示例：Qwen3.5-35B-A3B-FP8 (FP8) 示例：Qwen3-30B-A3B-GPTQ-Int4 (GPTQ_INT4) 示例：Kimi-K2.5 (RAWINT4) CPU ：x86-64 + AVX2 + FMA（Intel Haswell 2013+ / AMD Zen+） GPU ：NVIDIA 24GB+ 显存（RTX 3090/4090/5090 等） 内存 ：不少于模型权重大小（如 Qwen3-30B-A3B BF16 需 64GB+） 从源码编译安装（一键安装 kt-kernel + SGLang）： 在AVX512， AMX机器上， 也可以手动强制 AVX2 编译： 使用 --kt-method BF16 、 FP8 、 GPTQ_INT4 或 RAWINT4 ，KT-Kernel 会 自动检测 CPU 并在缺少 AVX512/AMX 时回退到 AVX2 后端。 示例：Qwen3-30B-A3B (BF16) 示例：Qwen3.5-35B-A3B-FP8 (FP8) 示例：Qwen3-30B-A3B-GPTQ-Int4 (GPTQ_INT4) 示例：Kimi-K2.5 (RAWINT4) 说明 ：以下命令针对 4x RTX PRO 6000 Blackwell（各 96GB）+ AMD Threadripper PRO 5995WX（64 核，1 NUMA 节点）+ 256GB RAM 优化。 --kt-cpuinfer 设为 物理核心数 --kt-threadpool-count 设为 NUMA 节点数 --kt-num-gpu-experts 越大 CPU 负担越小，但 GPU 显存占用越高 内存带宽往往是瓶颈，DDR5 高频内存有明显帮助 减小 --kt-num-gpu-experts 、 --chunked-prefill-size 、 --max-total-tokens 降低 --mem-fraction-static 本教程介绍如何使用 KTransformers 的 SYCL_GPTQ_INT4 后端，将 MoE 模型中的对称 GPTQ INT4 experts 放到 Intel 集成显卡（iGPU）上计算。 这是一个异构推理方案：Intel iGPU 负责未放在 CUDA GPU 上的 GPTQ INT4 experts；Attention、Embedding、LM Head 以及通过 --kt-num-gpu-experts 保留的 experts 仍由 CUDA GPU 执行。因此，当前实现不能替代 NVIDIA GPU，也不是纯 Intel GPU 推理后端。 Panther Lake XMX 的最终使用说明见 Intel iGPU XMX 后端 ， 源码构建步骤见 环境安装 。 Linux x86-64 平台。 通过 Intel Level Zero 驱动访问的 Intel GPU，主要面向共享系统内存的 Intel iGPU。 MoE expert 的对称 GPTQ INT4 权重。 sym=true 、 desc_act=false ，权重包含 qweight 和 scales ，不包含 qzeros 。 已重点验证 group_size=128 的模型。 单机、 tensor_parallel_size=1 、以 batch size 1 为主的端侧推理。 不支持把 BF16、FP8 或非对称 GPTQ 权重交给此后端。启动时必须使用： Intel GPU ：支持 Level Zero 和 shared USM 的 Intel iGPU。 CUDA GPU ：一张受当前 PyTorch 和 SGLang 支持的 NVIDIA GPU，用于模型的其余部分。 CPU ：x86-64，支持 AVX2 和 FMA。 系统内存 ：至少能够容纳模型权重、KV Cache 和 iGPU 运行缓冲区。iGPU 会与 CPU 共享系统内存带宽。 Intel oneAPI ：需要包含 icpx 、SYCL runtime 和 sycl-ls 的 oneAPI Base Toolkit。 Intel GPU 驱动 ：需要安装 Level Zero 用户态驱动。 CUDA Toolkit ：当前 SYCL 构建仍需要 CUDA runtime 提供 SGLang 与 CPUInfer 之间的 host callback 调度。 先确认 Intel GPU 能被 SYCL 正常识别： 输出中应当能看到类似下面的 Level Zero GPU： 还需要确认当前用户可以读写 GPU render node： 如果当前用户不在 render 用户组中，可以执行： 修改用户组后需要注销并重新登录。 从源码构建时， CPUINFER_USE_SYCL 默认关闭，需要显式设置为 1 ： 安装脚本会使用 icpx 构建 SYCL GPTQ INT4 后端。普通的 CPU/CUDA 构建不需要 oneAPI；只有启用 CPUINFER_USE_SYCL=1 时才需要上述环境。 SYCL、CUDA、ROCm、MUSA 和 MACA 是互斥的 kt-kernel 编译后端。如果当前 shell 曾经显式设置过其他后端，请先取消对应变量，例如： 未显式设置 CPUINFER_USE_CUDA 时不需要手动取消，安装脚本会在启用 SYCL 后端时自动关闭 CUDA kernel 后端。CUDA Toolkit 仍会用于链接 host callback 所需的 cudart 。 安装完成后，检查 Python 扩展是否包含 SYCL 后端： 如果机器上存在多个 SYCL GPU，可以在启动服务前指定设备： 只有一个可用 Intel GPU 时通常不需要设置该变量，后端会优先选择 Level Zero GPU。 下面使用 Qwen3.5-35B-A3B-GPTQ-Int4 作为示例： 使用其他模型时，请先确认它是 MoE 模型，并且 GPTQ 配置为 sym=true 、 desc_act=false 。目前推荐使用已经验证过的 group_size=128 权重。 下面的命令面向单用户、batch size 1 的端侧推理。请将模型路径和显存相关参数改为适合本机的值： 正常使用时不需要关闭 radix cache。 --disable-radix-cache 主要用于确保每次请求都执行完整 prefill 的性能测试，不建议加入日常启动命令。 服务启动后，可以通过 OpenAI 兼容接口发送请求： SYCL kernel 的 tile、subgroup 和 prefill 阈值已经使用当前验证过的默认值，不需要额外设置调试或调优环境变量。 提示 SYCL_GPTQ_INT4 backend not available 当前安装没有包含 SYCL 扩展。重新加载 oneAPI 环境并构建： 找不到 icpx 或 sycl-ls 确认已经安装 Intel oneAPI Base Toolkit，并在当前 shell 执行： 无法访问 /dev/dri/renderD* 将当前用户加入 render 用户组，注销并重新登录。不要通过长期设置 chmod 666 绕过设备权限。 提示没有可用的 SYCL GPU 先运行 sycl-ls 检查 Level Zero GPU。如果存在多个设备，可以显式选择： CMake 找不到 CUDA Toolkit 当前 SYCL 后端仍需要 cudart 完成 host callback 调度。确认 CUDA Toolkit 已安装；如果不在默认路径，可以设置： 提示多个 GPU backend 同时启用 SYCL 与 kt-kernel 的其他 GPU 编译后端互斥。检查并取消之前显式设置的 CPUINFER_USE_CUDA=1 、 CPUINFER_USE_ROCM=1 等变量。 模型提示 qzeros 、 sym=false 或 desc_act=true 不受支持 当前后端只支持对称 GPTQ INT4： sym=true 、 desc_act=false ，且不使用 qzeros 。需要更换为兼容的模型权重。 降低 --kt-num-gpu-experts 。 降低 --mem-fraction-static 、 --max-total-tokens 或 --chunked-prefill-size 。 降低 --kt-num-gpu-experts 会把更多 expert 计算交给 iGPU，可能降低速度并增加系统内存带宽压力。 确认系统处于高性能电源模式，并避免同时运行占用 iGPU 的桌面或视频任务。 iGPU 与 CPU 共享内存带宽，双通道或更高带宽的内存通常有明显帮助。 首次请求包含权重准备和运行时预热，性能比较应在 warmup 后进行。 1. SGLang “Using default MoE kernel config” warning at startup When using kt-kernel with SGLang, you may see a warning like: This warning is expected and can be safely ignored . kt-kernel replaces SGLang’s built-in MoE implementation with its own CPU/GPU hybrid dispatch, so SGLang’s fused-MoE Triton kernel configuration is never used. The warning is emitted by SGLang before kt-kernel takes over MoE execution and has no impact on performance or correctness. 2. Where can I find more help? Check the existing issues or open a new one .