Keyboard shortcuts Press ← or → to navigate between chapters Press S or / to search in the book Press ? to show this help Press Esc to hide this help High-performance kernel operations for KTransformers, featuring CPU-optimized MoE inference with AMX, AVX, KML and blis (amd library) support. Installation Option 1: Install from PyPI (Recommended for Most Users) Option 2: Install from Source (For Local Use or Custom Builds) Option 1: Install from PyPI (Recommended for Most Users) Option 2: Install from Source (For Local Use or Custom Builds) KT CLI Overview Integration with SGLang Installation Steps Complete Example: Qwen3-30B-A3B KT-Kernel Parameters Installation Steps Complete Example: Qwen3-30B-A3B KT-Kernel Parameters Direct Python API Usage Advanced Options Manual Configuration (Advanced) Advanced Options Manual Configuration (Advanced) Build Configuration Manual Installation (Without install.sh) Manual Installation (Without install.sh) Error Troubleshooting CUDA Not Found hwloc Not Found hwloc Not Found Weight Quantization Current Support Status: ✅ Native Precision with AVX512/AMX : Supported with AVX512 CPUs in FP8 , BF16 and RAWINT4 format - Guide ✅ Intel CPUs with AMX : Fully supported (using weights converted to INT4/INT8 format) ✅ Universal CPU (llamafile backend) : Supported (using GGUF-format weights) ✅ AMD CPUs with BLIS : Supported (for int8 prefill & decode) - Guide We are developing a simpler way to use KTransformers. Check out the KT-CLI Guide for more details. CPU-Optimized MoE Kernels : High-throughput MoE expert kernels optimized for instruction sets. AVX512 Native Precision Backend : FP8 / BF16 / INT4 native MoE backend for AVX512-capable servers. AMX INT4/INT8 Backend : INT4 / INT8 quantized expert inference backend for AMX-capable servers. Llamafile CPU Backend : AVX2/AVX512-based MoE backend built on Llamafile for universal CPU deployment. NUMA-Aware Execution : Thread pool and memory layout designed for multi-socket / multi-NUMA machines. Option 1: Install from PyPI (Recommended for Most Users) Install the latest version with a single command: Note : Check the latest version on PyPI ✅ Automatic CPU detection : Detects your CPU and loads the optimal kernel variant ✅ CPU multi-variant support : Includes AMX, AVX512 (Base/VNNI/VBMI/BF16), and AVX2 variants ✅ CUDA support included : GPU acceleration for NVIDIA GPUs (SM 80, 86, 89, 90) ✅ No compilation needed : Pre-built wheels for Python 3.10, 3.11, 3.12 ✅ Static CUDA runtime : No CUDA toolkit installation required ✅ Works on CPU-only systems : CUDA features automatically disabled when GPU not available Python 3.10, 3.11, or 3.12 Linux x86-64 (manylinux_2_17 compatible) CPU with AVX2 support (Intel Haswell 2013+, AMD Zen+) Optional: NVIDIA GPU with compute capability 8.0+ for CUDA features CUDA Installation (GPU Acceleration) For NVIDIA GPU-accelerated inference: ✅ Multi-architecture support : Single wheel supports SM 80/86/89/90 (Ampere, Ada, Hopper) ✅ Static CUDA runtime : No CUDA toolkit installation required ✅ Broad compatibility : Works with CUDA 11.8+ and 12.x drivers ✅ PyTorch compatible : Works with any PyTorch CUDA variant (cu118, cu121, cu124) Python 3.10, 3.11, or 3.12 Linux x86-64 (manylinux_2_17 compatible) NVIDIA GPU with compute capability 8.0+ (Ampere or newer) ✅ Supported: A100, RTX 3000/4000 series, H100 ❌ Not supported: V100, P100, GTX 1000/2000 series (too old) ✅ Supported: A100, RTX 3000/4000 series, H100 ❌ Not supported: V100, P100, GTX 1000/2000 series (too old) NVIDIA driver with CUDA 11.8+ or 12.x support (no CUDA toolkit needed) GPU Compatibility Matrix: CUDA Driver Compatibility (for GPU features): CUDA 11.8, 11.9, 12.0-12.6+: Full support CUDA 11.0-11.7: Not supported (upgrade driver or use CPU-only) CPU Variants Included: The wheel includes 6 optimized variants that are automatically selected at runtime based on your CPU: Verify installation: Environment Variables: Option 2: Install from Source (For Local Use or Custom Builds) Build from source for local installation or when you need AMD (BLIS), ARM (KML), or custom CUDA versions. First, initialize git submodules and create a conda environment: Quick Installation (Recommended) Simply run the install script - it will auto-detect your CPU and optimize for best performance: What happens automatically: Auto-detects CPU capabilities (AMX, AVX512_VNNI, AVX512_BF16) Installs system dependencies ( cmake , libhwloc-dev , pkg-config ) Builds optimized binary for your CPU only (using -march=native ) Software fallbacks : Automatically enabled for CPUs without VNNI/BF16 Optional: Two-step installation CPU Requirements by Backend: Software Fallback Support (AVX512 backends): ✅ VNNI fallback: Uses AVX512BW instructions ✅ BF16 fallback: Uses AVX512F instructions ✅ Older AVX512 CPUs (Skylake-X, Cascade Lake) can run RAWINT4 with fallbacks ⚠️ Portability Note: The default build is optimized for your specific CPU and may not work on different/older CPUs. For portable builds or binary distribution, see Manual Configuration below. ⚠️ AMD BLIS backend users: See installation guide for AMD-specific setup. After installation, verify that the CLI is working: Expected output: You can also verify the Python module directly: KT CLI Overview The kt command-line tool provides a unified interface for running and managing KTransformers models: Quick Start Example: Run kt --help for more options, or kt <command> --help for command-specific help. Integration with SGLang KT-Kernel can be used standalone via Direct Python API or integrated with SGLang for production deployment. This section describes SGLang integration to enable CPU-GPU heterogeneous inference, where “hot” experts run on GPU and “cold” experts run on CPU for optimal resource utilization. Installation Steps 1. Install SGLang Install the kvcache-ai fork of SGLang (required for kt-kernel support): Important: Use sglang-kt (kvcache-ai fork), not the official sglang package. If you have the official version installed, uninstall it first: pip uninstall sglang -y 2. Prepare Weights You need both GPU weights and CPU-side expert weights for heterogeneous inference. The exact format depends on the backend: GPU Weights (for all backends): Use the model weights required by SGLang for GPU inference (for example, the original or already-quantized model directory from Hugging Face). CPU Weights (AMX backend: AMXINT4 / AMXINT8 ): Quantize weights to AMX-optimized INT4/INT8 format using the provided script: --input-path : Path to GPU-side original weights --input-type : Depends on your GPU weights type ( fp8 , fp16 , or bf16 ) In SGLang integration, --kt-weight-path should point to this converted CPU weights directory. Supported input formats: FP8, FP16, BF16 → INT4/INT8. CPU Weights (LLAMAFILE backend: LLAMAFILE ): LLAMAFILE uses pre-quantized GGUF weights on the CPU side directly, without running convert_cpu_weights.py . You need to: Download a GGUF model directly from the web (e.g., GGUF repos on Hugging Face / Modelscope); In SGLang integration, use that GGUF directory as --kt-weight-path . KT-Kernel supports multiple GGUF quantization formats such as Q4_KM , Q4_K , Q5_K , etc. Choose based on your latency and accuracy requirements. 3. Launch SGLang Server Start the SGLang server with your normal SGLang parameters, and add the following KT-Kernel specific parameters to enable CPU-GPU heterogeneous inference: KT-Kernel Parameters to Add: --kt-method : Backend method (AMXINT4, AMXINT8, or LLAMAFILE) --kt-weight-path : Path to the converted CPU weights --kt-cpuinfer : Number of CPU inference threads (set to physical cores)

--kt-threadpool-count : Number of thread pools (set to NUMA node count) --kt-num-gpu-experts : Number of experts to keep on GPU --kt-max-deferred-experts-per-token : Deferred experts for pipelined execution See KT-Kernel Parameters section below for detailed parameter tuning guidelines. Complete Example: Qwen3-30B-A3B This example demonstrates the full workflow from downloading weights to launching the server, showing Native backend , AMX backend and LLAMAFILE backend options. Hardware Configuration: GPU : NVIDIA RTX 4090 24GB CPU : 2x Intel Xeon Gold 6454S (64 physical cores total, 128 threads, 2 NUMA nodes) Model : Qwen3-30B-A3B How to verify your system configuration: Parameter Rationale: --kt-cpuinfer 64 : Set to physical cores (64), not hyperthreads (128) --kt-threadpool-count 2 : 2 NUMA nodes detected (dual-socket system) --kt-num-gpu-experts 32 : With 24GB GPU memory, we can fit ~32 experts on GPU for this model (varies by model architecture and actual memory usage) --kt-max-deferred-experts-per-token 2 : Enable pipelined execution; allows CPU to process next batch while GPU completes current batch --kt-gpu-prefill-token-threshold 2048 : Use layerwise prefill strategy when token count exceeds 2048 (for native backends only) Option A: Native Backend (BF16) For AVX512 CPUs with BF16 support. Step 1: Download model weights Step 2: Launch SGLang server Option B: AMX Backend (AMXINT8) For Intel CPUs with AMX instruction set support. Step 1: Download model weights Step 2: Convert to CPU weights (AMXINT8) Step 3: Launch SGLang server Option C: LLAMAFILE Backend (GGUF) For universal CPUs (no AMX required), using pre-quantized GGUF weights directly. Step 1: Download GPU weights (original model) Step 2: Download CPU weights (GGUF format) Step 3: Launch SGLang server KT-Kernel Parameters Parameter Guidelines: kt-method : Choose based on your CPU and weight format: AMXINT4 : Best performance on AMX CPUs with INT4 quantized weights (May cause huge accuracy drop for some models, e.g., Qwen3-30B-A3B) AMXINT8 : Higher accuracy with INT8 quantized weights on AMX CPUs RAWINT4 : Native INT4 weights shared by CPU and GPU (currently supports Kimi-K2-Thinking model). See Kimi-K2-Thinking Native Tutorial for details. FP8 , FP8_PERCHANNEL : FP8 weights shared by CPU and GPU BF16 : BF16 weights shared by CPU and GPU LLAMAFILE : GGUF-based backend kt-method : Choose based on your CPU and weight format: AMXINT4 : Best performance on AMX CPUs with INT4 quantized weights (May cause huge accuracy drop for some models, e.g., Qwen3-30B-A3B) AMXINT8 : Higher accuracy with INT8 quantized weights on AMX CPUs RAWINT4 : Native INT4 weights shared by CPU and GPU (currently supports Kimi-K2-Thinking model). See Kimi-K2-Thinking Native Tutorial for details. FP8 , FP8_PERCHANNEL : FP8 weights shared by CPU and GPU BF16 : BF16 weights shared by CPU and GPU LLAMAFILE : GGUF-based backend kt-cpuinfer : Set to the number of physical CPU cores (not hyperthreads). Check physical cores: lscpu | grep -E "^CPU\(s\)|Thread\(s\) per core" Physical cores = CPU(s) / Thread(s) per core Example: If CPU(s)=128 and Thread(s) per core=2, then physical cores = 64 Important : Do NOT set to hyperthread count - this will degrade performance kt-cpuinfer : Set to the number of physical CPU cores (not hyperthreads). Check physical cores: lscpu | grep -E "^CPU\(s\)|Thread\(s\) per core" Physical cores = CPU(s) / Thread(s) per core Example: If CPU(s)=128 and Thread(s) per core=2, then physical cores = 64 Important : Do NOT set to hyperthread count - this will degrade performance kt-threadpool-count : Set to the number of NUMA nodes . Check NUMA count: lscpu | grep "NUMA node(s)" Or use: numactl --hardware | grep "available" Note : NUMA node count is NOT necessarily the number of physical CPUs It represents memory domains, which may be divided within a single CPU or across multiple CPUs Use the NUMA node count from lscpu , regardless of physical CPU count Typical values: 1-2 for single-socket, 2-4 for dual-socket systems This enables better memory bandwidth utilization across NUMA domains kt-threadpool-count : Set to the number of NUMA nodes . Check NUMA count: lscpu | grep "NUMA node(s)" Or use: numactl --hardware | grep "available" Note : NUMA node count is NOT necessarily the number of physical CPUs It represents memory domains, which may be divided within a single CPU or across multiple CPUs Use the NUMA node count from lscpu , regardless of physical CPU count It represents memory domains, which may be divided within a single CPU or across multiple CPUs Use the NUMA node count from lscpu , regardless of physical CPU count Typical values: 1-2 for single-socket, 2-4 for dual-socket systems This enables better memory bandwidth utilization across NUMA domains kt-num-gpu-experts : Determine based on GPU memory and profiling: More GPU experts = lower latency but higher GPU memory usage (May cause OOM) kt-num-gpu-experts : Determine based on GPU memory and profiling: More GPU experts = lower latency but higher GPU memory usage (May cause OOM) kt-max-deferred-experts-per-token : Enables pipelined execution: 0 : Synchronous execution (simpler, higher latency) 1-4 : Deferred execution (recommended range; good latency/quality balance, requires tuning) 5-7 : Highest latency reduction but may introduce noticeable accuracy loss; use with care kt-max-deferred-experts-per-token : Enables pipelined execution: 0 : Synchronous execution (simpler, higher latency) 1-4 : Deferred execution (recommended range; good latency/quality balance, requires tuning) 5-7 : Highest latency reduction but may introduce noticeable accuracy loss; use with care kt-gpu-prefill-token-threshold (FP8 and RAWINT4 only): Controls prefill strategy for native FP8 and INT4 inference: ≤ threshold : Uses hybrid CPU+GPU prefill. No extra VRAM needed, but performance degrades slowly as token count increases. > threshold : Uses layerwise GPU prefill. Performance scales better with longer sequences, but requires one MoE layer extra VRAM (e.g., ~9GB+ for Kimi-K2-Thinking and ~3.6GB for MiniMax-M2.1). Only applicable when --kt-method RAWINT4 or --kt-method FP8 is used. kt-gpu-prefill-token-threshold (FP8 and RAWINT4 only): Controls prefill strategy for native FP8 and INT4 inference: ≤ threshold : Uses hybrid CPU+GPU prefill. No extra VRAM needed, but performance degrades slowly as token count increases. > threshold : Uses layerwise GPU prefill. Performance scales better with longer sequences, but requires one MoE layer extra VRAM (e.g., ~9GB+ for Kimi-K2-Thinking and ~3.6GB for MiniMax-M2.1). Only applicable when --kt-method RAWINT4 or --kt-method FP8 is used. kt-enable-dynamic-expert-update : Enables dynamic expert placement updates during inference. During layerwise prefill, the system collects actual routing statistics and redistributes GPU experts accordingly. Requires --kt-gpu-prefill-token-threshold to be set, and prefill length must be ≥ the threshold value. Particularly effective at lower GPU expert ratios (10%-70%), where it can significantly outperform static strategies. See Expert Scheduling Tutorial for benchmarks and details. kt-enable-dynamic-expert-update : Enables dynamic expert placement updates during inference. During layerwise prefill, the system collects actual routing statistics and redistributes GPU experts accordingly. Requires --kt-gpu-prefill-token-threshold to be set, and prefill length must be ≥ the threshold value. Particularly effective at lower GPU expert ratios (10%-70%), where it can significantly outperform static strategies. See Expert Scheduling Tutorial for benchmarks and details. kt-expert-placement-strategy : Determines which experts are placed on GPU at server startup. uniform : Distributes GPU experts evenly across all MoE layers. Default option, no prior statistics needed. frequency : Places the most frequently activated experts on GPU. Best performance when activation statistics are available; requires --init-expert-location pointing to a .pt statistics file. front-loading : Fills GPU experts from the first MoE layer onwards. random : Randomly selects experts with a fixed seed (42). See Expert Scheduling Tutorial for strategy comparison. kt-expert-placement-strategy : Determines which experts are placed on GPU at server startup. uniform : Distributes GPU experts evenly across all MoE layers. Default option, no prior statistics needed. frequency : Places the most frequently activated experts on GPU. Best performance when activation statistics are available; requires --init-expert-location pointing to a .pt statistics file. front-loading : Fills GPU experts from the first MoE layer onwards. random : Randomly selects experts with a fixed seed (42). See Expert Scheduling Tutorial for strategy comparison. Direct Python API Usage For standalone usage without SGLang, you can use KT-Kernel directly via Python API: Advanced Options Manual Configuration (Advanced) For portable builds, binary distribution, or cross-machine deployment, you need to manually specify target instruction sets: Optional: Override VNNI/BF16 detection Optional: oneDNN INT8 BRGEMM on non-AMX AVX512 CPUs See ./install.sh --help for all available options. Build Configuration Manual Installation (Without install.sh) If you prefer manual installation without the install.sh script: 1. Install System Dependencies cmake (recommended: conda install -y cmake ) libhwloc-dev and pkg-config 2. Set Build Configuration Instruction Set Details: Example Configurations: 3. Build and Install Error Troubleshooting Make sure you have the CUDA toolkit installed and nvcc is in your system PATH. Try export CMAKE_ARGS="-D CMAKE_CUDA_COMPILER=$(which nvcc)" and reinstall again. hwloc Not Found Run sudo apt install libhwloc-dev if on a Debian-based system or build from source: https://www.open-mpi.org/projects/hwloc/. Weight Quantization For AMX backends ( AMXINT4 / AMXINT8 ), CPU-side experts must be converted to AMX-friendly INT4/INT8 format using the provided script: Supported formats: FP8, FP16, BF16 → INT4/INT8 For LLAMAFILE backend ( LLAMAFILE ), CPU-side experts are loaded directly from GGUF weights. You do not need to run the AMX conversion script; instead, download a GGUF model from the web (e.g., a GGUF repo on Hugging Face) and point weight_path / SGLang --kt-weight-path (or --model when appropriate) to that GGUF directory. KT-Kernel supports multiple GGUF quantization types such as Q4_KM , Q4_K , Q5_K , etc. For detailed documentation, advanced options, and low-memory mode, see scripts/README.md . Commit messages should follow the Conventional Commits specification: https://www.conventionalcommits.org/ Please format your code before committing: You may need a newer clang-format (at least version 18). In a conda environment: It’s also recommended to install black for Python code formatting: