Axolotl v0.21.1 Release Notes

45 commits since v0.21.0 (October 16, 2026).

This release makes long LoRA runs on large MoEs like Beam smoother, and includes a security update for the compromised eMudhra intermediate CAs.

We also added merge-aware LoRA for RL, per-group eval metrics, and full-turn labeling for tool-calling SFT, and fixed several bugs that trained silently on the wrong loss, activation, or gradients.

Highlights

Checkpoint Mirroring and Save Retries

You can now save checkpoints to fast local storage and keep training while they copy to shared storage in the background, and a save that hits a full disk pauses and retries so you can free space.

Set checkpoint_mirror and checkpoint_save_retry. Read the guide.

• Contributed by @winglian in #4184.

Sharded LoRA Adapter Checkpoints

You can now checkpoint FSDP2 LoRA and QLoRA runs without a rank 0 gather, cutting a Beam EP=8 checkpoint stall from about 2:20 to 11.6 s.

Set adapter_checkpoint_format: sharded; examples are in examples/expert_parallel/.

• Contributed by @winglian and @NanoCode012 in #4188.

Heads up: EP checkpoints saved model-only or sharded can no longer be resumed by older Axolotl versions.

Per-Group Eval Metrics

You can now track eval metrics per domain or task, such as eval/math/loss, alongside the aggregate.

Add eval_name to your test_datasets entries. Check the docs.

• Contributed by @winglian in #4177.

Full-Turn Labeling for Tool-Calling SFT

You can now train the whole assistant turn on Harmony-style templates, including the channel headers the default labeling left untrained.

Set generation_tags: true on a chat_template dataset. Tool results with only a tool_call_id also no longer fail to render. Learn how it works.

• Contributed by @winglian and @NanoCode012 in #4178, and @winglian in #4174.

Merge-Aware BF16 LoRA for RL

You can now train LoRA RL on exactly the BF16-rounded weights a merged vLLM sync serves. On Qwen3-0.6B-Base it reached standard LoRA's step-100 MATH-500 accuracy by about step 82.

Set lora_merge_aware: bf16. Merged-sync runs also now log how much of each LoRA update survives the rounding (trl.vllm_merge_check_interval, on by default). Read more.

• Contributed by @winglian in #4216.

Performance & Kernel Optimizations

• SonicMoE autotune (#4198 by @winglian): new routed row counts now reuse a power-of-two bucket's winner by default instead of starting a fresh 40 to 76 candidate sweep mid-run. quack_autotune: false skips sweeps, and autotune_profiles can pin quack configs.
• SonicMoE EP row padding and LoRA split (#4198 by @winglian): opt-in high-water row padding and local-pair compaction for EP, and experimental sonicmoe_lora_split to run expert LoRA as separate low-rank GEMMs.
• FA4 on Blackwell with nvidia-cutlass-dsl>=4.6.2 (#4187 by @winglian): runs no longer fall back silently to FA2, which is about 2 to 2.5x slower on attention at Beam's shapes.
• FA4 attention savable under selective checkpointing (#4191 by @winglian): save: [attention] previously recomputed it anyway.
• Faster NVFP4 LoRA merges (#4175, #4199, #4205 by @winglian): chunked expert merges, reordered tensors with opt-in merge_compile, and batched pinned transfers. A rank-128 merge into Beam is at least 27% faster with 8 GB less peak memory (#4199).

New Features

• Activation offloading without gradient checkpointing, and in GRPO (#4142 by @winglian and @NanoCode012): gradient_checkpointing now decides whether layers recompute, with or without an adapter.
• tree-recurrence package (#4214, #4217 by @winglian): a standalone package (pip install tree-recurrence) that lets TRL's prefix-shared tree packing handle hybrid recurrent models like Qwen3.5 and Nemotron-H. Axolotl does not use it yet.
• Multi-node axolotl vllm-serve (#4212 by @NanoCode012): one vLLM server can span several nodes with vllm.nnodes, vllm.master_addr, and --node-rank. Read the guide.
• label_keep_prob (#4181 by @winglian): randomly drops the labels of chosen token ids during SFT.
• CoreWeave CKS host_paths (#4180 by @winglian): load base_model from node-local NVMe instead of the shared PVC.
• Readable label-balancing logs (#4176 by @winglian): a short before/after summary at INFO.

Documentation

• Corrected check_interval default for dynamic checkpoints (#4147 by @NanoCode012), and fixed example sub-page links (#4167 by @NanoCode012).

Model & Framework Support

🚨 Deprecations

• activation_offloading: legacy is now a deprecated alias of true, and disk (with gradient_checkpointing: offload_disk) is rejected, since it never offloaded to disk (#4142 by @winglian and @NanoCode012).

Dependency Updates

• cut-cross-entropy v0.1.1 → v0.2.0 (#4204 by @NanoCode012): ports Liger's CE kernel fixes, refreshes every patch for transformers 5.20.1, fixes remote-code patching, and adds patches for several models.
• liger-kernel 0.8.4 → 0.8.5 (#4211 by @NanoCode012), replacing Axolotl's custom DeepSeek-V2 patch with Liger's native one.
• certifi >=2026.10.19 and refreshed ca-certificates in the Docker images (#4192 by @NanoCode012). Upgrade, or pull the new images.
• axolotl-ringmaster >=0.2.6 → >=0.2.7 (#4207 by @winglian), fixing an intermittent context-parallel failure when the Hub is unreachable.
• New axolotl[blackwell] extra pinning FA4 and quack (#4207 by @winglian).

Bug Fixes

Heads up: the CCE plugin now raises at load time for models with a logit softcap or scale and no dedicated CCE patch (#4182).

• Activation offloading corrupted gradients (#4142 by @winglian and @NanoCode012): DPO and KTO on the pinned TRL 1.14.1 got wrong gradients or CUDA illegal memory accesses.
• LoRA GRPO's merged vLLM sync drifted the frozen base weights (#4213 by @winglian): each merge/unmerge round trip in BF16 changed some entries (1.5% after 200 syncs at lr 1e-5). The base is now restored exactly.
• lora_fp32_gradients trained in BF16 for RL trainers under FSDP (#4215 by @winglian).
• CCE's generic patch skipped logit softcapping (#4182 by @winglian and @NanoCode012): Beam's step-0 eval loss read about 3.4 instead of 0.79.
• Fused LoRA GEGLU kernel used exact GELU on tanh-GELU models like Gemma (#4189 by @NanoCode012).
• SinkGD under pure expert parallelism crashed or skipped the cross-rank reduce (#4161 by @winglian).
• Saved generation_config now stops on eot_tokens (#4152 by @NanoCode012).
• Null optional dataset fields no longer crash or mis-render across prompt strategies (#4183 by @mottopanikeiku and @NanoCode012, #4196 by @NanoCode012), and field_tools is now honored (re-run preprocess if you set it).
• K2 Horizon turn labeling no longer crashes; every assistant turn needs a thinking field (#4190 by @NanoCode012).
• Non-EP NVFP4 SonicMoE experts with quack 0.6.5 (#4198 by @winglian).
• Gemma 4 NVFP4 MoE on B200 crashed on DeepGEMM's 128-multiple size check and now falls back to Marlin (#4207 by @winglian).
• Smaller fixes (#4147 by @NanoCode012): llama2_chat label alignment, TP model saves, the rl_beta check, and polora with ReLoRA.

Infrastructure

• Faster CI (#4202 by @NanoCode012): coverage's sysmon core halves the wait before GPU tests start, from 52 to 26 minutes.
• Test suite cleanup (#4147 by @NanoCode012), removing about 2,900 lines of stale tests.
• New images: a torch 2.15.0 RC base image (#4208 by @winglian), and base images now ship apache-tvm-ffi, cutlass-dsl, and nvshmem (#4200 by @winglian).
• Blackwell CI (#4201, #4207 by @winglian): a single-B200 lane for SM100-only tests, DeepEP built and tested in CI, and fixes for failing B200 and multi-GPU lanes.
• Agent guidance against prompt injections in review comments (#4206 by @winglian).
• GLM DSA capacity tests now use real nn.Module experts, fixing two kernel CI failures (#4179 by @winglian).
• GitHub Actions bumps (#4171, #4172, #4173 by @dependabot): setup-uv 10.3.0, download-artifact 8.0.2, and upload-artifact 7.0.2.

New Contributors

• @mottopanikeiku made their first contribution in #4183

Full Changelog: v0.21.0...v0.21.1

Axolotl v0.21.0 Release Notes

59 commits since v0.20.0 (September 30, 2026).

This release is headlined by Reflection's Beam, a 501B-parameter MoE you can QLoRA-tune on a single node. We also added native diffusion language model training, image datasets for DPO, KTO, and GRPO, and optimizations for the training loop.

Highlights

Reflection Beam

You can now fine-tune Reflection's Beam, a 501B-parameter MoE with 23B active parameters, using QLoRA on a single 8xH200 node.

QLoRA configs ship for FSDP2 and device_map, plus expert-parallel LoRA recipes for B200 in bf16, NVFP4, and NF4. Get started with the Beam guide.

• Contributed by @HamidShojanazeri and @winglian in #4148, and @NanoCode012 in #4163.

Native Diffusion LM Training

You can now fine-tune diffusion language models (Nemotron-Labs-Diffusion 3B and 8B, Dream v0, and DiffusionGemma) with a top-level diffusion_lm: block and your usual chat datasets, no plugin required.

examples/nemotron-diffusion/lora-smoke.yaml is a 3B LoRA starting point. Read more.

• Contributed by @winglian in #4113.

Typed-Decision Training on Diffusion LMs

You can now train a diffusion LM to fill typed decisions (labels, distributions, label sets, soft scores) at fixed slots, including image-grounded decisions on Nemotron-Labs-Diffusion-VLM-8B.

Enable DiffusionDecisionPlugin and start from the 8B text and VLM recipes in examples/nemotron-diffusion/. Learn how it works.

• Contributed by @winglian in #4117, #4126, #4127, and #4130.

Multimodal DPO, KTO, and GRPO

You can now run DPO, IPO, KTO, GRPO, and GDPO on image datasets with vision-language models such as Qwen, SmolVLM2, and Gemma.

axolotl vllm-serve also now runs vLLM's native OpenAI-compatible server. Check the docs.

• Contributed by @NanoCode012 in #4039, with @ved1beta and @winglian.

Heads up: vllm>=0.30.0 is now required.

Context Parallelism Under DeepSpeed

You can now set context_parallel_size alongside deepspeed:, which maps onto DeepSpeed's Ulysses implementation.

Qwen3.5 and Qwen3-Next gated-delta layers also move to Ringmaster's context-parallel kernel interface and now trace under torch.compile. Read the guide.

• Contributed by @winglian in #4091, and @thad0ctor and @winglian in #4102.

Plugins That Load Themselves

You can now ship an Axolotl plugin or model-support profile as a pip package, registered through the axolotl.plugins or axolotl.model_support entry points, with no plugins: entry needed.

Context and expert parallelism now load this way, so EP configs no longer list a plugin. Learn how to publish one.

• Contributed by @winglian in #4089.

Performance & Kernel Optimizations

• Fewer host-device syncs (#4143 by @winglian and @NanoCode012): packed flash-attention LoRA on Llama-3.1-8B now pays 1 sync per micro-step instead of 390, and Qwen3-8B LoRA trains about 8% faster in eager mode.
• FlashAttention varlen metadata precomputed in the collator for compiled packed runs (#4100 by @thad0ctor and @winglian), now on for every packed run (#4143).
• Faster large-MoE loading (#4151 by @winglian): fused-expert conversion on a 500B-class MoE dropped from about 20 to 6 minutes.
• Kernel autotune caching (#4151 by @winglian): autotune_cache_path, autotune_profiles, and autotune_neighborhood stop EP ranks from re-benchmarking mid-run, and the default search space is smaller (#4116 by @winglian).
• Context-parallel shards cut in the collator (#4144 by @winglian), removing a per-step broadcast.

Heads up: gradient checkpointing now defaults to non-reentrant (except DeepSpeed ZeRO-3), FSDP2 keeps LoRA modules inside their decoder layer's unit (#4143), and packing capacity now follows the effective batch size (#4136).

New Features

• CoreWeave (#4150 by @winglian and @NanoCode012): launch training on SUNK or CKS with axolotl train --cloud, publish checkpoints to Dedicated Inference, and run GRPO rollouts there. Setup guide.
• balance_labels (#4136 by @winglian): balances supervised tokens across batches and optimizer steps, cutting per-step label stddev from 990 to 322 on FineTome.
• Staged NF4 QLoRA under expert parallelism (#4151 by @winglian).
• Fused LoRA kernels for DPO, IPO, and KTO (#4151 by @winglian): previously crashed, they now attach after TRL wraps the adapter.
• dpo_ld_alpha (#4165 by @winglian): LD-DPO length desensitization, so longer responses stop winning pairs on token count alone. Liger DPO also no longer rejects ipo or warns on multiple loss types.
• sync_profiler_steps (#4143 by @winglian and @NanoCode012): counts host syncs per micro-step by call site.
• optimizer: muon_8bit (#3983 by @ved1beta).
• Diffusion decision follow-ups (#4158 by @winglian), including Nemotron-Labs-TwoTower and DiffusionGemma ScatterMoE and NVFP4 paths.
• Beam H100 autotune profiles (#4159 by @winglian).
• Multi-turn synthetic datasets (#4087 by @winglian): min_turns / max_turns shape synthetic rows like chat or agentic traces.
• Node-local expert parallelism (#4091 by @winglian): EP all-to-all stays within a node when ep × tp fits, and routed-expert LoRA can resume on an EP-sliced model.
• full_determinism (#4110 by @winglian), with model initialization now seeded from seed.
• cuDNN attention in selective checkpointing (#4084 by @winglian).

Documentation

• Flash attention without flash_attn (#4149 by @winglian): FA2/3/4 resolve to Hub kernels, so the packages are optional.
• Offline and air-gapped deployment guide (#4105 by @winglian).
• Nebius CLI authentication (#4164 by @SalikovAlex): the guide now uses nebius profile create instead of the nonexistent nebius auth login.
• AI-assisted contribution policy (#4096, #4106 by @winglian), and September README updates (#4101 by @winglian).

Model & Framework Support

🚨 Deprecations

• axolotl[flash-attn] extra removed (#4108 by @winglian): Flash Attention 2 now comes from Hub kernels.
• diffusion: and DiffusionPlugin deprecated in favor of diffusion_lm: (#4113 by @winglian); legacy configs are translated with a warning.
• Legacy axolotl.scripts / trl.scripts values for vllm.serve_module and vllm.worker_extension_cls ignored (#4039 by @NanoCode012), falling back to vLLM's native server with a warning.
• GRPO SequenceParallel classes renamed to ContextParallel (#4091 by @winglian), with deprecated aliases kept.

New Model Support

• GLM-5.3-Flash (#4077 by @ved1beta).

Dependency Updates

• transformers 5.17.0 → 5.20.1, peft 0.21.0 → 0.21.2, tokenizers 0.23.1 → 0.23.3, and kernels 0.16 → >=0.17.2 (#4097, #4157 by @winglian, #4163 by @NanoCode012).
• huggingface_hub 1.18.0 → >=1.31.0, trl 1.13.0 → 1.14.1, datasets 4.8.4 → 5.0.1, liger-kernel 0.8.1 → 0.8.4, mistral-common 1.11.5 → 1.12.0, hf_xet 1.4.3 → >=1.5.2, and trackio 0.19.0 → 0.40.0 (#4097 by @winglian). TRL's removed PRM trainer is now vendored so process reward models keep working.
• gradio 6.10.0 → 6.29.1 (#4039 by @NanoCode012).
• torch <2.15.0 for torch 2.14.1 (#4124 by @winglian).
• axolotl-ringmaster >=0.2.2 → >=0.2.6 (#4090, #4102, #4144 by @winglian).
• pydantic 2.12.5 → 2.13.5 and ray[train] >=2.52.1 → >=2.59.0 (#4117 by @winglian).
• New axolotl[coreweave] extra (#4150 by @winglian and @NanoCode012).

Bug Fixes

• SonicMoE EP silently ignored the expert LoRA on NVFP4 (#4151 by @winglian): runs trained attention LoRA only.
• EP checkpoint resume overwrote other EP groups' expert weights and optimizer state (#4114 by @winglian and @NanoCode012, #4134 by @winglian); restores now also validate shapes before copying.
• FSDP2 QLoRA FULL_STATE_DICT saves crashed with quantized MoE experts (#4114 by @winglian and @NanoCode012, fixes #4080).
• Context parallelism could pick schedules that break sliding-window and chunked attention masks (#4090 by @winglian).
• LoRA adapters saved empty under FSDP2 activation checkpointing (#4091 by @winglian).
• LoRA+ dropped embedding_lr, and LoRA merge dropped trainable-token rows (#4110 by @winglian).
• QLoRA merge quantized the wrong modules (#4120 by @NanoCode012).
• New validation errors (#4151 by @winglian): nvfp4_merge_aware with use_scattermoe, and quantize_moe_experts with use_sonicmoe.
• MoE and preference fixes (#4151 by @winglian): an int32 overflow in NF4 dequant on large MoEs, axolotl evaluate under FSDP2, and reasoning_content in DPO and ORPO datasets.
• Tied embeddings and embedding LoRA broke under autocast and peft_ensure_weight_tying (#4139 by @winglian).
• FSDP2 cpu_ram_efficient_loading with activation checkpointing raised KeyError (#4118 by @ved1beta, fixes #4079), a regression from #4045.
• eval_strategy: no ignored (#4110 by @winglian): it is now honored and disables eval_on_start.
• EP and loader fixes (#4091 by @winglian): NVFP4 experts under pure EP, and max_memory device maps keep decoder layers whole.
• GLM-4.7-Flash examples targeted the wrong attention projections (#4076 by @ved1beta): they now target the MLA projections, and lora_qkv_kernel / lora_o_kernel raise a clear error on unsupported layouts.
• **Example conf...

Axolotl v0.20.0 Release Notes

There has been 67 commits since v0.19.0 (September 10, 2026).

We have added a bunch of features including GGUF export, Ringmaster context parallelism, native NVFP4 LoRA, and expert parallelism without DeepEP. Python 3.12 and torch 2.13.0 are now the minimums, and FSDP1 is removed.

Highlights

GGUF Export

You can now convert a trained checkpoint to GGUF for llama.cpp, Ollama, LM Studio, and llamafile with axolotl export config.yml --quantize Q4_K_M,Q8_0, which emits the base conversion plus one file per requested quant type.

See the guide.

• Contributed by @ved1beta in #3950.

Context Parallelism with Ringmaster

You can now train long-context runs with packed sequences under Ulysses, Ring, or USP context parallelism, with experimental support for recurrent layers (GDN, KDA, Mamba2) that carry state across shards.

Setting context_parallel_size, or a context_parallel: block to pick the backend and Ulysses/Ring sizes, enables it without a plugins: entry; install with pip install axolotl[ringmaster]. examples/distributed-parallel/llama3-8b-ringmaster-cp.yaml pairs it with FSDP2. See the guide.

• Contributed by @winglian in #3844 and #4053.

Native NVFP4 LoRA with Merge-Aware Training

You can now LoRA-train on a frozen native NVFP4 base under FSDP2, DeepSpeed ZeRO-1/2/3, or tensor parallelism, and the merged checkpoint matches what you trained: the forward runs on the same quantized base-plus-adapter weight that axolotl merge-lora writes.

Merge-aware training is on by default for supported ModelOpt/SonicMoE and native TorchAO paths, and nvfp4_merge_aware: false opts out. See the guide.

• Contributed by @winglian in #4045, #4044, and #4035.

Expert Parallelism Without DeepEP

You can now run expert parallelism on any GPU setup through a plain all_to_all backend, and compose it with FSDP2, HSDP, and context parallelism for full-parameter or LoRA MoE training at node scale.

expert_parallel_backend: auto uses DeepEP when installed and the torch backend otherwise, and expert_parallel_dispatch_chunks overlaps dispatch with the expert GEMMs. EP now requires fsdp_version: 2 with state_dict_type: FULL_STATE_DICT, and EP with tensor parallelism is rejected. See the guide.

• Contributed by @winglian in #4070.

CPU-Staged NF4 Loading for FSDP2 QLoRA

You can now load an FSDP2 QLoRA run without the load-time GPU OOM: rank zero stages and quantizes the NF4 weights on CPU and distributes the shards, and bitsandbytes quantization is chunked so tensors above the int32 element limit no longer fail.

qlora_sharded_model_loading now defaults on for FSDP2 QLoRA with cpu_ram_efficient_loading, and nf4_backend: torchao selects torchao's NF4Tensor as an alternative to bitsandbytes, MoE experts included. Prequantized bitsandbytes 4-bit checkpoints also load this way, and their adapters merge to a bf16 base. See the guide.

• Contributed by @winglian in #4003, #4023, and #4027.

Remote Training on More Clouds

You can now launch axolotl train --cloud on Baseten and, experimentally, Nebius Serverless Jobs alongside Modal, and new providers can ship as plugins through the axolotl.cloud_providers entry point.

Set provider: in your cloud config, plus image or image_build to run a prebuilt or locally built image. See the docs. Hugging Face Jobs has its own guide.

• Contributed by @winglian in #4028, @SalikovAlex in #3994, and @davanstrien in #4042.

Nemotron-3 LatentMoE on the MoE Expert Kernels

You can now LoRA-train nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 through ScatterMoE or SonicMoE. Nemotron-3's experts are non-gated relu² in moe_latent_size width, which both kernels previously rejected outright.

The routed experts stay packed NVFP4 while the rest of the model dequantizes to bf16 at load. Config: examples/nemotron-h/super-120b-a12b-nvfp4-lora.yaml. See the support matrix.

• Contributed by @winglian in #3866.

Weighted Data Mixing

You can now downweight a large corpus without materializing a pre-mixed dataset: a weight in (0, 1.0] on an SFT dataset entry shuffles and subsamples it to weight * len(dataset) examples before tokenization.

The weight is folded into the dataset fingerprint, so caches are not reused across different mixes. Streaming datasets are not subsampled and warn at validation.

• Contributed by @madScientist10 and @winglian in #3690.

New Features

• MixLoRA (#3535 by @OnePunchMonk): adapter: mixlora with the MixLoraPlugin turns each FFN of a dense model into a routed mixture of LoRA experts, set by mixlora_num_experts and mixlora_top_k.
• Dynamic Fine-Tuning (#4008 by @ved1beta): use_dft: true reweights each token's cross-entropy by its own detached probability, per Wu et al., 2025.
• SCAO optimizer (#3624 by @whispering3): optimizer: scao after pip install scao.
• FLA backend for Mamba and Mamba2 (#4053, #4058 by @winglian): model_config.mamba_backend: fla swaps in FLA mixers for full fine-tuning and LoRA on in_proj/out_proj, with packing and context parallelism. Transformers stays the default. See the guide.
• Nemotron-H and native NVFP4 on the LoRA kernels, with FP32 LoRA gradients (#4032 by @winglian): the optimized LoRA projections now cover Nemotron-H attention and ReLU² MLPs and native weight-only NVFP4 bases, and opt-in lora_fp32_gradients keeps LoRA parameters and gradients in fp32 across single-device, DDP, FSDP2, and DeepSpeed. See the docs.
• Sample packing for Mamba, Mamba2, and Falcon-Mamba (#4030, #4034 by @winglian): packing now works on the Hugging Face checkpoints, with recurrent and convolution state reset at document boundaries, and Mamba1 packs without causal_conv1d installed.
• Liger and fused attention for GLM-4.7-Flash (#4029 by @winglian): Liger RoPE, RMSNorm, and SwiGLU (dense and shared-expert MLPs), and fused_attn_kernel fuses partial RoPE with the MLA Q/K/V assembly.
• Selective activation checkpointing save rules and FSDP2 x TP (#4070 by @winglian): save_modules and save_matmul_min_k choose what selective checkpointing keeps, and FSDP2 with tensor parallelism works again on accelerate 1.15 / transformers 5.17. See the docs.
• CCE fp32 and chunked classifier-gradient accumulation (#4037 by @winglian): cut_cross_entropy_accum_c_fp32 accumulates the lm_head gradient in fp32 for large vocabularies, and cut_cross_entropy_c_grad_chunk_size: auto bounds that fp32 buffer to cut peak memory.
• Dynamic work queue for dataset tokenization (#3460 by @Nero10578): with dataset_num_proc > 1, workers now pull chunks of examples as they finish instead of datasets.map's fixed shards, so a few long examples no longer leave the rest of the pool idle.
• Telemetry reports real final loss and learning rate (#3997 by @winglian): metrics are now carried forward across log entries, so runs no longer report 0 when evaluation or partial logs land after the training log. See the telemetry docs.

Documentation

• Dead links and a wrong anchor fixed across the Ministral and Mistral Small READMEs, docs/export.qmd, and the FAQ (#4015 by @simpleqt), plus a duplicated word in the rope_scaling deprecation warning (#4014 by @simpleqt).
• Dead cloud-provider links in the installation guide fixed (#4065 by @pratikgx).
• Nebius Serverless guide reorganized around setup, a first run, logs, outputs, and resume (#4040 by @SalikovAlex).

Model & Framework Support

🚨 Deprecations

• Python 3.12 and torch 2.13.0 are the new minimums (#4036, #4043 by @winglian): requires-python moves from 3.10 to 3.12 and torch from 2.11.0 to 2.13.0, since the Hub kernel builder only publishes builds for the latest two torch releases. Docker builds and CI now cover torch 2.13.0 and 2.14.0 (#4005 by @winglian, #4036), with Python 3.14 added to the test matrix.
• FSDP1 removed (#4024 by @winglian): FSDP2 is the default and the only supported version, following its deprecation in transformers. fsdp_version: 1, the top-level fsdp list, sharding_strategy (use reshard_after_forward), and forward_prefetch now raise, while other FSDP1-only options such as sync_module_states and use_orig_params are dropped with a warning. See the migration guide.
• ring-flash-attn extra replaced by ringmaster (#3844 by @winglian): install axolotl[ringmaster] for context parallelism. ring_attn_func and heads_k_stride are now ignored with a warning.
• Legacy MambaLMHeadModel loading removed (#4030 by @winglian): raw mamba_ssm checkpoints such as state-spaces/mamba-2.8b with model_type: MambaLMHeadModel no longer load. Use Hugging Face format checkpoints (e.g. state-spaces/mamba-130m-hf) instead.
• Modal is now an optional extra (#4028 by @winglian): modal moved out of the core dependencies, so Modal cloud users need pip install axolotl[modal] (Baseten uses axolotl[baseten]).
• PRM requires trl<=1.13.0 (#4017 by @NanoCode012): TRL [removed](https:/...

Axolotl v0.19.0 Release Notes

72 commits since v0.18.0 (July 17, 2026).

We have a variety of changes this release including a new declarative model-support system, seven new model families plus ternary (BitNet b1.58) QAT, two new optimizers, a big chat-template tokenization speedup, and PyTorch 2.13 support.

Highlights

Declarative Model Support Profiles

Adding a new architecture used to mean edits scattered across the loaders. A model now picks a family template (VANILLA_CAUSAL_LM, IMAGE_TEXT_TO_TEXT) and declares only what makes it different.

Existing descriptors keep working, and there are no config changes. See the guide.

• Contributed by @thad0ctor in #3869 and @winglian in #3884.

Qwen3.8-Flash-Next

You can now fine-tune Qwen's 176.94B multimodal MoE on a single B300, in as little as 120 GiB with ple_cpu_offload: true, for a minor throughput tradeoff.

QLoRA, vision QLoRA, and NVFP4 MoE-LoRA configs all ship. See the guide.

• Contributed by @NanoCode012 in #3965.

Ternary (BitNet b1.58) QAT

qat: weight_dtype: ternary trains a model with its weights restricted to {-1, 0, 1}, natively, with no extra dependency. examples/neutrino/ has the two-stage recipe Fermion used for Neutrino-1: ternary continued pretraining, then instruction tuning on top.

See the docs.

• Contributed by @ved1beta in #3933.

Two New Optimizers: PoLoRA and AdamC

PoLoRA, a Muon-like optimizer for LoRAs. Set optimizer: polora with LoRA or QLoRA, on single GPU, DDP, or FSDP2.

optimizer: adamc adds Adam with corrected weight decay, which scales decay by learning rate.

• Contributed by @NanoCode012 in #3938 and @ved1beta in #3929.

Chat-Template Tokenization speedup

Long multi-turn conversations no longer make preprocessing look like it has hung. An 80-turn conversation on Qwen2.5 went from 1577.6 ms to 96.4 ms, a 16.4x speedup.

The gains are most apparent with conversation length.

• Contributed by @ved1beta in #3750.

Six More Model Families

Ling 3.0 (Flash and Tiny), Muse Glimmer 30B, Cohere North Micro Vision Instruct, Shieldstral, Qwen3.8 27B, and Nemotron 3.5 Lightning 30B-A3B, each with example configs.

• Contributed by @NanoCode012 in #3931, #3932, and #3922, and @ved1beta in #3935, #3945, and #3949.

Performance & Kernel Optimizations

• Faster bnb NF4 dequant (#3882 by @NanoCode012): pass raw pointers and ints to the bitsandbytes dequant entry points when the 0.50.0+ argtypes are present.

New Features

• TRL 1.9.0 and new GRPO knobs (#3875 by @ved1beta): cfg.trl now surfaces entropy regularization (entropy_coef, use_adaptive_entropy, entropy_target, entropy_coef_delta/min/max), bidirectional importance-sampling clipping (vllm_importance_sampling_clip_min/max), and log_multimodal; scale_rewards widens to accept group/batch/none alongside the old boolean. See the GRPO docs.
• Explicit flash_attention_4 and flash_attention_torch backends (#3847 by @NanoCode012): FA4 was previously reachable only by auto-upgrade from FA2; it is now a canonical attn_implementation value, alongside torch's own varlen backend, both supporting varlen sample packing. flash_attention_torch has a preflight validator that fails clearly when the installed transformers does not register it.
• Configurable FP8 scaling recipes (#3969 by @JimmyWang0417): fp8_config.recipe selects TorchAO's tensorwise (the default), rowwise, or rowwise_with_gw_hp. fp8_enable_fsdp_float8_all_gather remains supported only with tensorwise, and incompatible rowwise combinations are rejected during validation.
• Liger kernel backend selector (#3896 by @thad0ctor): liger_kernel_impl: cutile or cutedsl (Blackwell-tuned) opts into liger 0.8.1's alternative kernel backends, while fused linear cross-entropy stays Triton on both. Gemma 4 now routes through liger's native dispatch instead of our hand-patch, picking up the multimodal FLCE forward, and ipo under dpo_use_liger_kernel fails at config time rather than mid-run.
• CLI help from dataclass metadata (#3855 by @vineethsaivs): argument help text is read from the help metadata key on dataclass fields, so plugin-defined args document themselves.

Documentation

• SkyPilot guide (#3948 by @alex000kim): new SkyPilot docs covering single-node, multi-node, and spot instances, checkpoint mounting, secrets, and dataset reuse. Re-adds material dropped from the README in #2295.
• Full attribute docstrings in the API reference (#3946 by @NanoCode012): a custom Quarto renderer so config attribute docs render in full instead of being truncated.
• S3-compatible endpoints for dataset loading (#3978 by @ved1beta): setting AWS_ENDPOINT_URL lets you load datasets from MinIO, Tigris, Cloudflare R2, or any other S3-compatible store while keeping the s3:// path format. See the dataset loading docs.
• delinearize-llama4 CLI invocation (#3912 by @latent-9): the documented --model / --output flags do not exist; the command takes positional arguments.
• Broken relative links and a stale docstring (#3986 and #3985 by @simpleqt): fixes dead README and example-config links across the diffusion, llm_compressor, and nemo_gym integrations, and corrects the parameter name in the do_vllm_serve docstring.

Model & Framework Support

New Model Support

• Qwen3.8-Flash-Next (#3965 by @NanoCode012): QLoRA, vision QLoRA, and NVFP4 MoE-LoRA configs in examples/qwen3.8-flash-next/, plus optional PLE CPU offload and CCE support.
• Ling 3.0 Flash and Tiny (#3949 by @ved1beta): in-tree modeling copy with packing support, LoRA and QLoRA configs in examples/ling3/. Ling-3.0-tiny LoRA measures 25.0 GiB reserved at 12.3 s/step on one H100 80GB.
• Muse Glimmer 30B (#3931 by @NanoCode012): multimodal, with examples/muse-glimmer/ QLoRA text and vision configs and Liger support.
• Cohere North Micro Vision Instruct (#3932 by @NanoCode012): a cohere_compass multimodal profile with image-text processing, plus QLoRA and FFT examples.
• Shieldstral (#3922 by @NanoCode012): text and vision LoRA configs, Liger for Mistral3/Ministral3, improved Mistral3 role masking and tokenizer discovery.
• Qwen3.8 27B (#3935 by @ved1beta): QLoRA example config, along with sample-packing fixes across the Qwen3.5 base/text/MoE variants.
• Nemotron 3.5 Lightning 30B-A3B (#3945 by @ved1beta): a QLoRA config for the new checkpoint, with the Nemotron-H modeling patch rewritten against upstream and the stale gradient-checkpointing restriction removed.

Dependency Updates

• transformers 5.14.1 → 5.16.1 (#3931 by @NanoCode012, #3957 by @ved1beta), with tokenizers 0.22.2 → 0.23.1.
• PyTorch ceiling raised to 2.13.0 (#3877 by @winglian), with GPU container builds for 2.12.1 and 2.13.0.
• TRL 1.8.0 → 1.9.0 (#3875 by @ved1beta), which raises the vLLM extra floor to >=0.17.0.
• peft 0.19.1 → 0.20.0 (#3895 by @thad0ctor).
• bitsandbytes 0.49.1 → 0.50.2 (#3882 by @NanoCode012, #3956 by @ved1beta).
• liger-kernel 0.8.0 → 0.8.1 (#3896 by @thad0ctor), which brings native Gemma 4 and DeepSeek V4 support.
• kernels 0.15.2 → >=0.16.0,<0.17.0 (#3931 by @NanoCode012).
• fla-core and flash-linear-attention 0.4.1 → 0.5.2 (#3980 by @NanoCode012).

Bug Fixes

• orig_mod prefix in FSDP2 + compile checkpoints (#3958 by @NanoCode012): drop the now-redundant accelerate patch that upstream fixed, which was leaking _orig_mod into saved state dicts, and stop PhiForCausalLM being mangled to hiForCausalLM by an lstrip("FSDP") that strips characters rather than a prefix. Saved tensors verified bit-identical. Also rejects cpu_ram_efficient_loading with load_in_4bit on FSDP2, where the rank-0-only quantization deadlocks the state-dict scatter.
• Fully-masked samples were never dropped (#3864 by @vineethsaivs): drop_no_trainable_tokens compared a Python list to -100 under np.any, which always evaluates truthy, so samples with zero trainable tokens survived dataset preparation. The comparison is now element-wise, and a dataset left empty by the filter raises with a pointer at train_on_inputs / roles_to_train / the chat template.
• Silent out-of-bounds writes in MoE expert kernels (#3865 by @winglian): expert weight stacks over 2^31 elements (512x5120x2048 at Nemotron-3-Ultra scale) overflowed the i32 expert-index times stride products, corrupting neighboring allocations in the grouped dW kernel. Expert indices and row ids are promoted to i64 at every stride multiplication, across both the ScatterMoE LoRA kernels and the SonicMoE NVFP4 Triton kernels.
• NF4 dequant queued work on a stale CUDA stream (#3888 by @NanoCode012): the fast dequant path cached current_stream() per device, but the current stream belongs to the caller's context, so a caller running under a non-default stream had its dequant queued on the wrong one. The stream is now resolved per call, read from torch rather than a bitsandbytes internal that upstream may remove.
• SSM state leaking across packed sequences on Nemotron (#3945 by @ved1beta): a Mamba2 chunk scan that silently ignores seq_idx mixes state between packed samples. The resolved scan is now probed once at startup and raises with an install hint instead of...

Axolotl v0.18.0 Release Notes

We've been hard at work doing low level improvements in the kernels. Over 90 commits since v0.17.0 (June 3, 2026), themed around fine-tuning very large sparse-MoE models cheaply: 4-bit expert LoRA/QLoRA (NVFP4, MXFP4, bnb) that runs fast and stays memory-flat at long context on Blackwell and Hopper, plus a supply-chain hardening pass on CI.

Highlights

NVFP4 MoE-LoRA: ScatterMoE and SonicMoE

4-bit NVFP4 expert LoRA now runs fast on both MoE kernel backends.

ScatterMoE

ScatterMoE gains fused NVFP4 (Marlin/DeepGEMM) and bnb-4bit expert paths for Gemma 4 (128 experts) and DeepSeek-V4. Active VRAM stays nearly flat from 4k to 32k context: NVFP4 is fastest at short sequences, bnb uses the least memory.

Configs: examples/gemma4/26b-a4b-moe-nvfp4-lora.yaml (ScatterMoE)

• Contributed by @winglian in #3747

SonicMoE

SonicMoE adds native FP4-activation MoE-LoRA (W4A4: 4-bit weights and activations) through the quack/CUTLASS kernels. On B200 it beats the Marlin W4A16 (16-bit-activation) path at every sequence length at matched quality, validated on the Qwen3-30B-A3B and Qwen3-Next-80B-A3B NVFP4 checkpoints.

It also runs on consumer Blackwell (RTX 50xx / sm120) via the quack 0.6 migration (#3830), and the new nvfp4_merge_aware mode keeps the adapter bitwise-consistent when merged back into the NVFP4 base (#3822).

Configs: examples/qwen3/30b-a3b-nvfp4-lora.yaml (SonicMoE).

• Contributed by @NanoCode012 in #3780, #3822, and #3830.

GLM-5.2 (DSA) Fine-tuning with 2D Expert Parallelism

Fine-tune GLM-5.2 (glm_moe_dsa) from its NVFP4 checkpoint on multi-GPU FSDP2. It follows the DeepSeek-V3.2 sparse-MLA lineage: 256 routed experts with Lightning-Indexer token selection. DeepEP expert parallelism now composes in 2D (EP × cp), and dedicated DSA attention kernels (use_glm_dsa_kernels) with rank0 + broadcast loading keep the ~250 GB 4-bit experts from OOMing on load. Config: examples/glm_moe_dsa/glm-5.2-nvfp4-lora.yaml (requires deep_ep).

• Contributed by @winglian in #3759, with post-merge fixes in #3781 (@NanoCode012) and #3832.

Hidden-States Activation Offloading for Long-Context Full-Parameter Training

New activation_offloading: hidden_states offloads only the per-layer checkpoint input to CPU and recomputes the rest, so PCIe stays within budget where the offload-everything mode saturates it. On Qwen3-8B full-param it reaches 128k context where plain gradient checkpointing OOMs, and the memory advantage widens with sequence length (1.23x less at 64k, tied throughput) with bit-exact gradients.

• Contributed by @winglian in #3733 and #3776.

Multi-Turn Inference Chat Interface

axolotl inference config.yaml --chat starts an interactive multi-turn chat with streaming token output, runtime-adjustable generation parameters, and streaming reasoning blocks for thinking models.

• Contributed by @NanoCode012 in #3723.

Performance & Kernel Optimizations

• Blackwell (sm120) MoE-LoRA (#3714 by @winglian): makes the vendored ScatterMoE Triton path the working MoE+LoRA story on sm120 where the SonicMoE CUTLASS kernel can't compile. EP sentinel-skip drops masked remote rows (2 to 10x fwd+bwd at ep 2 to 8), gpt_oss layout support (9.5 to 48x vs eager), fused MXFP4 with no dequant (1.7 to 3.7x, 16 to 24x less transient memory), and a SonicMoE-to-ScatterMoE fallback on sm120.
• Grouped-Gram dA/dB for large-E MoEs (#3712 by @winglian): recompute-free grouped-Gram LoRA weight grads plus a sync-free dX_lora path, up to 2.2x fwd+bwd on Qwen3-MoE / DeepSeek (E>=128), bit-identical to the split kernel.
• LoRA kernel memory (#3704 by @winglian): removes an intermediate materialization in the LoRA kernel op.
• Faster multimodal assistant-only masking (#3672 by @thad0ctor): vectorized role-boundary scanner (byte-identical to the reference) plus a fused process_labels, ~1.3 to 1.5x on Gemma 3/4 and Qwen 2 under DataLoader-worker conditions.
• torch.compile coverage for 4-bit dequant (#3677 by @thad0ctor): registers the NF4 dequant fast path as a Dynamo-opaque torch.library.custom_op, removing the ctypes trace failure and get_ptr recompile thrash on torch_compile: true QLoRA. Eager path is byte-identical.
• Custom torch ops for in-repo kernels (#3788 by @winglian): registers ~25 axolotl:: custom ops (attention, SwiGLU/GeGLU, RMSNorm, dsv4/glm_dsa) so kernels compile without graph breaks; a follow-up (#3789) hoists multidoc routing out of the d512 attention path to clear the last graph breaks under torch.compile.
• Fused LoRA for GatedDeltaNet (#3732 by @thad0ctor): route Qwen3.5 GatedDeltaNet linear-attention LoRA projections through the fused kernel under lora_qkv_kernel / lora_o_kernel, avoiding a bf16/fp32 activation round-trip.

New Features

• Multi-adapter MoE LoRA (#3719 by @winglian): multi-LoRA (multi-tenant) support for the ScatterMoE and SonicMoE kernels with optimized routing and gradient computation.
• Selective activation checkpointing (#3786 by @winglian): selective_checkpointing saves attention outputs (SDPA/flash-attn) instead of recomputing them, fully eager, with optional CPU offload of the saved tensors.
• Per-module LoRA rank/alpha (#3673 by @thad0ctor): lora_rank_pattern / lora_alpha_pattern set per-module LoRA rank and alpha, forwarded to PEFT and honored by the memory-efficient merger.
• optimizer: sinkgd (#3763, #3785 by @winglian): stateless gradient multi-normalization optimizer (Scetbon et al., 2025), ~87% less optimizer-state memory on an 8B full finetune (~24% lower peak vs 8-bit AdamW) at matched loss, extended with width transfer, spectral norm, and fused Triton kernels (#3785).
• torch_compile_options (#3692 by @thad0ctor): expose an allowlist of torch._inductor.config flags (e.g. max_autotune_gemm) via config, validated at preprocess.
• Plugin CLI subcommands (#3840 by @winglian): integrations register axolotl subcommands via the axolotl.cli_commands entry-point group, resolved lazily so --help no longer imports torch.
• FSDP min_num_params for size-based wrap (#3710 by @thad0ctor): expose fsdp_config.min_num_params in YAML for SIZE_BASED_WRAP.
• Lower-RAM SFT dataset loading (#3711 by @ved1beta): fixes host-RAM OOM when loading large preprocessed SFT datasets.

Documentation

• Support matrix (#3782 by @NanoCode012): a model/feature support matrix in the docs (closes #3440).
• Contributor & agent testing guidance (#3748 by @thad0ctor): improved test docs.
• DeepEP pin for expert parallelism (#3778 by @trevorgordon981): pin DeepEP to v1.2.1 in the DISABLE_NVSHMEM build instructions, since DeepEP main removed the code path the patches edit.

Model & Framework Support

Deprecations

• Non-uv Docker images removed (#3740 by @NanoCode012): the pip-based image variants are dropped; -uv images are the supported path, with general Docker cleanup alongside.

New Model Support

• GLM-5.2 (glm_moe_dsa) (#3759 by @winglian): see Highlights.
• Gemma 4 Unified (encoder-free multimodal) (#3706 by @NanoCode012): text + vision LoRA configs, a dedicated chat template, and Liger + Cut Cross Entropy support.
• DeepSeek-V4 MoE-LoRA kernels (#3747 by @winglian): fused NVFP4/bnb expert paths for DeepSeek-V4 alongside gemma4 (see Highlights).
• MiniMax M2 (MoE) (#3702 by @ved1beta): fine-tuning support with QLoRA, kernel-optimization patches, and an example config.
• PaddleOCR-VL (#3774 by @thad0ctor): multimodal image-text support with a dedicated processing strategy (no packing/Liger/CCE), plus QLoRA and full fine-tune examples.

Dependency Updates

• transformers 5.9.0 → 5.14.1 (#3706, #3759, #3761, #3822): now pinned ==5.14.1.
• trl 1.5.1 → 1.8.0 (#3784 by @ved1beta, #3822 by @NanoCode012).
• liger-kernel 0.7.0 → 0.8.0 (#3713 by @thad0ctor): adopts liger's native auto-patch for gemma4_text / qwen3_5_text / ministral / nemotron while keeping axolotl's gated-RMSNorm handling for qwen3_5 / qwen3_5_moe; a follow-up (#3826) drops the now-shadowed llama4 / qwen3 / qwen3_moe hand-patches.
• huggingface_hub 1.1.7 → 1.17.0 (#3706, #3804 by @NanoCode012): now pinned ==1.17.0.
• httpx added to the uv Docker image (#3770, #3775 by @NanoCode012): pinned <1 to avoid pulling a pre-release.
• numpy loosened to >=2.1,<3.0 (#3843 by @winglian): numba metadata governs the effective ceiling.
• pydantic pinned ==2.12.5 (#3761, #3804).
• cu130 builds gated to torch 2.11.0+ (#3761 by @winglian); uv Docker images for torch 2.11, 2.12, and 2.13 (#3715, #3816 by @winglian).

Bug Fixes

• LoRA merge on quantized bases (#3771 by @winglian): dequantize quantized (NVFP4/bnb) expert bases before folding the LoRA delta so fused expert weights are preserved.
• Gemma 4 shared-KV key on transformers >=5.8 (#3701 by @thad0ctor): key the shared K/V by layer_type instead of the removed attribute.
• CCE update and post-merge bugs (#3757 by @NanoCode012).
• DPO user_defined strategy KeyError on custom field names (#3742 by @vineethsaivs).
• KTO user_defined dataset transform crash (#3730 by @Ayushhg...

Axolotl v0.17.0 Release Notes

Another packed release. ~84 commits since v0.16.1 last month, bringing Expert Parallelism for MoE training, BitNet 1.58-bit fine-tuning, remote training via Tinker, context parallelism for hybrid SSM models, MXFP4 ScatterMoE-LoRA, fused RMSNorm+RoPE kernels for the Qwen3 family, a systemic fix for multimodal loss masking, a uv-first install path, and a long tail of stability/perf work on FSDP2, Gemma 4, and DPO.

Highlights

Expert Parallelism (EP) via DeepEP

Distributed MoE training across ranks via DeepSeek's DeepEP all-to-all kernels, verified on 2×A100, 4×A100, 8×H100, and EP + FSDP composition. Hopper low-latency kernels and TP/CP composition are follow-ups. See the Expert Parallelism docs and DeepEP integration guide.

• Contributed by @NanoCode012 in #3632.

Train on Remote Compute via Tinker-compatible APIs

Run training against a Tinker / Hatchery API endpoint instead of local hardware, with example SFT and GRPO configs plus a math reward function for RL workflows.

• Contributed by @winglian in #3614.

BitNet 1.58-bit Fine-tuning

Full fine-tuning support for BitNet via the onebitllms library, with config-validation guards against incompatible LoRA setups and a setup guide for post-training conversion.

• Contributed by @younesbelkada in #3634 and #3636.

Q-GaLore Optimizer

New memory-efficient optimizer q_galore_adamw8bit based on Q-GaLore. Requires FSDP2, full fine-tuning, and bf16. Not compatible with adapters.

• Contributed by @ved1beta in #3654.

MoRA / ReMoRA Integration

Optional MoRA (Mixture-of-Rank-Adapters) support with ReMoRA restart scheduling, registered through a new plugin-based adapter system so future custom adapters can slot in the same way. A companion ReLoRA cleanup fixes the optimizer reset scope, adds a configurable relora_prune_method, and renames relora_steps → jagged_restart_steps (breaking, see Deprecations).

• Contributed by @winglian in #3647 and #3646.

Multimodal Assistant-only Loss Masking

Fixes a long-standing bug where train_on_inputs / roles_to_train / train_on_eos were silently ignored in the multimodal collator: every model except Gemma 3n was training on the full sequence regardless of config. New per-template strategies for Gemma 4, Llama 3.2 Vision, Llama 4, Pixtral, and Mistral V7 Tekken, plus an opt-in cfg.role_boundaries override for unverified templates.

Heads up: if you were training multimodal models on assistant-only data, your loss values will change after upgrading. This is expected.

• Contributed by @thad0ctor in #3625.

DPO Loss Types & SimPO LoRA fix

New dpo_loss_type (list) and dpo_loss_weights config knobs expose the full TRL ≥ 0.29 loss menu and let users mix multiple DPO losses with custom weightings, restoring losses like RPO that broke after the TRL 0.29 refactor. Separately, rl: simpo + LoRA no longer raises ValueError: You passed a PeftModel instance together with a peft_config.

Deprecation: rl: ipo is deprecated. Use rl: dpo with dpo_loss_type: ["ipo"] instead.

• Contributed by @BrownianNotion in #3566 and @ved1beta in #3665.

Context Parallelism for Hybrid SSM Models

Context parallel support for hybrid attention + Mamba2 SSM models (Nemotron-H, Falcon-H1, Bamba, Granite MoE Hybrid, and Zamba2), plus seq_idx threading so SSM state resets correctly at packed-sequence boundaries. Uses an exact additive correction (exploiting SSM linearity) with one P2P round per layer, not the O(world_size) of ring attention.

• Contributed by @ved1beta in #3572.

uv-first installs and Docker images

uv is now the recommended package manager. New -uv Docker image variants ship with a generated lockfile and a migration guide from pip; minimum PyTorch is bumped to 2.9.1.

• Contributed by @NanoCode012 in #3545.

Gemma 4 Hybrid Attention + Fused RMSNorm/RoPE Kernels

Mixed FA + SDPA dispatch for Gemma 4 (FA2 on standard layers, SDPA where head_dim=512 OOMs FA2), plus a fused RMSNorm+RoPE Triton kernel. Enable via gemma4_hybrid_attn_impl: true. A VRAM leak in this path under activation checkpointing was also fixed.

• Contributed by @winglian in #3598 and @thad0ctor in #3611.

Fused RMSNorm+RoPE Kernels for Qwen3 / Qwen3.X

Generalizes the Gemma 4 fused RMSNorm+RoPE Triton kernel to the Qwen3 family (Qwen3, Qwen3-MoE, Qwen3.5, Qwen3.6, Qwen3-VL) behind a new opt-in cfg.fused_attn_kernel, and auto-enables Liger's fused (m-)rope for the Qwen-VL models.

• Contributed by @thad0ctor in #3680.

ScatterMoE-LoRA: MXFP4 Weights and Tiled-MLP for Long Context

Adds MXFP4-quantized expert weights to ScatterMoE-LoRA for memory-efficient MoE adapter training, plus a tiled-MLP path (FSDP2 reshard fix, grad-accumulator dtype fix, and a shard-count heuristic worth 3.2× at long context). An INT64 indices fix in the scatter2scatter Triton family resolves silent cuBLAS failures when routed-token offsets cross the 2³¹ boundary.

• Contributed by @winglian in #3663 (MXFP4), #3666 (tiled-MLP), and #3667 (INT64 indices).

Performance & Kernel Optimizations

• Pre-cache eot token ids (#3594 by @winglian): avoids recomputing on every iteration.
• DPO collation padding (#3601 by @winglian): new pad_to_multiple_of pads to buckets to avoid blowing up FLA autotune memory on Qwen 3.5 DPO/ORPO.
• gc_steps → torch_empty_cache_steps (#3604 by @SuperMarioYL): splits cache-clear and Python GC into separate native knobs; gc_steps is auto-migrated.

New Features

• fp32_norms for FSDP2 (#3670 by @winglian): keep RMSNorm/LayerNorm in fp32 while training in bf16/fp16. Required for models like AFMOE that declare fp32 norms.
• excess_length_strategy for RL trainers (#3578 by @yurekami): DPO/IPO/ORPO/SimPO/KTO now respect drop / truncate / raise (previously always dropped).
• FineGrainedFP8Config quantization (#3587 by @madScientist10): loads FP8-quantized models with optional dequantize: true for FFT.
• Skip redundant eval on checkpoint resume (#3575 by @joaquinhuigomez)
• processor_kwargs YAML field (#3612 by @thad0ctor): overrides for e.g. Gemma 4 image_seq_length or Qwen-VL min_pixels.
• field_messages support in multimodal collator (#3628 by @cyc00518)
• Multimodal AutoProcessor support (#3656 by @ved1beta)
• String-formatted messages field (#3607 by @brightwind26): JSON-string messages fields are now decoded transparently.
• Trainable / masked spans in content and reasoning_content (#3592 by @winglian): per-part train / weight flags; replaces train_details.
• Attention implementation refactor (#3602 by @winglian): unified attn_implementation (eager / flash / sdpa / xformers / flex / sage / s2 / fp8), with new FP8 attention backend. Legacy flags still honored.
• Multi-GPU Qwen3.5 FFT example config (#3605 by @ved1beta)
• DO_NOT_TRACK / AXOLOTL_DO_NOT_TRACK now properly honored (#3580 by @maximegmd)

Documentation

• Easier doc discovery for agents (#3579 by @winglian): surfaces agent-optimized docs more prominently.
• Document jinja2 file path support (#3588 by @NanoCode012): clarifies that chat templates can be loaded from a local jinja file.
• Update Docker docs (#3623 by @NanoCode012): refreshed for the uv-first builds and the new image variants.
• Security policy updated (#3645 by @NanoCode012): now uses email rather than Discord.
• Cut Cross Entropy uninstall hint (#3583 by @floaty3): better message when an import mismatch is detected.

Model & Framework Support

Deprecations

• relora_steps renamed to jagged_restart_steps (#3646 by @winglian): breaking config change, no migration shim. Existing ReLoRA configs need a one-line update.
• rl: ipo deprecated in favor of rl: dpo, dpo_loss_type: ["ipo"] (#3566 by @BrownianNotion)
• gc_steps deprecated in favor of torch_empty_cache_steps + gc_collect_steps (#3604 by @SuperMarioYL)
• Legacy attention flags soft-deprecated by attn_implementation (#3602 by @winglian): still honored

New Model Support

• Mistral Medium 3.5 (#3633 by @NanoCode012): base config and QLoRA examples for text + vision; reasoning-trace compatible.

Dependency Updates

• transformers 5.5.0 → 5.9.0 (#3593, #3603, #3696 by @winglian, #3650 by @NanoCode012)
• trl 0.29.0 → 1.5.1 (#3603, #3696 by @winglian)
• peft 0.18.1 → 0.19.1 (#3671 by @winglian)
• datasets 4.5.0 → 4.8.4
• kernels 0.12.2 → 0.13.0
• vLLM ≥ 0.19.0 for torch 2.10.0 (#3582 by @winglian)
• PyTorch 2.12 base images (#3621, #3697 by @winglian): adds 2.11/2.12 bases and prunes unused image variants.
• Blackwell Docker variant (#3641 by @ved1beta): CUDA 13.0 + PyTorch 2.9.1 with TORCH_CUDA_ARCH_LIST="9.0 10.0 10.3 12.0+PTX" covering B100/B200/B300 and RTX 50-series.
• typer < 0.26.0 pinned (#3684 by @winglian): latest typer broke the HF CLI.

Bug Fixes

• Gemma 4 VRAM leak in hybrid FA2+SDPA path (#3611 by @thad0ctor): shared_kv_states captured by checkpointing partials under FSDP2 pinned K/V tensors across steps. Routed through TLS; VRA...

Axolotl v0.16.1 Release Notes

Gemma 4 Support

Example YAML: https://github.com/axolotl-ai-cloud/axolotl/blob/main/examples/gemma4/26b-a4b-moe-qlora.yaml

• gemma4 support by @winglian in #3574

Full Changelog: v0.16.0...v0.16.1

Axolotl v0.16.0 Release Notes

We’re very excited to share this new packed release. We had ~80 new commits since v0.15.0 (March 6, 2026).

Highlights

Async GRPO — Asynchronous Reinforcement Learning Training (#3486)

Full support for asynchronous Group Relative Policy Optimization with vLLM integration. Includes async data producer with replay buffer, streaming partial-batch training, native LoRA weight sync to vLLM, and FP8 compatibility. Supports multi-GPU via FSDP1/FSDP2 and DeepSpeed ZeRO-3.

Achieves up to 58% faster step times (1.59s/step vs 3.79s baseline on Qwen2-0.5B).

ScatterMoE + LoRA Fused Triton Kernels (#3513)

Custom fused Triton kernels for training MoE models with LoRA adapters. By fusing the base expert matmul and LoRA computation into a single kernel pass, these kernels achieve up to 15x faster forward passes and 40x less activation memory vs the eager baseline.

SonicMoE Fused LoRA (#3519)

LoRA support for SonicMoE (CUTLASS-based MoE kernels for Hopper/Blackwell GPUs).

This results in upto 1.45x speedup and 30% reduction over grouped_mm baseline on a 1xH100 SXM GPU (Qwen3.5-35B-A3B 8-bit LoRA tuning).

GRPO Flattening & Packing (#3552)

Enables batch flattening and sample packing for GRPO training, improving token efficiency and training throughput (~10%).

Flash Attention 4 support (#3481)

Support for Flash Attention 4 on Hopper and Blackwell GPUs with automatic fallback to FA2/3 depending on hardware.

We found improvements mainly on higher sequence lengths and on Blackwell GPUs.

NeMo Gym Integration (#3516)

Full integration with NVIDIA NeMo Gym for single-turn and multi-turn RL training. Re-use existing environments in Axolotl now! https://github.com/NVIDIA-NeMo/Gym

Energy-Based Fine-Tuning (EBFT) (#3527)

Novel RL training method that matches generated completions to ground-truth feature embeddings without requiring external reward models. Supports structured (vLLM) and strided (single GPU) modes.

Performance & Kernel Optimizations

• Custom Triton Kernels for RL (#3510) — Fused Triton kernels for entropy_from_logits (single-pass online algorithm) and selective_log_softmax (fused forward+backward). Both avoid materializing the full [B, L, V] softmax intermediate. Benchmarked on Qwen vocab (V=151,936), RTX 5090:
  • entropy_from_logits: 5.2x faster, near-zero memory overhead (0.2 MB vs 120 MB); on non-contiguous tensors: 7.3x faster, saves up to 10 GB by avoiding .contiguous() copy
  • selective_log_softmax: 2.9x faster forward, 3.2x faster fwd+bwd, memory reduced from ~5 GB to <0.2 MB (at B=8, L=2048) — the original OOMs at B=16 where Triton succeeds
• LoRA Kernel Enhancements (#3528) — Extended LoRA kernels to support bias, dropout, DoRA, and embeddings with FSDP2 compatibility — making them production-ready for all LoRA configurations.
• Liger Kernels for Qwen 3.5 (#3531) — Fused RMSNorm + gated activation kernels for Qwen 3.5 and Qwen 3.5 MoE.
• Auto TF32 (#3473) — Automatically enables TF32 matmul/cuDNN when hardware supports it.
• Reduced Autotune Search Space (#3525) — Faster kernel autotuning startup.

New Features

• CPU Layer Offloading (#3512) — Offload frozen decoder layers to CPU during LoRA training, streaming to GPU only during forward/backward. Dramatically reduces VRAM usage for large models.
• Multiple Custom Optimizers (#3457) — End-to-end support for Flash AdamW, Optimi AdamW, ADOPT AdamW, Muon, Dion, Schedule-Free AdamW, and CAME PyTorch.
• Memory-Efficient LoRA Merging (#3095) — Iterate through weight bins without loading entire layers into memory, enabling merging of massive models on resource-constrained systems.
• Custom MoE Routing (#3526) — Support for Ernie 4.5 MoE and Hunyuan V1 MoE custom routing strategies beyond standard softmax-topk.
• Synthetic Benchmarking Datasets (#3518) — Built-in synthetic datasets for benchmarking and testing training pipelines.
• MX Quantization-Aware Training (QAT) (#3553) — Support for MX format QAT with torchao integration, including state dict hooks for saving dequantized safetensors from transformers.
• dtype Propagation for torchao (#3569) — Proper dtype handling through torchao quantization and optimizer paths.
• Lazy Trainer Loading (#3568) — Lazy-load trainer classes to avoid unnecessary imports, improving startup time.

Documentation

• Greatly improved documentation for GRPO and Agents use (#3564) — New dedicated guides for GRPO training, vLLM serving, training stability and monitoring, debugging, and choosing training methods. Adds agent-specific documentation and deduplicates content across pages.

Model & Framework Support

Deprecation

• Deprecated torch 2.8.0 support (#3550)
• Removed dead SDPA patches (#3488)

New Model Support

• Mistral Small 4 (#3502)
• Qwen 3.5 / Qwen 3.5 MoE (#3515, #3523, #3554)
• NeMo Super (Nemotron-H) Support (#3508)

Updates

• Transformers 5.4.0 (#3562)
• lm-eval updated for Transformers v5 (#3571)
• Torchao 0.17.0 (#3569)
• Gemma 3 config fixes (#3500)
• Docker builds for py312-cu128-torch2.9.1 (#3489)

Bug Fixes

• Fixed high eval loss with sample packing (#3478)
• Fixed double sequence partition with Context Parallelism (#3498)
• Fixed DPO compatibility with transformers v0.29 (#3560)
• Fixed Ray train crashing after succeeding (#3542)
• Fixed race condition in patching checks (#3543)
• Fixed token state JSON and Mistral tokenizer issue (#3522)
• Fixed async model loading with quantized BNB models (#3477)
• Fixed MoE quant patch for merge mismatches (#3483)
• Fixed connection error handling for user whoami check (#3529)
• Patches only applied when CUDA is available (#3561)
• Fixed shell injection in modal CLI (#3487)
• Fixed DPO tool role KeyError (#3217), dataset hash output_dir (#3303), config validators (#3538)
• Added precompute_ref_log_probs to config schema (#3555)
• Added troubleshooting note for GLM4 GGUF MTP mismatch (#3559)
• Allow bf16 flag with deprecation warning (#3563)

Infrastructure

• Scored rollouts dispatched to plugins with extended external plugin paths (#3549)
• CI improvements: codecov informational only, better error handling (#3534, #3517)
• Make CI fail GitHub Actions on test failures (#3517)
• Consolidate routing behavior in ScatterMoE kernels (#3475)
• roundup_power2_divisions no longer needed with newer PyTorch versions (#3540)
• Logging cleanup (#3482)
• Update pre-commit hooks (#3567)
• Update README (#3503)
• Docker: ubuntu user improvements for uv images (#3491, #3492, #3494, #3495, reverted #3496)

What We Tried (So You Don't Have To)

A few optimizations we prototyped but found little-to-no improvement — sharing so others can skip these rabbit holes:

• Fused NF4 dequant inside the ScatterMoE Triton GEMM kernel — reads NF4-packed expert weights directly in the inner loop, avoiding the bf16 intermediate buffer entirely. Correctness-verified (zero numerical error), but 3x slower than the separate dequant + bf16 path. BnB's dequant kernel reads data linearly at full memory bandwidth (~1 TB/s), while the fused kernel's scattered byte addressing and per-element NF4 codebook lookups destroy memory coalescing.

• Batched NF4 dequantization — concatenating gate_up + down_proj packed NF4 data into a single dequant call. Produced identical performance (1.00x) — the kernel is purely bandwidth-bound, so one large call = two smaller calls at the same total bytes.

• Dequant buffer pool for NF4 MoE — pre-allocating reusable bf16 buffers to avoid 512 alloc/free cycles per step during NF4 expert dequantization. Reduced memory fragmentation from 20....

Axolotl v0.15.0 Release Notes

This release brings new model support, significant MoE improvements, infrastructure updates with Torch 2.10.0 and uv builds, and a collection of quality-of-life fixes across the board.

🚀 Major Changes

Torch 2.10.0 & uv Builds

We have upgraded to Torch 2.10.0 and introduced uv-based Docker builds for faster, more reproducible image creation. Python 3.14 is now used in unit tests.

• Contributed by @winglian in #3429, #3430, #3431, and #3450.

ScatterMoE LoRA & SonicMoE

Added LoRA support for ScatterMoE and introduced SonicMoE as a new MoE kernel option for faster and more memory-efficient MoE training compared to grouped_mm in transformers.

• ScatterMoE LoRA by @winglian in #3410.
• SonicMoE by @NanoCode012 in #3411.

MoE Expert Quantization

Added support for quantizing MoE expert weights in Transformers v5, which dramatically reduces peak reserved memory. For example, GLM-4.7-Flash QLoRA drops from ~127 GiB to ~23 GiB reserved memory. Activate via quantize_moe_experts: true. See the expert quantization docs for details.

• Contributed by @NanoCode012 in #3439.

🎉 New Features

• New Model Support:
  • GLM models with dedicated patch. (#3329 by @ved1beta)
  • Step3.5 for Cut Cross Entropy. (#3384 by @NanoCode012)
  • Qwen3.5 and Qwen3.5 MoE models with packing support. (#3442 by @ved1beta)
• SageAttention: Added SageAttention integration for efficient attention computation. (#2823 by @NanoCode012)
• MXFP4 Quantization: Added support for MXFP4 quantization. (#3375 by @ved1beta)
• Hub Revision Support: Added hub_revision for specifying a branch when pushing checkpoints. (#3387 by @madScientist10)
• Dot-Notation CLI Args: Support for dot-notation CLI arguments for nested config options. (#3419 by @ManasVardhan)
• Sample Generation for SFT: New sample generation support to SFT training. (#3240 by @ved1beta)
• train_per_sec_per_gpu Metric: Added a new training throughput metric. (#3364 by @ved1beta)

⚠️ Breaking Changes

• dataset_processes → dataset_num_proc: The config field dataset_processes has been renamed to dataset_num_proc. (#3352 by @tgoab)

🐛 Bug Fixes

• Context Parallel:
  • Fixed state dict saving and eval. (#3382 by @ved1beta)
  • Corrected total_num_steps and batch_size calculation. (#3444 by @Yatimai)
• GRPO:
  • Fixed config not accepting max_prompt_length. (#3390 by @NanoCode012)
  • Moved rollout logic into set_training_kwargs. (#3392 by @ved1beta)
• Telemetry:
  • Removed telemetry warning and improved logging. (#3397, #3398 by @NanoCode012)
  • Disabled telemetry on non-master ranks. (#3438 by @NanoCode012)
• LoRA Kernels: Improved failure messaging and handling of trust_remote_code. (#3378 by @NanoCode012)
• Generation Mode: Fixed add_special_tokens handling and enabled test mode for generation. (#3396 by @NanoCode012)
• CCE Generic Patch: Fixed generic patch for Cut Cross Entropy. (#3405 by @winglian)
• MistralProcessor: Updated for Transformers v5 compatibility. (#3423 by @NanoCode012)
• Adapter Config Validation: Set allowed values for adapter in Pydantic config. (#3415 by @NanoCode012)
• Tokenizer/Processor Revision: Pass revision parameter to tokenizer and processor loaders. (#3388 by @madScientist10)
• Excess Length Truncation: Fixed excess_length_strategy truncation method. (#3401 by @rlronan)
• Scheduler Fallback: Use self.optimizer if optimizer not passed to SchedulerMixin.create_scheduler(). (#3435 by @kallewoof)
• Dataset Deduplication: Fixed saving de-duplicated dataset during pre-processing. (#3427 by @ManasVardhan)
• FSDP2 Sharding: Fixed FSDP2 sharding and validated AO version for LR groups. (#3403 by @bekk02)
• Qwen3-Next packing: Fixed causal-conv1d call for packing via FLA instead of native pytorch. (#3437 by @NanoCode012)
• Quantize MoE validation: Fixed validation to prevent lora_target_linear to be true when quantizing moe experts (#3461 by @NanoCode012)
• TP batch_size calculation: Fix calculation (#3462 by @Yatimai)

📦 Dependency & Infrastructure Updates

• transformers to 5.3.0, accelerate to 1.13.0, trl to 0.29.0, kernels to 0.12.2, torchao to 0.16.0. (#3407 #3459 by @winglian)
• Trackio 0.16.1: Updated trackio to 0.16.1. (#3425 by @winglian)
• Cut-Cross-Entropy Bump: Bumped cut-cross-entropy to 58d6572. (#3424 by @NanoCode012)
• BunnyCDN for CI: CI assets now served via BunnyCDN. (#3422 by @winglian)
• Telemetry Params: Included number of params and rounded estimates in PostHog telemetry. (#3455 by @winglian)

📖 Documentation

• Clarified ScatterMoE usage. (#3408 by @NanoCode012)
• Clarified lm_eval plugin usage. (#3404 by @NanoCode012)
• Added limitation note for unfrozen_parameters. (#3416 by @NanoCode012)
• Added expert quantization docs, GLM4.5 Air example configs, and updated README. (#3452 by @NanoCode012)
• Add lint information to contributor doc (#3458 by @winglian)

Other Changes

• set 0.15.0.dev0 version by @winglian in #3380
• Fix typo in dataset_processes field by @lorenzbaraldi in #3426
• mark slow tests that are timing out in CI by @winglian in #3428
• don't install torch ao on arm64 by @winglian in #3448
• uv cloud image should use uv w pip by @winglian in #3449
• fix python version typo for building 3.11 by @winglian in #3454
• fix uv cache subcommand by @winglian in #3447
• extend pytest-sdist timeout to 30 min for slow/flaky tests by @winglian in #3456
• chore: update pre-commit hooks by @github-actions[bot] in #3381

New Contributors

• @tgoab made their first contribution in #3352
• @madScientist10 made their first contribution in #3387
• @lorenzbaraldi made their first contribution in #3426
• @ManasVardhan made their first contribution in #3419
• @rlronan made their first contribution in #3401
• @bekk02 made their first contribution in #3403
• @Yatimai made their first contribution in #3444

Full Changelog: v0.14.0...v0.15.0

This is a major release marking our migration to Transformers v5. Along with this significant core dependency upgrade, we are introducing major performance optimizations for MoE models and new fine-tuning methods.

🚀 Major Changes

Transformers v5 Upgrade

We have upgraded the underlying transformers dependency from v4 to v5. This has been a long-term effort to ensure Axolotl remains compatible with the latest ecosystem advancements, stability improvements, and future model architectures.

• Contributed by @winglian in #3272 and #3376.

Faster MoE Training

We've added support for selecting MoE kernels via transformers: batched_mm and grouped_mm. Alongside this, we have added our custom integration for scattermoe. This significantly speeds up training and reduces VRAM usage for Mixture of Experts (MoE) models.

• Contributed by @winglian in #3377.

🎉 New Features

• EAFT Support: Added support for Efficient Adaptation Fine-Tuning (EAFT). (#3366 by @salmanmohammadi)
• New CCE Support: Added Cut Cross Entropy support for GLM 4.7 Flash, GLM Image, GLM 4.6v, and Exaone 4. (#3373 by @NanoCode012)

Full Changelog: v0.13.2...v0.14.0