Model Merging

Model merging combines the weights of compatible models into a single model without additional training. It can be useful when you have several fine-tunes of a common base model and want to combine their capabilities, or when you want to interpolate between a model and a further fine-tuned checkpoint.

Sentence Transformers uses mergekit to merge weights through its in-memory API. The result has the same architecture as the inputs and can be used, trained, and saved like any other Multi-Vector Encoder.

Merging can improve some tasks while making others worse. Evaluate the individual inputs and the merged model on the tasks you care about before choosing a merge.

Installation

Install the merge extra, which includes mergekit>=0.1.5:

pip install "sentence-transformers[merge]"

Merge Two LateOn Models

LateOn and LateOn-regularized share compatible architectures, tokenizers, and query/document length settings:

from sentence_transformers import MultiVectorEncoder

merged = MultiVectorEncoder.merge(
    models=["lightonai/LateOn", "lightonai/LateOn-regularized"],
    method="linear",
    parameters={"weight": [0.6, 0.4]},
)
merged.save_pretrained("merged-lateon")

queries = merged.encode_query(["What is model merging?"])
documents = merged.encode_document(["Model merging combines fine-tuned weights."])
print(merged.similarity(queries, documents))

See model_merging.py for the runnable script.

Choosing a Merge Method

Select a mergekit method with method and pass its options through parameters. See mergekit’s method documentation for descriptions, parameters, and defaults.

Start with linear as a baseline. You can also compare slerp for two models, or ties, dare_linear, and dare_ties when you have a compatible common base checkpoint. The shared model merging guide shows how to pass parameters and a base model.

See MultiVectorEncoder.merge for the supported method names, arguments, and return value.

Choosing Inputs and Evaluating

Matching transformer weights alone is not enough. The projection layers, tokenizers, and module configurations must also be compatible, including query/document lengths and query expansion settings. Models with different token embedding dimensions cannot be merged with this API.

The output inherits configuration, prompts, and processors from the first input. Compare the individual models and the merge on representative retrieval data, using the same query and document processing settings.

Merging defaults to float32 on CPU and holds the inputs and the result in memory. See Memory and Precision for details.

Results on NanoBEIR

The evaluation below replaces LateOn in the opening example with GLInt and uses a 50/50 linear merge. The table covers all 13 NanoBEIR datasets. Scores are nDCG@10 multiplied by 100, averaged equally across datasets. All three models used the same prompts and length limits: 32 tokens for queries and 8192 for documents, expanding the parents’ document limit from 300 tokens.

Model All 13 datasets MS MARCO Other 12 datasets
GLInt 69.96 72.85 69.71
LateOn-regularized 69.79 70.78 69.71
50/50 merge 70.61 73.62 70.36

The merge gained 0.66 points over GLInt on average and outperformed both parents on 5 of 13 datasets. It scored below both parents on SciDocs and SciFact.

Use the Published Merge

Load multi-vector-encoder/merged-GLInt-LateOn directly with MultiVectorEncoder:

from sentence_transformers import MultiVectorEncoder

merged = MultiVectorEncoder("multi-vector-encoder/merged-GLInt-LateOn")

The model card includes the results for each dataset and a model creation script. The script aligns the parents’ special token configuration before merging and sets the document limit to 8192 tokens.

Evaluate Your Merge

Run the same evaluator on each parent and the merged model, using matching prompts, length limits, and query expansion settings. The default MultiVectorNanoBEIREvaluator covers all 13 NanoBEIR datasets:

from sentence_transformers.multi_vector_encoder.evaluation import MultiVectorNanoBEIREvaluator

evaluator = MultiVectorNanoBEIREvaluator(batch_size=4, corpus_chunk_size=128)
results = evaluator(merged)
print(results[evaluator.primary_metric])

Other Model Types

The same API is available for all four model types: