Linkup Research Unveils SPARSEUP: An Open-Source Sparse Embedding Model with 149M Parameters
Linkup Research released SPARSEUP, an open-source learned sparse embedding model under Apache 2.0, built on a 149M-parameter ModernBERT backbone. It averages 56.4 nDCG@10 on BEIR-13, reportedly the best among public vocabulary-based sparse encoders under 150M parameters. Trained with contrastive learning on LightOn's fine-tuning data using logit shifting, per-position top-k, and case folding, it integrates with inverted indexes and is available on Hugging Face.
If you work on search or retrieval and want a lightweight sparse embedding model you can actually deploy, Linkup Research's new release is worth a look. The team has introduced SPARSEUP, an open-source learned sparse encoder built on a 149M-parameter ModernBERT backbone and released under the Apache 2.0 license. According to Linkup, it averages 56.4 nDCG@10 on BEIR-13, which they describe as the best result they know of among public vocabulary-based sparse encoders under 150M parameters. Weights are available on Hugging Face, and the model loads through Transformers or Sentence Transformers with trust_remote_code=True.
Why Sparse, and Why This Release Matters
Most open retrieval models today are dense, meaning every piece of text becomes a single vector. Sparse models work differently: instead of one dense vector, they produce weights across a vocabulary, where each dimension corresponds to an actual token. That design brings real advantages. Sparse vectors slot neatly into inverted indexes, remain human-readable, and often handle rare words better than dense approaches.
What prompted the release? LightOn's recent DenseOn and LateOn launch. LightOn published open training data, a full training recipe, a dense model, and a late-interaction model—but no sparse model. SPARSEUP fills that gap. Because it uses the same backbone family and the same fine-tuning data, researchers can now compare all three retrieval styles on equal footing.
How SPARSEUP Was Built
Training begins from LateOn-unsupervised, a checkpoint that lacked an MLM head, so the team grafted ModernBERT's original one back on. Fine-tuning then used LightOn's fine-tuning mixture with contrastive learning only. Each query is trained with 7 hard negatives drawn from a pool of 50, plus in-batch negatives. Notably, there's no cross-encoder distillation, and the whole training run fits on a single H100.
The tricky part was taming SPLADE-style expansion. A vanilla SPLADE setup on this backbone produced enormous bags of terms stuffed with stopwords. Linkup addressed this with three changes:
- Logit shifting: The encoder applies log(1 + ReLU(x - 15)). ModernBERT's MLM logits were too high out of the box, saturating the log and making the term bags dense from the start.
- Per-position top-k: Each input token keeps only its 12 strongest vocabulary dimensions before max pooling. This limits expansion per token rather than capping total vector size.
- Case folding: Byte-level BPE treats heat, Heat, Ġheat, and ĠHeat as separate ids. SPARSEUP merges them into one id and keeps the largest weight, shrinking output dimensions from roughly 50k to about 34k.
Queries and documents receive [Q] and [D] prefixes respectively, and scoring is a simple dot product. Evaluation uses a max length of 128 tokens for queries and 512 for documents.
Benchmark Results
On BEIR-13 (nDCG@10, excluding MS MARCO), per the model card, SPARSEUP compares as follows:
| Model | BEIR-13 average |
|---|---|
| SPARSEUP | 56.4 |
| opensearch-neural-sparse-encoding-doc-v3-gte | 54.6 |
| opensearch-neural-sparse-encoding-v1 | 52.44 |
| ModernBERT-VT | 52.4 |
| splade-v3 | 51.7 |
| granite-embedding-30m-sparse | 50.6 |
| LACONIC-1B (1B parameters, larger class) | 58.7 |
The controlled comparison is less flattering. With the backbone and data held constant, LateOn scores 58.9 and DenseOn 57.9, versus 56.4 for SPARSEUP. It also matters that SPARSEUP uses approximate Seismic search while LightOn reports exact search. The model does win on ArguAna and Touché and edges out DenseOn on HotpotQA, but it trails on more semantic datasets, with FiQA showing the largest gap. DBPedia is another weak point.
On decontaminated BEIR, the gap to DenseOn narrows to just 0.17 points. Linkup does caution that decontaminated NQ and MS MARCO contain only 21 and 46 queries respectively, so those figures are noisy.
Speed and Sparsity
On MS MARCO, SPARSEUP averages 47 non-zero terms per query and 190 per document, compared with 25 and 170 for SPLADE-v3. Paired with the Seismic inverted index, it hits over 97% recall against exact search at roughly 380 microseconds per query, single-threaded. Linkup notes that inflating vector size could add 1 to 2 BEIR points, but the team deliberately chose to stay sparse.
Key Takeaways
- SPARSEUP is Linkup Research's first open model: a 149M-parameter sparse encoder under Apache 2.0.
- It scores 56.4 nDCG@10 on BEIR-13, reportedly the best among public sparse encoders under 150M parameters.
- Three engineering fixes make it work: a logit shift of 15, top-12 expansion per token, and case folding.
- With identical training data, it trails DenseOn by 1.52 points and LateOn by 2.5 on BEIR-13.
- It achieves over 97% recall at around 380µs per query using Seismic on MS MARCO.
For teams that want sparse retrieval in a compact, permissively licensed package, SPARSEUP is a strong new option—just keep in mind the dense and late-interaction alternatives still lead on raw scores under identical conditions.
The post Linkup Research Releases SPARSEUP: A 149M-Parameter Open-Source Sparse Embedding Model appeared first on MarkTechPost.
Comments (0)