Skip to content
Mamba Keeps a Fixed-Size Running Summary Where Attention Rereads Every Word, and Hybrids of the Two Now Run in IBM's Granite 4.0, NVIDIA's Nemotron Models, Jamba and Sarvam's Saaras V4
AI & Technology

Mamba Keeps a Fixed-Size Running Summary Where Attention Rereads Every Word, and Hybrids of the Two Now Run in IBM's Granite 4.0, NVIDIA's Nemotron Models, Jamba and Sarvam's Saaras V4

Illustration by tuput

English

IBM's Granite 4.0-H-Small has 36 Mamba-2 layers and only 4 attention layers, and IBM says the design cuts the memory needed for long inputs by more than 70%. The idea runs from a 2021 paper called S4 through Mamba in 2023 to hybrids from AI21, Zyphra, NVIDIA, IBM and Sarvam.

· · 10 min read

IBM’s Granite 4.0-H-Small, released on 2 October 2025 under the Apache 2.0 licence, has 32 billion parameters (the adjustable numbers a model learns in training), and its Hugging Face model card lists 36 Mamba-2 layers and only 4 attention layers. IBM says the design needs over 70% less memory than conventional transformer models when the input is long or many users are served at once.

Mamba is a way of building a language model that does not reread every earlier word. Albert Gu and Tri Dao posted it on 1 December 2023, and models that mix it with a few attention layers now come from AI21, Zyphra, NVIDIA, IBM and, for speech recognition, Sarvam in India. This article follows the papers and maker pages behind each step and marks every speed or memory figure that is the maker’s own.

What attention costs as text gets longer

Most chatbots are transformers, and the core of a transformer is attention, the step in which each word is compared with the words before it. Our explainer on the 2017 transformer paper covers how that works.

The cost shows up as text grows. NVIDIA’s Nemotron-H paper says that to generate one token (a word or piece of a word), a self-attention layer does work that grows with the number of tokens seen so far, and must also keep a store of earlier results, the KV cache, whose size grows in proportion to the length of the text. Our article on DeepSeek V4.1-Flash explains the KV cache in detail. IBM puts the same point another way: if the input length doubles, a transformer must do four times the calculations over the whole input.

A notebook of fixed size

Picture a reader allowed one notebook of fixed size. After each word, the reader decides what to write down and what to erase, and never looks back at the text itself. The notebook never grows, so each new word costs the same effort whether the text is ten words or a million. The price is that whatever the notebook did not keep is gone.

That analogy is this article’s own. Gu and Dao frame it in their Mamba paper as compressing the context into a smaller state: attention, they write, is effective and inefficient because it does not compress context at all, while a recurrent model with a finite state is efficient but limited by how well that state holds the context. In a state-space model, the “state” is the notebook.

S4 in 2021: state-space layers that reach very long sequences

State-space models come from control engineering, and by 2021 they had a known weakness. Gu, Goel and Ré (Stanford) wrote in the abstract of their S4 paper, posted on 31 October 2021, that conventional models “struggle to scale to very long sequences of 10000 or more steps”, and that the earlier state-space approach was too costly in computation and memory to use in general. S4 changed how the layer is set up so it could be computed far more efficiently.

The authors report 91% accuracy on sequential CIFAR-10 (an image-classification test read as one long sequence) with no data augmentation, on par with a larger 2-D ResNet. They also report solving Path-X, a task with sequences of length 16,000 that, in their words, all prior work fails on, and generation 60 times faster than Transformers on image and language modelling tasks while closing much of the quality gap. These are the authors’ results.

Mamba in 2023: letting the model choose what to keep

The S4 layer applied the same fixed rules to every input. Gu and Dao’s change was to make the layer’s parameters depend on the current input, so that the model can, in their words, “selectively propagate or forget information” along the sequence. The paper pairs this with a hardware-aware algorithm that computes the layer efficiently on a GPU (a graphics processor, the chip that runs most AI work) and with a simplified network that uses neither attention nor the usual feed-forward blocks.

The paper reports that memory and computation grow linearly with sequence length in training and that each step of generation takes constant time because there is no cache of earlier items. In its speed test, run on one A100 80 GB GPU with a 2,048-token prompt and 128 generated tokens, Mamba reached 4 to 5 times the inference throughput (words produced per second) of a Transformer of similar size, partly because with no KV cache it could use much larger batches. The abstract rounds this to 5 times. The authors also report that their 3-billion-parameter Mamba beat Transformers of the same size and matched ones twice its size, on their tests. They released code and trained checkpoints.

Mamba-2 in 2024: attention and state-space models turn out to be close relatives

Dao and Gu’s second paper, posted on 31 May 2024 as “Transformers are SSMs”, argues the two families are closely related. Their “state space duality” framework connects state-space models and variants of attention through a class of structured matrices. The practical result is Mamba-2, whose core layer the authors say is 2 to 8 times faster than Mamba’s, and which can hold a much larger running summary. They report that a 2.7-billion-parameter Mamba-2 trained on 300 billion tokens of the Pile, an open text collection, outperformed the original Mamba 2.8B and the Pythia 2.8B and 6.9B models trained on the same data.

Why the notebook is not enough on its own

A fixed notebook can lose detail, which is why the hybrids keep some attention. The Jamba paper tested it directly. At 1.3 billion parameters and 250 billion training tokens, a pure attention model, a pure Mamba model and a mix scored as follows on IMDB, a test of labelling film reviews as positive or negative.

ModelIMDBQuACNarrativeQA
Attention only84.127.945.8
Mamba only48.820.227.7
Attention and Mamba mixed90.926.643.7

QuAC and NarrativeQA are question-answering tests. The Jamba authors report that the pure Mamba model often broke the answer format on IMDB, replying with things like “Very Good” or “3/10” instead of the two permitted labels, and they link this to in-context learning, the ability to pick up a task from examples in the prompt. At 7 billion parameters after 50 billion tokens, they found pure Mamba “quite competitive” but slightly behind attention, and the mixed model ahead of both. The Zamba2 report and IBM’s announcement make the same point, that pure state-space models do less well at in-context learning.

The Mamba-2 paper tested how much attention a hybrid needs. On a 350-million-parameter model with 48 layers, perplexity (a score of how surprised the model is by real text, where lower is better) was 8.60 for pure Mamba-2, 8.68 for an all-Transformer recipe and 8.26 with six attention layers. The authors found that about 10% of layers being attention worked best. They suggest, as a hypothesis, that the state-space layers handle general sequence processing while attention layers work as a lookup that can jump back to earlier tokens instead of forcing the model to squeeze everything into its state.

Four hybrids in the makers’ own words

The table gives the attention share each source states. The sources count layers differently (Nemotron-H lists feed-forward layers separately, for example), so the shares are close but not identical measures.

ModelDateAttention in the stack, as the source states it
Jamba (AI21)28 March 20241 attention layer for every 7 Mamba layers
Zamba2 (Zyphra)22 November 2024Mamba-2 backbone with one or two shared attention blocks, whose weights are reused
Nemotron-H (NVIDIA)4 April 2025roughly 8% of layers: 4 of 52 in the 8B model, 10 of 118 in the 56B
Granite 4.0-H (IBM)2 October 20254 attention and 36 Mamba-2 layers in all three hybrid sizes, a 9 to 1 ratio

Jamba interleaves attention and Mamba layers and adds mixture of experts (a design in which each word is routed to a few sub-networks, covered in our explainer). The released model has 52 billion parameters in total, 12 billion active at a time, and supports 256,000 tokens of context under the Apache 2.0 licence. The paper’s table puts the KV cache at 256,000 tokens in 16-bit numbers at 4 GB for Jamba, against 32 GB for Mixtral and 128 GB for Llama-2 6.7B. The authors also report 3 times Mixtral’s throughput at 128,000 tokens on four A100 GPUs. AI21 calls Jamba the first production-grade Mamba-based model, which is the company’s own description.

Zyphra’s Zamba2 comes in 1.2, 2.7 and 7.4 billion parameter sizes. The report says the weights are open under Apache 2.0, that the training dataset (Zyda-2) is open as well, and that models were trained on up to three trillion tokens. Zyphra says the models match or beat the leading open-weights models of their size with gains in latency, throughput and memory use, which is the company’s own comparison.

NVIDIA’s Nemotron-H paper says it replaced most attention layers with Mamba layers that need constant memory per generated token. NVIDIA reports up to 3 times faster inference than similar-sized transformers at similar accuracy. The test used a 65,536-token input and 1,024 generated tokens on H100 GPUs, NVIDIA’s own early implementation for Nemotron-H, and the vLLM serving software for the comparison models. For the 56B model NVIDIA measured 2.4 times the output per GPU of Qwen-2.5-72B and Llama-3.1-70B. The paper adds that transformers have been heavily optimised for years and that it expects further speedups for Nemotron-H.

NVIDIA’s later Nemotron 3 family uses a mixture-of-experts hybrid Mamba-Transformer design, according to its family page (published 15 December 2025) and white paper (24 December 2025). The page lists context up to 1 million tokens and describes the Nano model as 31.6 billion parameters in total with 3.2 billion active. NVIDIA reports 3.3 times the throughput of Qwen3-30B-A3B at 8,000 tokens in and 16,000 out on one H200 GPU. The page, read on 19 November 2026, carries an update saying Nemotron 3 Ultra is released and gives no date for it.

IBM’s Granite 4.0 has three hybrid models: H-Small (32 billion parameters, 9 billion active), H-Tiny (7 billion, 1 billion active) and H-Micro (3 billion, dense), plus a 3-billion-parameter conventional transformer for platforms that cannot yet run hybrids. IBM says the hybrids use no positional encoding, the added signal that tells a transformer the order of words, because Mamba reads tokens in order. IBM also says the hybrid design contributes to efficiency, but that most of Granite 4.0’s accuracy gains over Granite 3.3 come from better training data and methods, since even its conventional 3B model beats the older 8B model. The 70% memory figure is IBM’s own claim against conventional transformer models.

Sarvam’s Saaras V4

Sarvam published Saaras V4 on 25 September 2026. Its post says the speech-recognition system pairs an audio encoder with a language-model decoder of 3 billion parameters, described as a “hybrid state space model” trained from scratch in-house. The post does not name Mamba or the exact layer design, offers Saaras V4 through a REST API and a batch API, and does not describe a release of the model weights. Our article on Saaras V4 covers its 22 languages and test results.

What the sources do not show

Every speed figure above is the maker’s own and was measured on a different model, on different hardware, against different baselines: 5 times in the Mamba paper, 2 to 8 times for the Mamba-2 layer, 3 times in Nemotron-H and 3.3 times for Nemotron 3 Nano. They are not comparable with one another, and this article does not add them up.

None of the pages opened gives an independent head-to-head of a hybrid against a pure transformer at the scale of today’s largest models, and none says that a chat product from another lab uses Mamba layers. The Saaras V4 architecture is Sarvam’s description of its own system. Granite 4.0’s announcement says its models were trained on samples up to 512,000 tokens long and validated on tasks up to 128,000 tokens.

Share
Copied!

Sources & further reading

  1. arXiv: Gu, Goel and Ré, Efficiently Modeling Long Sequences with Structured State Spaces (S4, 31 October 2021)
  2. arXiv: Gu and Dao, Mamba: Linear-Time Sequence Modeling with Selective State Spaces (1 December 2023)
  3. arXiv: Dao and Gu, Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality (31 May 2024)
  4. arXiv: Lieber et al., Jamba: A Hybrid Transformer-Mamba Language Model (28 March 2024)
  5. AI21: Introducing Jamba (28 March 2024)
  6. arXiv: Glorioso et al., The Zamba2 Suite: Technical Report (Zyphra, 22 November 2024)
  7. arXiv: NVIDIA, Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models (4 April 2025)
  8. NVIDIA Research: NVIDIA Nemotron 3 Family of Models (published 15 December 2025)
  9. arXiv: NVIDIA, NVIDIA Nemotron 3: Efficient and Open Intelligence (24 December 2025)
  10. IBM: IBM Granite 4.0, hyper-efficient, high performance hybrid models for enterprise (2 October 2025)
  11. Hugging Face: ibm-granite/granite-4.0-h-small model card
  12. Sarvam AI: Introducing Saaras V4 (25 September 2026)

Researched and written with the help of AI tools and edited for accuracy. Provided for general information and discussion only, not professional advice. See our editorial standards and disclaimer. Spotted an error? Tell us.

#mamba#state space models#hybrid models#jamba#nemotron#ibm granite#sarvam ai#long context

Enjoyed this? Get the next one.

One good read at a time, straight to your inbox. No spam, unsubscribe anytime.

More in AI & Technology
Eight Google Authors Posted the Transformer Paper on 12 June 2017, and Its Big Model Scored 28.4 BLEU in English to German After 3.5 Days on Eight GPUs
A walk through the design behind today's chatbots, using the paper's own tables and Google's own example of a word that has to look at a river. It also covers the parts of the 2017 design that later models kept, changed or dropped.
DeepSeek's V4.1-Flash, an Open-Weight 552-Billion-Parameter Model Released on 10 September 2026, Cuts Its Per-Token Memory Cache to 890 Bytes, About a Quarter of Its Predecessor's
A model's short-term memory of a long conversation is expensive to keep. DeepSeek's September release compresses it by design, and its own report says where the model still trails. Here is the mechanism, the timeline from 2024, and what outsiders have checked.
Sarvam's 105-Billion-Parameter Model Computes With 10.3 Billion of Them for Each Token, Using the Mixture-of-Experts Design That Runs From a 1991 Paper to Mixtral and DeepSeek
A small router sends every word to a few of 128 sub-networks, so most of the model sits idle for any one word. Here is where the design came from, how Mixtral, DeepSeek and Sarvam use it, and what it does not save.
← all articles