Context window explained

The context window is the maximum length of input a large language model (LLM) can consider at once. In the development and maturation of LLM technology expanding the context window has been a major goal.[1] [2] The length of a context window is measured in tokens. In 2025, the Gemini LLM had the largest context window with two million tokens.[3]

In some models the context length is limited by the size of inputs during the training runs.[4] However, attention mechanisms can be adopted to allow LLMs to interpret sequences that are much longer than those observed at training time.[5]

Notes and References

  1. Ratner . Nir . Levine . Yoav . Belinkov . Yonatan . Ram . Ori . Magar . Inbal . Abend . Omri . Karpas . Ehud . Shashua . Amnon . Leyton-Brown . Kevin . Shoham . Yoav . Parallel Context Windows for Large Language Models . Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 2023 . 6383–6402 . 10.18653/v1/2023.acl-long.352.
  2. Dong . Zican . Li . Junyi . Men . Xin . Zhao . Wayne Xin . Wang . Bingning . Tian . Zhen . Chen . Weipeng . Wen . Ji-Rong . Exploring context window of large language models via decomposed positional vectors . Proceedings of the 38th International Conference on Neural Information Processing Systems . 10 December 2024 . 37 . 10320–10347 . Curran Associates Inc..
  3. Web site: Yeung . Ken . 2024-05-14 . Google announces Gemini 1.5 Flash, a rapid multimodal model with a 1M context window . 2025-08-26 . VentureBeat . en-US.
  4. Wu . S . 2023 . BloombergGPT: A Large Language Model for Finance . 2303.17564 . LG.
  5. Press. Ofir . 2021. Train short, test long: Attention with linear biases enables input length extrapolation . 2108.12409 . LG.