Dream 7B: A Powerful and Open Diffusion Language Model
The last few years have shown that large language models (LLMs) have impressive capabilities in analyzing, understanding and generating text in a human-like way — and they’re increasingly making inroads into people’s daily lives, for better or for worse.
But even the most powerful LLMs have limitations, especially when it comes to tasks that require bidirectional and long-term reasoning, complex planning, and maintaining coherence over extended contexts. These limitations arise because many of the current LLMs are based on a statistical technique called autoregression, meaning they are designed to process and predict text tokens in a sequential, left-to-right manner. With more complex tasks, the computational requirements and potential for latency increases significantly.
However, there are now a growing number of diffusion-based large language models that are making their mark, challenging the autoregressive paradigm underpinning many of today’s LLMs.
Dream 7B is one such diffusion language model (DLM) that leverages the same approach as diffusion models that generate images. Created by researchers from University of Hong Kong and Noah’s Ark Lab at Huawei, Dream 7B (which stands for Diffusion REAsoning Model) is one of the most powerful open diffusion LLMs to date, matching or even exceeding state-of-the-art autoregressive models (ARMs) of similar size on general tasks, math and coding.
Autoregression vs. Diffusion
In contrast to conventional autoregressive language models that generate outputs in a sequential manner, diffusion language models will gradually “de-noise” a sequence of words that are initially concealed by noise. It’s the same method used by diffusion-based, image-generating AI models, and in both cases, this de-noising process occurs in parallel, meaning that it is much faster than the sequential, step-by-step approach of autoregressive models.
How a diffusion-based model de-noises in order to get an output (source).
Thanks to this parallel processing of tokens, Dream 7B is performs significantly better than its autoregressive counterparts in these areas:
- Bidirectional context modeling: Dream 7B has a more holistic view in considering the context of a word in a sentence. Rather than just analyzing the phrase in a unidirectional fashion, Dream 7B can simultaneously evaluate both the beginning and the end of a sentence, boosting accuracy, contextual awareness and retention, as well as extending coherence for more complex tasks.
- Flexible text generation: Text can be generated in any random order, rather than sequentially. That means diffusion models like Dream 7B are better suited to a wider range of applications like text completion, infilling and controlled generation.
- Improved planning abilities: The ability to reason and plan in multiple steps, or solving problems with multiple parameters, is noticeably enhanced when compared to ARMs.
- Dynamic control: Dream 7B allows users to adjust the number of steps in the diffusion process in order to optimize output quality and computational efficiency.
Architecture and Features of Dream 7B
According to the researchers, Dream 7B’s architecture is based on the team’s previous work in reparameterized discrete diffusion models (RDM), which introduces a different statistical approach that simplifies training and allows for flexible sampling.
Autoregressive Weight Initialization
In addition, Dream 7B expands upon novel enhancements behind DiffuLLaMA, leveraging that model’s strategy in readapting certain elements and training paradigms of autoregressive models into diffusion text models. This means that instead of training the model from scratch, Dream 7B was initialized using the model weights from Qwen2.5 7B, thus reducing training time and cost.
The team also found the rate of learning to be a significant factor in ensuring that language knowledge was preserved from the initialization stage, without subsequently inhibiting the diffusion training process.
“If the learning rate is set too high, it can quickly wash away the left-to-right knowledge in the initial weights, providing little help in the diffusion training; while if it’s set too low, it can hinder diffusion training,” explained the team.
“We meticulously selected this parameter along with the other training parameters. Thanks to the existing left-to-right knowledge in the autoregressive model, the diffusion model’s any-order learning can be accelerated, significantly reducing the tokens and computation required for pre-training.”
Context-Adaptive Token-Level Noise Rescheduling Mechanism
Another innovation introduced by Dream 7B is a context-adaptive token-level noise rescheduling mechanism, which dynamically readjusts the noise level for each token depending on its context.
In contrast to other diffusion training paradigms that apply uniform levels of noise to an entire sentence, this is a more targeted and granular approach that helps the model to learn more effectively, and to generate more accurate outputs that take into account the surrounding context.
How Does Dream 7B Stack Up?
In evaluating Dream 7B against well-known autoregressive models like LLaMA3 8B and Qwen2.5 7B in general language tasks, mathematical reasoning and code generation, the team found that Dream 7B’s performance either matched or exceeded that of these traditional LLMs.
Additionally, the team discovered that Dream 7B’s capabilities significantly surpassed those of similarly sized models in tasks like Countdown and Sudoku, which both require complex reasoning and intensive planning.
“In our previous work, we demonstrated that text diffusion exhibits superior planning capabilities in the small-scale, task-specific context,” noted the research team. “However, it remains uncertain whether a general, scaled diffusion model possesses similar abilities. Now, with Dream 7B, we can better answer this question.”
Examples of Qwen 2.5 7B and Dream 7B in three planning tasks.
Looking Ahead: More Diffusion LLMs
Diffusion-based text generation models like Dream 7B have only recently emerged, but they are already poised to upend the dominance of autoregressive LLMs in the future. As the AI field continues to develop and grow, diffusion language models could potentially be the next stage of evolution for embodied AI, autonomous AI agents, and decision-making systems that require extensive long-term planning and organization.
Find out more on Dream 7B’s project page, and check out the model repository on GitHub.
