Basics on Diffusion Models

1. Introduction

Denoising diffusion probabilistic models (DDPM) are a class of generative models inspired by an idea from non-equilibrium statistical physics, which states:

We can gradually convert one distribution into another using a Markov chain.

The essential idea is to first systematically and gradually destroy the structure of the data distribution through a diffusion process. Then, we learn a model for the reverse process to restore the data. This two-step procedure yields a highly flexible generative model with the following advantages: 1) it is easy to sample from the learned distribution, 2) the data becomes more “explainable” because the learned distribution is simpler than the original one, and 3) samples become more diverse since it is easier to avoid the “mode collapse” effect observed in GANs. Here, we will discuss diffusion models with a focus on image generation.

2. An Overview of the Diffusion Process

The diffusion generative modeling framework comprises two steps: 1) a forward diffusion process iteratively injects noise into the image, and 2) a reverse diffusion process iteratively denoises the image back to the original data. While the first step can be easily defined, the second step requires a model to learn how to denoise the image into a meaningful sample from the original distribution.

Forward diffusion process

The goal of the forward process is to convert a complex distribution into a distribution that is tractable and easier to sample from, while achieving some level of “explainability”.

The forward process starts by slowly and iteratively adding random noise to corrupt the image. The goal is to move away from its original and complex distribution. Noise is iteratively injected into the image following a schedule that controls the amount of noise added at each step. After adding noise to the image for \(T\) steps, the image becomes unrecognizable and belongs to a different distribution. For example, images of cats will eventually belong to an isotropic Gaussian after iteratively adding Gaussian noise.

Reverse diffusion process

The goal of the reverse diffusion process is to recover the original image distribution from the less complex and more tractable data distribution. This is achieved by iteratively applying a denoising model to the corrupted input image. At each step \(t\) (in reverse), the model predicts the denoised image \(x_{t-1}\) from input \(x_t\).

Why do we need to iteratively denoise the data? Why not go directly from noise to a true data sample in a single step? Predicting these small perturbations is more tractable than attempting to describe the full data distribution in a single step. Think of it as going back from one subspace to another by following the traces of the original path.

3. Some Formality

In a diffusion process, both forward and backward steps are modeled by a Markov process. The forward process, defined by the transitions \(q(x_t|x_{t-1})\), is modeled by a Markov chain that gradually adds Gaussian noise according to a fixed variance schedule. The reverse process \(p_{\theta}(x_{t-1}|x_t)\) is also a Markov chain with learned distribution parameters \(\theta\) that iteratively denoise the image.

Forward process

The forward process is modeled by a Markov chain that corrupts the input data with Gaussian noise according to a fixed variance schedule parameterized by \(\beta \in [0,1]\). The probability distribution of the forward diffusion process is given by:

where transition potentials are modeled by a Gaussian kernel:

Given a sufficiently large \(T\), and a stable schedule for \(\beta_t\), the sample at step \(T\) belongs to an isotropic standard normal distribution \(x_T \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\).

Variance schedule

The variance \(\beta\), also known as the “diffusion rate”, controls the injection of noise into the image. The variance \(\beta_t\) of the process at each step \(t\) is computed using a fixed schedule and is used to add Gaussian noise to the image from the previous step \(x_{t-1}\).

Given the original image \(x_0\) and a diffusion step \(t\), we can compute any \(x_t\) directly by using the reparameterization trick \(\mathbf{z} = \boldsymbol{\mu}_x + \boldsymbol{\sigma}_x\epsilon\). We define:

where \(\beta_t\) is the variance computed at step \(t\) and \(\epsilon \sim \mathcal{N}(0,1)\). Applying this formulation iteratively generates samples at each step: first for \(x_1\), then for \(x_2\), then \(x_3\), and so on.

To obtain a more efficient way to go directly from \(x_0\) to any desired \(x_t\) (for example, from \(t=0\) to \(t=4\)), we first apply a change of variables:

Using the definition of \(\alpha_t\), we obtain:

To iteratively go down to \(0\) from \(t\), we need to compute the recurrence for the previous steps \(t-1, t-2,\) etc., and substitute them to expand the recurrence. For example, for \(x_{t-1} = \sqrt{\alpha_{t-1}}x_{t-2}+\sqrt{1-\alpha_{t-1}}\epsilon_{t-1}\), we use it in the expression for \(x_t\):

The underbraced terms above are samples from two independent normal distributions \(\mathcal{N}_1(0, \alpha_t(1-\alpha_{t-1}))\) and \(\mathcal{N}_2(0, 1-\alpha_t)\). The sum of samples from these distributions is also Gaussian, with parameters \(\mathcal{N}(0, \alpha_t(1-\alpha_{t-1}) + (1-\alpha_t))\), which simplifies to \(\mathcal{N}(0, 1-\alpha_t\alpha_{t-1})\). We can therefore sample from this combined distribution using the reparameterization trick and use it in the recurrence:

Finally, by applying these substitutions for \(x_{t-2}\) down to \(t=0\), we obtain a formulation to compute noisy samples at step \(t\) directly from \(x_0\):

and its corresponding diffusion kernel:

Types of \(\beta_t\) scheduling

In the diffusion process, the noise schedule controls the variance added at each step \(\beta_t\). Intuitively, it defines how fast we “destroy” the image. The schedule directly impacts model learning, as we want to make the denoising problem well-conditioned at each step. If noise increases too fast, early steps lose image structure too rapidly, making it hard for the model to learn global structure. If noise increases too slowly, learning becomes inefficient because there will be many redundant steps.

Linear schedule. In [2] the authors proposed a linear schedule going from \(\beta_1=10^{-4}\) up to \(\beta_T=0.02\). This was in itself an improvement over the original paper from 2015 [1]. However, this schedule is rather aggressive in the sense that images lose information too early in the process.

Cosine schedule. In [3], the authors proposed a softer schedule that prevents information loss too early in the forward process. They introduced a cosine schedule of the form:

Reverse process

The reverse process is also modeled by a Markov chain. This time, we attempt to reconstruct the input \(x_0\) from the corrupted image \(x_T\). For small \(\beta_t\), the reverse conditional \(p(x_{t-1}|x_t)\) is also a normal distribution. Since estimating \(p(x_{t-1}|x_t)\) directly is not easy because it requires using all the data, we instead use a neural network to approximate the parameters of the conditional distribution (that is, its mean and variance).

The reverse step starts from the last time step in the forward process, \(x_T\), with \(p(x_T) = \mathcal N(x_T; 0, \mathbf I)\). At each step, we apply a neural network parameterized by \(\theta\) to the input \(x_t\) in order to estimate the mean \(\boldsymbol \mu\) and covariance \(\boldsymbol \Sigma\) of the distribution at step \(t\). The reverse process is described as:

From the previous definition, we gain the intuition that the walks from \(0\) to \(T\), and vice versa, are paths between the simple and complex data distributions. These paths look noisy because of the stochastic sampling process. Note as well that to generate a single image, the model has to be executed \(T\) times.

Learning a diffusion model

The ELBO

Our goal is to find the generative model that matches the original data distribution. We estimate the parameters of the distribution using the log-likelihood. First, we define the probability that the generative model assigns to the data:

Computing this probability exactly is intractable because it requires computing all the marginals of the steps \(0, 1, ..., T\). Instead, we observe that diffusion models can be interpreted as latent variable models through the processes \(q\) and \(p_{\theta}\), similar to variational autoencoders (VAEs). This allows us to replace the intractable likelihood with a tractable surrogate objective, the Evidence Lower Bound (ELBO), which we can optimize during training.

To determine the training objective, we start by computing the relative probability, or likelihood ratio, between the forward and reverse processes (\(p_{\theta}/q\)). We then average it over the forward trajectories:

The ELBO is obtained by applying Jensen’s inequality to the log-likelihood in the expectation above. The inequality states that \({\color{blue}\mathbb{E}[f(X)] \geq f(\mathbb{E}[X])}\) for a convex function \(f\). Since \(\log\) is concave, the lower bound becomes:

To derive the learning objective, we compute the expectation over all samples in the true data distribution \(q(x_0)\) of the ELBO:

which, to minimize, we simply multiply by \(-1\) and invert the inequality.

Simplification of the training loss

The training objective can be further transformed into a much simpler formulation, making the learning process easier. We start by expanding the distributions in the expectation and applying log properties:

Our true goal during learning is to compare the “true reverse step” and the learned reverse step (through \(p_{\theta}\)). To access the true reverse step, we can manipulate the terms in the inner sum using Bayes’ rule so that it appears through \(q\). In particular, using \(q(x_t|x_{t-1}) = q(x_t)q(x_{t-1}|x_t)/q(x_{t-1})\), we obtain:

Here it is worth noting that the term \(q(x_t|x_{t-1})\) has high uncertainty because we are trying to predict \(x_{t-1}\) only from \(x_t\), that is, many clean images could have generated \(x_t\). To address this, we introduce conditioning on \(x_0\) to reduce uncertainty, make the computation tractable, and provide proper supervision (we now have a unique answer) [1].

Expanding the underbraced term, we notice that many terms cancel because the recurrence index is shifted by 1:

and this can be further simplified by combining it with the last term of the expectation:

Bringing these terms back into the training objective, we obtain:

where the colored terms can be expressed as KL divergences:

Recall that the final latent at step \(T\) in the diffusion process is a normal isotropic Gaussian distribution. Therefore, there are no learnable parameters in \(L_T\), and we can ignore this term (we will also ignore \(L_0\)).

Putting things together

The learning process involves comparing the learned reverse distribution \(p_{\theta}\) to the true reverse distribution \(q\). Specifically, we have:

• \(\mathbf{p_{\theta}(x_{t-1}|x_t)}\): The reverse process that involves predicting the parameters of a normal distribution \(\mu_{\theta}\) and \(\Sigma_{\theta}\) at each time step. However, since the variances are modeled with a known variance schedule, the potentials are approximated as in [2]: \(p_{\theta}(x_{t-1}|x_t) = {\color{purple}\mathcal{N}(x_{t-1}; \boldsymbol{\mu}_{\theta}(x_t, t), \beta_t\mathbf I)}\). Then, we are left only with estimating \(\boldsymbol{\mu}_{\theta}\).
• \(\mathbf{q(x_t| x_{t-1})}\): The forward process potentials. Here, we found that they become tractable when conditioning on \(x_0\). Moreover, we have access to the “true reverse distribution” by using Bayes’ rule and writing \(q(x_{t-1}|x_t,x_0)\). The forward process is modeled with a Gaussian with fixed variance schedule: \(q(x_{t-1}|x_t,x_0)={\color{red}\mathcal{N}(x_{t-1}; \tilde{\boldsymbol{\mu}}(x_t, x_0), \tilde\beta_t\mathbf I)}\).

Replacing \({\color{purple}\mathcal{N}}\) and \({\color{red}\mathcal{N}}\) into \(L_{t-1}\) gives:

where we used the Gaussian likelihood, applied \(\exp\), which cancels with \(\log\), and grouped terms that do not depend on \(\boldsymbol\mu_{\theta}\) or \(\tilde{\boldsymbol\mu}\) into a constant \(C\). This formulation directly motivates learning a parameterized model \({\color{purple}\boldsymbol\mu_{\theta}(x_t, t)}\) to approximate the true posterior mean \({\color{red}\tilde{\boldsymbol\mu}(x_t, x_0)}\).

Deriving \(\tilde{\boldsymbol{\mu}}\) and \(\tilde\beta_t\) requires writing the potentials using Bayes’ rule: \[ q(x_{t-1}|x_t,x_0) = \frac{q(x_t|x_{t-1},x_0)q(x_{t-1}|x_0)}{q(x_t|x_0)}, \] and then deriving the parameters of the Gaussian distribution. The final expressions provided in [2] are:

and we can use the closed form of \(x_t\) derived from \(x_t=\sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\epsilon\) to replace \(x_0\), giving:

During training, both \(x_t\) and \(x_0\) are available, which allows us to compute this posterior in closed form and use it as a supervision signal. Although \(x_0\) is used to compute the target, the model conditions only on \(x_t\), thereby learning to infer the reverse transition without direct access to \(x_0\) at inference time. As a consequence, we can choose a parameterization of our learned model as:

where \(\epsilon_{\theta}\) is now a function approximator intended to predict \(\epsilon\) from \(x_t\). This parameterization provides important advantages, as it transforms the learning problem into predicting a fixed, time-invariant distribution \(\epsilon \sim \mathcal{N}(0, \mathbf I)\), rather than one that changes with the noise level at each timestep through \(x_t\). Using this parameterization in our learning objective leads to:

where \(K= (1-\alpha_t)/ (\sigma_t\sqrt{2}\sqrt{\alpha_t(1-\bar\alpha_t}))\). The authors in [2] found that removing the scaling factor \(K\) is beneficial for sample quality. Using the formulation for \(x_t\), the training objective simplifies to:

which is a standard MSE loss.

Implementation details

Training and sampling algorithms

The figure above shows the training and sampling algorithms provided in [2]. During training, we start by picking a real example from the dataset and then deliberately corrupt it by adding noise at a randomly chosen timestep. Since we know exactly how much noise we added, the task becomes simple: we train the model to predict that noise using an MSE loss. By sampling different timesteps each time, the model gradually learns how to undo noise at every stage of the process, effectively learning all denoising steps in parallel.

At generation time, we reverse this idea. Instead of starting from a real image, we begin with pure noise and repeatedly apply the model to remove noise step by step. After many such steps, the noise is gradually transformed into a coherent data sample.

Deterministic sampling

The standard DDPM sampling procedure defines a stochastic reverse process which requires a large number of steps \(T\) (for example, 1000) to generate high-quality samples. DDIM (Denoising Diffusion Implicit Models) generalizes this process by defining a family of non-Markovian reverse processes that share the same training objective but allow deterministic sampling. Instead of sampling from a Gaussian at each step, DDIM defines a direct mapping between \(x_t\) and \(x_{t-1}\) through the predicted clean sample:

with reverse updates:

Note that DDIM uses the same training objective as DDPM, namely the noise-prediction loss, because both methods rely on the same forward noising process \(q(x_t|x_0)\). The difference lies only in the sampling procedure, not in the learned model. In practice, DDIM enables sampling with significantly fewer steps (for example, 20–50 instead of 1000), leading to much faster generation. While reducing the number of steps may slightly decrease sample diversity and, in extreme cases, quality, DDIM typically preserves high perceptual quality even with aggressive step reduction.

Sample code

Below there's sample code that illustrates the implementation of DDPM and DDIM sampling in Pytorch. While both of the sampling look similar, DDIM reduces the amount of denoising steps using uniformly spaced steps and computing an estimate of \(x_0\).

# Schedule
T = 1000
beta = torch.linspace(1e-4, 0.02, T)
alpha = 1.0 - beta
alpha_bar = torch.cumprod(alpha, dim=0)
model.eval()

# x_T ~ N(0, I)
xt = torch.randn(batch_size, channels, image_size, image_size)

for t in range(T - 1, -1, -1):
    t_batch = torch.full((batch_size,), t, dtype=torch.long)

    # epsilon_{theta}
    with torch.no_grad():
        eps = model(xt, t_batch)

    a_t = alpha[t]
    ab_t = alpha_bar[t]
    b_t = beta[t]

    # DDPM mean:
    mean = (1.0 / torch.sqrt(a_t)) * (
        xt - (b_t / torch.sqrt(1.0 - ab_t)) * eps
    )

    # update samples
    if t > 0:
        z = torch.randn_like(x)
        sigma = torch.sqrt(b_t) 
        xt = mean + sigma * z
    else:
        xt = mean
            

T = 1000
# Reduced number of sampling steps than DDPM
num_sampling_steps = 50

# Schedule
beta = torch.linspace(1e-4, 0.02, T)
alpha = 1.0 - beta
alpha_bar = torch.cumprod(alpha, dim=0)
timesteps = torch.linspace(T - 1, 0, num_sampling_steps).long()
model.eval()

# x_T ~ N(0, I)
xt = torch.randn(batch_size, channels, image_size, image_size)

for i in range(len(timesteps)):
    t = timesteps[i]
    t_prev = timesteps[i + 1] if i < len(timesteps) - 1 else torch.tensor(-1)
    t_batch = torch.full((batch_size,), t.item(), dtype=torch.long)

    with torch.no_grad():
        eps = model(xt, t_batch)

    ab_t = alpha_bar[t]

    # predict x0
    x0_pred = (xt - torch.sqrt(1.0 - ab_t) * eps) / torch.sqrt(ab_t)

    if t_prev >= 0:
        ab_prev = alpha_bar[t_prev]
        # pure DDIM deterministic (no random component) with eta=0
        xt = torch.sqrt(ab_prev) * x0_pred + torch.sqrt(1.0 - ab_prev) * eps
    else:
        xt = x0_pred
            

Architecture of the predictor

Diffusion models typically use a U-Net architecture with an encoder-decoder structure and skip connections, allowing the model to capture both global structure and fine details. In [2], the network is built from residual blocks (as shown in the figure below) in a Wide ResNet style and uses group normalization for stable training. Self-attention layers are added at intermediate resolutions (for example, \(16\times 16\)) to model long-range dependencies. The diffusion timestep \(t\) is incorporated via sinusoidal embeddings injected into each residual block, allowing the model to adapt to different noise levels.