Diffusion model in web browser

Online Demo! Run it in your web browser

• 第一集: https://www.bilibili.com/video/BV1tz4y1h7q1 | 正态分布 | 基本设定 | 公式推导 |
• 第二集: https://www.bilibili.com/video/BV1xQ4y1w7ex | 神经网络 | 概率空间 | 边缘概率 | 各向同性高斯分布 |
• 第三集: https://www.bilibili.com/video/BV1hZ421y7id | 三维动画展示全过程
• 第四集: https://www.bilibili.com/video/BV1gK421b7W9 | 神经网络的学习目标以及训练
• 第五集: https://www.bilibili.com/video/BV12y421z7Mh/ | 花絮 | 热度图制作细节揭秘

Youtube

UPDATE:

• 2024-07-14 : Update online sample to use WebGPU if possible
• 2024-07-15 : Added DDIM sampling method

---

1. DDPM Introduction

• $q$ - a fixed (or predefined) forward diffusion process of adding Gaussian noise to an image gradually, until ending up with pure noise
• $p_{\theta }$ - a learned reverse denoising diffusion process, where a neural network is trained to gradually denoise an image starting from pure noise, until ending up with an actual image.

Both the forward and reverse process indexed by $t$ happen for some number of finite time steps $T$ (the DDPM authors use $T$=1000). You start with $t=0$ where you sample a real image $x_0$ from your data distribution, and the forward process samples some noise from a Gaussian distribution at each time step $t$, which is added to the image of the previous time step. Given a sufficiently large $T$ and a well behaved schedule for adding noise at each time step, you end up with what is called an isotropic Gaussian distribution at $t=T$ via a gradual process

2. Forward Process $q$

$$x_0\stackrel{q(x_1|x_0)}{\to }{x}_1\stackrel{q(x_2|x_1)}{\to }{x}_2\to \cdots \to x_{T-1}\stackrel{q(x_t|x_{t-1})}{\to }{x}_T$$

This process is a markov chain, $x_t$ only depends on $x_{t-1}$. $q(x_t|x_{t-1})$ adds Gaussian noise at each time step $t$, according to a known variance schedule ${\beta }_t$

$$x_t=\sqrt{1-{\beta }_t}\times x_{t-1}+\sqrt{{\beta }_t}\times {ϵ}_t$$

• ${\beta }_t$ is not constant at each time step $t$. In fact one defines a so-called "variance schedule", which can be linear, quadratic, cosine, etc.

$$0<{\beta }_1<{\beta }_2<{\beta }_3<\cdots <{\beta }_T<1$$

• ${ϵ}_t$ Gaussian noise, sampled from standard normal distribution.

$$x_t=\sqrt{1-{\beta }_t}\times x_{t-1}+\sqrt{{\beta }_t}\times {ϵ}_t$$

Define $a_t=1-{\beta }_t$

$$x_t=\sqrt{a_t}\times x_{t-1}+\sqrt{1-a_t}\times {ϵ}_t$$

2.1 Relationship between $x_t$ and $x_{t-2}$

$$x_{t-1}=\sqrt{a_{t-1}}\times x_{t-2}+\sqrt{1-a_{t-1}}\times {ϵ}_{t-1}$$

$$\Downarrow$$

$$x_t=\sqrt{a_t}(\sqrt{a_{t-1}}\times x_{t-2}+\sqrt{1-a_{t-1}}{ϵ}_{t-1})+\sqrt{1-a_t}\times {ϵ}_t$$

$$\Downarrow$$

$$x_t=\sqrt{a_t{a}_{t-1}}\times x_{t-2}+\sqrt{a_t(1-a_{t-1})}{ϵ}_{t-1}+\sqrt{1-a_t}\times {ϵ}_t$$

Because $N({\mu }_1,{\sigma }_1^2)+N({\mu }_2,{\sigma }_2^2)=N({\mu }_1+{\mu }_2,{\sigma }_1^2+{\sigma }_2^2)$

Proof

$$x_t=\sqrt{a_t{a}_{t-1}}\times x_{t-2}+\sqrt{a_t(1-a_{t-1})+1-a_t}\times ϵ$$

$$\Downarrow$$

$$x_t=\sqrt{a_t{a}_{t-1}}\times x_{t-2}+\sqrt{1-a_t{a}_{t-1}}\times ϵ$$

2.2 Relationship between $x_t$ and $x_{t-3}$

$$x_{t-2}=\sqrt{a_{t-2}}\times x_{t-3}+\sqrt{1-a_{t-2}}\times {ϵ}_{t-2}$$

$$\Downarrow$$

$$x_t=\sqrt{a_t{a}_{t-1}}(\sqrt{a_{t-2}}\times x_{t-3}+\sqrt{1-a_{t-2}}{ϵ}_{t-2})+\sqrt{1-a_t{a}_{t-1}}\times ϵ$$

$$\Downarrow$$

$$x_t=\sqrt{a_t{a}_{t-1}{a}_{t-2}}\times x_{t-3}+\sqrt{a_t{a}_{t-1}(1-a_{t-2})}{ϵ}_{t-2}+\sqrt{1-a_t{a}_{t-1}}\times ϵ$$

$$\Downarrow$$

$$x_t=\sqrt{a_t{a}_{t-1}{a}_{t-2}}\times x_{t-3}+\sqrt{a_t{a}_{t-1}-a_t{a}_{t-1}{a}_{t-2}}{ϵ}_{t-2}+\sqrt{1-a_t{a}_{t-1}}\times ϵ$$

$$\Downarrow$$

$$x_t=\sqrt{a_t{a}_{t-1}{a}_{t-2}}\times x_{t-3}+\sqrt{(a_t{a}_{t-1}-a_t{a}_{t-1}{a}_{t-2})+1-a_t{a}_{t-1}}\times ϵ$$

$$\Downarrow$$

$$x_t=\sqrt{a_t{a}_{t-1}{a}_{t-2}}\times x_{t-3}+\sqrt{1-a_t{a}_{t-1}{a}_{t-2}}\times ϵ$$

2.3 Relationship between $x_t$ and $x_0$

• $x_t=\sqrt{a_t{a}_{t-1}}\times x_{t-2}+\sqrt{1-a_t{a}_{t-1}}\times ϵ$
• $x_t=\sqrt{a_t{a}_{t-1}{a}_{t-2}}\times x_{t-3}+\sqrt{1-a_t{a}_{t-1}{a}_{t-2}}\times ϵ$
• $x_t=\sqrt{a_t{a}_{t-1}{a}_{t-2}{a}_{t-3}...a_{t-(k-2)}{a}_{t-(k-1)}}\times x_{t-k}+\sqrt{1-a_t{a}_{t-1}{a}_{t-2}{a}_{t-3}...a_{t-(k-2)}{a}_{t-(k-1)}}\times ϵ$
• $x_t=\sqrt{a_t{a}_{t-1}{a}_{t-2}{a}_{t-3}...a_2{a}_1}\times x_0+\sqrt{1-a_t{a}_{t-1}{a}_{t-2}{a}_{t-3}...a_2{a}_1}\times ϵ$

$$\bar{a}{t} := a{t}a_{t-1}a_{t-2}a_{t-3}...a_{2}a_{1}$$

$$x_t=\sqrt{{\overline{a}}_t}\times x_0+\sqrt{1-{\overline{a}}_t}\times ϵ,ϵ\sim N(0,I)$$

$$\Downarrow$$

$$q(x_t|x_0)=\frac{1}{\sqrt{2\pi }\sqrt{1-{\overline{a}}_t}}{e}^{(-\frac{1}{2}\frac{(x_t-\sqrt{{\overline{a}}_t}{x}_0{)}^2}{1-{\overline{a}}_t})}$$

3.Reverse Process $p$

Because $P(A|B)=\frac{P(B|A)P(A)}{P(B)}$

$$p(x_{t-1}|x_t,x_0)=\frac{q(x_t|x_{t-1},x_0)\times q(x_{t-1}|x_0)}{q(x_t|x_0)}$$

$x_t=\sqrt{a_t}{x}_{t-1}+\sqrt{1-a_t}\times ϵ$	~	$N(\sqrt{a_t}{x}_{t-1},1-a_t)$
$x_{t-1}=\sqrt{{\overline{a}}_{t-1}}{x}_0+\sqrt{1-{\overline{a}}_{t-1}}\times ϵ$	~	$N(\sqrt{{\overline{a}}_{t-1}}{x}_0,1-{\overline{a}}_{t-1})$
$x_t=\sqrt{{\overline{a}}_t}{x}_0+\sqrt{1-{\overline{a}}_t}\times ϵ$	~	$N(\sqrt{{\overline{a}}_t}{x}_0,1-{\overline{a}}_t)$

$$q(x_t|x_{t-1},x_0)=\frac{1}{\sqrt{2\pi }\sqrt{1-a_t}}{e}^{(-\frac{1}{2}\frac{(x_t-\sqrt{a_t}{x}_{t-1}{)}^2}{1-a_t})}$$

$$q(x_{t-1}|x_0)=\frac{1}{\sqrt{2\pi }\sqrt{1-{\overline{a}}_{t-1}}}{e}^{(-\frac{1}{2}\frac{(x_{t-1}-\sqrt{{\overline{a}}_{t-1}}{x}_0{)}^2}{1-{\overline{a}}_{t-1}})}$$

$$q(x_t|x_0)=\frac{1}{\sqrt{2\pi }\sqrt{1-{\overline{a}}_t}}{e}^{(-\frac{1}{2}\frac{(x_t-\sqrt{{\overline{a}}_t}{x}_0{)}^2}{1-{\overline{a}}_t})}$$

$$\frac{q(x_t|x_{t-1},x_0)\times q(x_{t-1}|x_0)}{q(x_t|x_0)}=\left[\frac{1}{\sqrt{2\pi }\sqrt{1-a_t}}{e}^{(-\frac{1}{2}\frac{(x_t-\sqrt{a_t}{x}_{t-1}{)}^2}{1-a_t})}\right]\ast \left[\frac{1}{\sqrt{2\pi }\sqrt{1-{\overline{a}}_{t-1}}}{e}^{(-\frac{1}{2}\frac{(x_{t-1}-\sqrt{{\overline{a}}_{t-1}}{x}_0{)}^2}{1-{\overline{a}}_{t-1}})}\right]÷\left[\frac{1}{\sqrt{2\pi }\sqrt{1-{\overline{a}}_t}}{e}^{(-\frac{1}{2}\frac{(x_t-\sqrt{{\overline{a}}_t}{x}_0{)}^2}{1-{\overline{a}}_t})}\right]$$

$$\Downarrow$$

$$\frac{\sqrt{2\pi }\sqrt{1-{\overline{a}}_t}}{\sqrt{2\pi }\sqrt{1-a_t}\sqrt{2\pi }\sqrt{1-{\overline{a}}_{t-1}}}{e}^{[-\frac{1}{2}(\frac{(x_t-\sqrt{a_t}{x}_{t-1}{)}^2}{1-a_t}+\frac{(x_{t-1}-\sqrt{{\overline{a}}_{t-1}}{x}_0{)}^2}{1-{\overline{a}}_{t-1}}-\frac{(x_t-\sqrt{{\overline{a}}_t}{x}_0{)}^2}{1-{\overline{a}}_t})]}$$

$$\Downarrow$$

$$\frac{1}{\sqrt{2\pi }\left(\frac{\sqrt{1-a_t}\sqrt{1-{\overline{a}}_{t-1}}}{\sqrt{1-{\overline{a}}_t}}\right)}\exp [-\frac{1}{2}(\frac{(x_t-\sqrt{a_t}{x}_{t-1}{)}^2}{1-a_t}+\frac{(x_{t-1}-\sqrt{{\overline{a}}_{t-1}}{x}_0{)}^2}{1-{\overline{a}}_{t-1}}-\frac{(x_t-\sqrt{{\overline{a}}_t}{x}_0{)}^2}{1-{\overline{a}}_t})]$$

$$\Downarrow$$

$$\frac{1}{\sqrt{2\pi }\left(\frac{\sqrt{1-a_t}\sqrt{1-{\overline{a}}_{t-1}}}{\sqrt{1-{\overline{a}}_t}}\right)}\exp [-\frac{1}{2}(\frac{x_t^2-2\sqrt{a_t}{x}_t{x}_{t-1}+a_t{x}_{t-1}^2}{1-a_t}+\frac{x_{t-1}^2-2\sqrt{{\overline{a}}_{t-1}}{x}_0{x}_{t-1}+{\overline{a}}_{t-1}{x}_0^2}{1-{\overline{a}}_{t-1}}-\frac{(x_t-\sqrt{{\overline{a}}_t}{x}_0{)}^2}{1-{\overline{a}}_t})]$$

$$\Downarrow$$

$$\frac{1}{\sqrt{2\pi }\left(\frac{\sqrt{1-a_t}\sqrt{1-{\overline{a}}_{t-1}}}{\sqrt{1-{\overline{a}}_t}}\right)}\exp [-\frac{1}{2}\frac{{(x_{t-1}-(\frac{\sqrt{a_t}(1-{\overline{a}}_{t-1})}{1-{\overline{a}}_t}{x}_t+\frac{\sqrt{{\overline{a}}_{t-1}}(1-a_t)}{1-{\overline{a}}_t}{x}_0))}^2}{{\left(\frac{\sqrt{1-a_t}\sqrt{1-{\overline{a}}_{t-1}}}{\sqrt{1-{\overline{a}}_t}}\right)}^2}]$$

$$\Downarrow$$

$$p(x_{t-1}|x_t)\sim N(\frac{\sqrt{a_t}(1-{\overline{a}}_{t-1})}{1-{\overline{a}}_t}{x}_t+\frac{\sqrt{{\overline{a}}_{t-1}}(1-a_t)}{1-{\overline{a}}_t}{x}_0,{\left(\frac{\sqrt{1-a_t}\sqrt{1-{\overline{a}}_{t-1}}}{\sqrt{1-{\overline{a}}_t}}\right)}^2)$$

Because $x_t=\sqrt{{\overline{a}}_t}\times x_0+\sqrt{1-{\overline{a}}_t}\times ϵ$, $x_0=\frac{x_t-\sqrt{1-{\overline{a}}_t}\times ϵ}{\sqrt{{\overline{a}}_t}}$. Substitute $x_0$ with this formula.

$$p(x_{t-1}|x_t)\sim N(\frac{\sqrt{a_t}(1-{\overline{a}}_{t-1})}{1-{\overline{a}}_t}{x}_t+\frac{\sqrt{{\overline{a}}_{t-1}}(1-a_t)}{1-{\overline{a}}_t}\times \frac{x_t-\sqrt{1-{\overline{a}}_t}\times ϵ}{\sqrt{{\overline{a}}_t}},\frac{{\beta }_t(1-{\overline{a}}_{t-1})}{1-{\overline{a}}_t})$$

$$p(x_{t-1}|x_t)\sim N(\frac{\sqrt{a_t}(1-{\overline{a}}_{t-1})}{1-{\overline{a}}_t}{x}_t+\frac{\sqrt{{\overline{a}}_{t-1}}(1-a_t)}{1-{\overline{a}}_t}\times \frac{x_t-\sqrt{1-{\overline{a}}_t}\times ϵ}{\sqrt{{\overline{a}}_t}},\frac{{\beta }_t(1-{\overline{a}}_{t-1})}{1-{\overline{a}}_t})$$

Note: This README.md is intended solely for previewing on the Github page. If you wish to view the rendered page locally, please consult README.raw.md.