[Stable Diffusion] Depth-to-Imageã¢ãã«ãåŠç¿ãªãã§ç¹å®ã®ãã¡ã€ã³ã«é©å¿ããã
ã¯ããã«
ããã«ã¡ã¯ã
æ©éæ¬é¡ã«å ¥ããŸãã
[2023/01/20远èš]
ãã®èšäºã§èª¬æããŠããDepth-to-Imageã¢ãã«ã®ãã¡ã€ã³é©å¿ãšãé©å¿ããã¢ãã«ã®AUTOMATIC1111ããã®WebUIã§ã®äœ¿çšãè¡ããColab Notebookãå
¬éããŸããã

ãããªæãã§äœ¿ããŸã
Depth-to-Imageã¢ãã«ã®èª¬æ
StabilityAIãå ¬éããStable Diffusion V2ç³»ã®ã¢ãã«ã®äžã«ãdepthã¢ãã«ãšãããã®ããããŸãã
ãã®ã¢ãã«ãã§ããããšã¯ãäžè¬çãªStable Diffusionã®img2imgãã§ããããšãšåºæ¬çã«ã¯åãã§ãç»åãšããã¹ããå ¥åãšããå ¥åãããç»åã¹ã¿ã€ã«ãããã¹ãã®æç€ºã«åŸã£ãŠå€æããããšãããã®ã§ãã
ãŸãã¯ãäžè¬çãªã¢ãã«ïŒããã§ã¯Waifu Diffusion V1.4 Epoch1ã䜿çšããŠããŸãïŒã®img2imgçµæãèŠãŠã¿ãŸãããã
倿å
-
å ¥åç»å

-
å ¥åããã¹ã: masterpiece, 1boy, anime
倿åŸ

.
.
.
äžã®äŸãèŠãŠããããšãããããšæããŸãããåŸæ¥çãªimg2imgã¯èª²é¡ãæ±ããŠããŸãã
ããã¯ãå
¥åç»åã®ã¹ã¿ã€ã«å€æïŒäžã®äŸã ãšå®åâã¢ãã¡èª¿ãžã®å€æïŒããã£ãããšè¡ãããã«ã¯ãå
¥åç»åãããçšåºŠç Žå£çã«å€æŽããŠå€æããŠãããå¿
èŠãããããã®å ŽåãäŸã®ããã«åœ±ããšãã€ãäœãã«å€åãããããªãå
¥åç»åãã諞ã
倧ããããé¢ããåºåããåŸãããªãããšããããŸãã
ã ãããšãã£ãŠå ¥åã«å¿ å®ãªå€æãããããšãããšãããŸãã¹ã¿ã€ã«å€æãã§ããŸããã
å
¥åã«å¿ å®ãªå€æãããããšããå Žå(Denoising Strength=0.35)

ããããæãã§ã¯ãªã
ã¢ãã¡èª¿ã«å€æãããã£ãã¯ããããªããšããããªã¢ã«ç³»ã®ã€ã©ã¹ãã«å€æãããŠããŸããŸãããå ¥åç»åã«ãããªããžã§ã¯ãã®äœçœ®é¢ä¿ã圢ç¶ã«ã€ããŠç Žå£ç倿Žãããªãããç»åã倿ããå ŽåãçŸå®ã®åçãã倧ããé¢ããã¹ã¿ã€ã«ã§ããã¢ãã¡èª¿ã«ã¯èŸ¿ãçããªããšããããšã§ãã
以äžã«æžããããã«ãçŸç¶ã®img2imgãæ±ããåãåãã®æªãã«å¯ŸåŠãã¹ãStabilityAIãå ¬éããã®ããç»åã®æ·±åºŠæ å ±ãä¿ã¡ã€ã€ã¹ã¿ã€ã«å€æ(depth-to-image, depth2img)ãè¡ãããšãã§ãããdepthã¢ãã«ã§ãã
ãã®ã¢ãã«ã«ããdepth2imgã¯ãäžè¬çãªStable Diffusionãšç°ãªããUNetã«ãã€ãºãèŒããç»åïŒã®ç¹åŸŽéïŒã ããå ¥åããã®ã§ã¯ãªããå¥ã®æ·±å±€åŠç¿ã¢ãã«ã«ãã£ãŠæšæž¬ãããç»åã®æ·±åºŠæ å ±ãå ¥åããUNetã«å¯ŸããŠçŽã«ç»åå ã®ãªããžã§ã¯ãã®å Žæãé è¿ãæã蟌ã¿ãŸããUNetã¯æ·±åºŠæ å ±ãšäžè²«æ§ãä¿ã£ãåºåãäœæããããã«åŠç¿ãããŠãããããç»åã«å¯ŸããŠåŒ·ãã®ã¹ã¿ã€ã«å€æããããå Žåã§ããç»åå ã®ãªããžã§ã¯ããã©ã£ãã«è¡ã£ããå€ãªãã®ã远å ããããããªãããã«ããããšãå¯èœã«ãªã£ãŠããŸãã
å
¬åŒãªããžããªã®depth2imgãã¬ãã¥ãŒ

1æç®ãå
¥åç»åã§ã2æç®ã®çœããã€ã深床æ
å ±ã3æç®ä»¥éãçæ
äžã®ç»åã¯å ¬åŒãªããžããªã«ããdepth2imgã®äŸã§ãã1æç®ã¯å ¥åç»åã§ãMiDaSãšããæ·±åºŠæšå®ã¢ãã«ãå ¥åç»åã®depthãæšæž¬ããçµæã2æç®ãšãªããŸãããã®2æã®ç»åãšããã¹ããUNetã«å ¥åããããšã§ãå ¥åç»åå ã«ãããªããžã§ã¯ãïŒããã§ã¯ãããããã§ããïŒã®äœçœ®é¢ä¿ãå€ããããšããèæ¯ã®çœãç ãäœè ãã«å€åããããšããªãããããç»åå ã«ãããªããžã§ã¯ãã«å¯ŸããŠå€§ããªå€æŽãå ããããšãã§ããŸãã
.
.
.
ããããããã§åé¡ããããŸãã
ããã¯ããããªã«åªããã¢ãã«ãªã®ã«ãé¢ãããã誰ããã®ã¢ãã«ããã¡ã€ã³ãã¥ãŒãã³ã°ãããã®ãå
¬éããŠããªãïŒ2023幎1æ13æ¥çŸåšïŒãããèšç®è³æºãåŠç¿çšããŒã¿ã«æµãŸããªãæã
ã¯ãStable Diffusionå
¬åŒã®depthã¢ãã«ãæã€ããã³ã·ã£ã«ã®ç¯å²å
ã§ãããæ·±åºŠæ
å ±ãæŽ»çšããã¹ã¿ã€ã«å€æã¢ãã«ã®å®åã享åã§ããªããšããããšã§ãã
äŸãã°äžã§è¡ã£ããããªã¢ãã¡èª¿ãžã®å€æã§ããããã®depthã¢ãã«ã¯ã»ãšãã©ã¢ãã¡èª¿ã®ç»åãåŠç¿ããŠããªãã®ããããŸããããããªãç»åããçæã§ããŸããã

ãããŠãååãªã¯ãªãªãã£ãŒã§ã®ã¢ãã¡èª¿ãžã®å€æãå¯èœã«ããã«ã¯ã倧éã®åŠç¿çšç»åãšæœ€æ²¢ãªèšç®è³æºãéããé«ã黿°ä»£ãæã£ãŠãã®ã¢ãã«ã«æ°ããªãã¡ã€ã³ãåŠç¿ãããªããã°ãªããŸããã
å æããã®ã¢ãã«ã®Dreamboothã¹ã¯ãªããã芪åãªèª°ãã«ãã£ãŠå ¬éãããã®ã§ãããã䜿ãã°äžå¿åŠç¿ã¯é²ããããŸãããdepth2imgãçšããŠããããããšãå ¬åŒã®depthã¢ãã«ã®ç¥ã£ãŠãããã¡ã€ã³ãšããé¢ããŠããå ŽåãDreamboothã®ãããªå°èŠæš¡ã®åŠç¿ã§å ¬åŒã¢ãã«ãç®çã®ãã¡ã€ã³ã«é©å¿ãããã®ã¯ããªãé£ãããçŸç¶ã¯ãã®ã¢ãã«ãæã¿ã®ãã¡ã€ã³ã«å¯ŸããŠå€§èŠæš¡ã«è¿œå åŠç¿ããŠãããŠããªããã€ç¡æã§å ¬éããŠãããç¥åŸ ã¡ã®ç¶æ ãšèšã£ãŠãéèšã§ã¯ãããŸããã
ãšããæãã§ãäŸã«æŒããèªåãå€§èŠæš¡ãªåŠç¿ãè¡ãããããªèšç®è³æºãæã¡åãããŠããªãããããªããšãããŠå°æå ã ãã§äžæããããªãããšèããŸããã
Task Arithmetic ã«ã€ããŠ
ããã§ããã®è«æãé ãããããŸãã
ã¢ãã¹ãã©ã¯ããåŒçšããŸãã
Changing how pre-trained models behaveâe.g., improving their performance on a downstream task or mitigating biases learned during pre-trainingâis a common practice when developing machine learning systems. In this work, we propose a new paradigm for steering the behavior of neural networks, centered around task vectors.
A task vector specifies a direction in the weight space of a pre-trained model, such that movement in that direction improves performance on the task.
We build task vectors by subtracting the weights of a pre-trained model from the weights of the same model after fine-tuning on a task. We show that these task vectors can be modified and combined together through arithmetic operations such as negation and addition, and the behavior of the resulting model is steered accordingly. Negating a task vector decreases performance on the target task, with little change in model behavior on control tasks.
Moreover, adding task vectors together can improve performance on multiple tasks at once. Finally, when tasks are linked by an analogy relationship of the form âA is to B as C is to Dâ, combining task vectors from three of the tasks can improve performance on the fourth, even when no data from the fourth task is used for training. Overall, our experiments with several models, modalities and tasks show that task arithmetic is a simple, efficient and effective way of editing models.
ãã£ããèšããšã
- äºååŠç¿æžã¿ã¢ãã«ã®éã¿ãšãã¡ã€ã³ãã¥ãŒãã³ã°ãããã¢ãã«ã®éã¿ã®å·®ãã¿ã¹ã¯ãã¯ãã«ãšããã
- ã¿ã¹ã¯ãã¯ãã«ã¯ãäºååŠç¿æžã¿ã¢ãã«ã®éã¿ç©ºéã«ãããŠããã®ã¢ãã«ãããã¿ã¹ã¯ã«å¯Ÿããæ§èœãåäžãããæ¹åãå®ãããã®ã
- åäžã®äºååŠç¿æžã¿ã¢ãã«ãã远å åŠç¿ãããã¢ãã«å士ã§ããã°ãã¿ã¹ã¯Aã«ã€ããŠè¿œå åŠç¿ãããã¢ãã«ããæœåºããã¿ã¹ã¯ãã¯ãã«ãå¥ã¢ãã«ã«å ç®ããããšã«ãã£ãŠãå¥ã¢ãã«ã®ã¿ã¹ã¯Aã®æ§èœãåäžãããïŒéãç¶ãïŒ
- ã¿ã¹ã¯ãã¯ãã«ãè¶³ãåãããããšã§ãè€æ°ã®ã¿ã¹ã¯ã«å¯Ÿããæ§èœãåäžãããããšãã§ããã
ãšããå 容ã§ããã¿ã¹ã¯æžç®ã»é¡æšã®è©±ã¯ããªã瀺åã«å¯ããã®ããããå 容ãçµæ§è»œãã®ã§ããªãããããã®è«æã§ãã

è©±ãæ»ããŸãã
ã€ãŸãããã®è«æããä»åã®è©±ã«ç¹ããããšãšããŠã
åäžã®äºååŠç¿æžã¿ã¢ãã«ãããã¡ã€ã³ãã¥ãŒãã³ã°ããããããããå¥ã ã®ã¢ãã«ããæœåºããã¿ã¹ã¯ãã¯ãã«ãè¶³ãåãããã¢ãã«ã¯ããã®å¥ã ã®ã¿ã¹ã¯ã«å¯Ÿããæ§èœãåäžãã
ãšããããšã«ãªããŸãã
ãããŸã§ã®è©±ãèããŠããŠãStable Diffusionãã远å åŠç¿ãããã€ã©ã¹ãçæã¢ãã«ãããè§Šã£ãŠããæ¹ã ã¯ãªããšãªãå¿åœããããããããããŸããããããã¯ã¢ãã«ã®ããŒãžãšãã¡ããã¡ã䌌ãŠãã話ã§ãã
ãããäºç¹éã£ãŠããŠãäžã€ã¯Task Arithmeticãã¢ãã«ãšã¢ãã«ã®å éå¹³åã§ã¯ãªãããšã«ãããŸãã
ãã®ããã¿ã¹ã¯ãã¯ãã«ã¯ããã®äœæã®ããŒã¹ãšãªã£ãã¢ãã«ãåäžã§ããã°ãæ¯çã®åèšã1ã«ãªããªããããªéã¿ã¥ãã§å ç®ã»æžç®ã§ããŸãã
ãŸãããäžç¹ã«ã€ããŠãããã¿ã¹ã¯ã«ç¹åããã¢ãã«ã®æ§é ãã远å åŠç¿å
ã®ããŒã¹ã¢ãã«ã®æ§é ãšæ¯èŒããŠå°ãå€ãã£ããããŠããŠããåæ¹ã§å
±æããŠããã¬ã€ã€ãŒã®éã¿ç©ºéã®åœ¢ç¶ãæ ¹æ¬çã«ç°ãªãããã§ã¯ãªããã°ãã¿ã¹ã¯ãã¯ãã«ã¯ïŒããããïŒäž¡ã¢ãã«ã®å
±ééšåã«ãããŠè¶³ãåãããããšãã§ããã¯ãã§ãã
äžæ¹ãã¢ãã«ããŒãžã¯äž¡è
åãã¬ã€ã€ãŒãæã£ãŠããªããšè¡ããŸããã
ããã§ã¯ãdepthã¢ãã«ã®ããŒã¹ã«ãªã£ãã¢ãã«ãèŠè¿ããŠã¿ãŸãããã
This stable-diffusion-2-depth model is resumed from stable-diffusion-2-base (512-base-ema.ckpt) and finetuned for 200k steps. Added an extra input channel to process the (relative) depth prediction produced by MiDaS (dpt_hybrid) which is used as an additional conditioning.
https://huggingface.co/stabilityai/stable-diffusion-2-depth
æ°ããå
¥åãã£ã³ãã«ã远å ãããŠããŠãéã¿ç©ºéã®åœ¢ç¶ãããªãå€åããŠããå¯èœæ§ããããã®ã®ãããŒã¹ã¢ãã«ã¯Stable Diffusion V2.0ãããã§ãã
ãšããããšã§ããããŸã§ã®èããèžãŸãããšãdepthã¢ãã«ã¯Stable Diffusion V2.0ãã远å åŠç¿ãããã¢ãã«ã®ã¿ã¹ã¯ãã¯ãã«ãšå ç®ããããšãã§ããã®ã§ã¯ãªããïŒ ãšããã®ãä»åã®ä»®èª¬ã§ã詊ããããšã§ãã
ãããŸã§ã§ãã倧äœåãã£ããšããæ¹ã¯ããã²æå ã§è©ŠããŠã¿ãŠãã ããã
å¥ã¢ãã«ã®ã¿ã¹ã¯ãã¯ãã«ãdepthã¢ãã«ã«å ç®ãã
ãŸããçšæããå¥ã¢ãã«ã§ãããæäœéStable Diffusion V2ç³»ãã远å åŠç¿ãããã¢ãã«ã§ããå¿ èŠãããïŒSD-V1ç³»ã§ã¯å ç®ã§ããªãããšãç¢ºèªæžã¿ïŒãããWaifu Diffusion V1.4ã«ããŸãã
ãªããWaifu Diffusion V1.4ã®ã¿ã¹ã¯ãã¯ãã«ãdepthã¢ãã«ã«å ç®ããããšã¯ãStable Diffusion V2.0ãããŒã¹ã¢ãã«ãšããŠãdepthã¢ãã«ã®ã¿ã¹ã¯ãã¯ãã«ãšWaifu Diffusion V1.4ã®ã¿ã¹ã¯ãã¯ãã«ãå ç®ããããšãšå矩ã§ãã
ã¿ã¹ã¯ã®å ç®ãè¡ãã«ããã£ãŠãŸããæ¬åœã«å ç®ã§ããã®ãïŒ ã確èªããŸããåæ¹ã®éã¿ç©ºéã®åœ¢ç¶ãããŸãã«ã䌌ãŠããªãã£ããããããå ç®ã¯ã§ããªããšããããšãåããã®ã§ãåæ¹ã®UNetã§åœ¢ç¶ãåãåã¬ã€ã€ãŒã®ã³ãµã€ã³é¡äŒŒåºŠãèŠãŠã¿ãŸãããã
import torch
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
depth = torch.load("./512-depth-ema.ckpt")
waifu1_4 = torch.load("./wd-1-4-anime_e2.ckpt")
sim_dict = {}
for key in set(depth["state_dict"].keys()) & set(waifu1_4["state_dict"].keys()):
# skip weight of an input block whose dimension is different from each other
if "model.diffusion_model" in key and "model.diffusion_model.input_blocks.0.0" not in key:
depth_weight = depth["state_dict"][key].numpy().flatten()
waifu1_4_weight = waifu1_4["state_dict"][key].numpy().flatten()
# calc cosine similarity
cos_sim = (
np.dot(depth_weight, waifu1_4_weight)
/ (np.linalg.norm(depth_weight) * np.linalg.norm(waifu1_4_weight))
)
sim_dict[key] = cos_sim
df = pd.DataFrame.from_dict(
sim_dict,
orient="index",
columns=["cosine_similarity"],
)
df.plot(
kind="hist",
figsize=(10, 10),
ylabel="count",
xlabel="cosine similarity",
title="Cosine similarity between depth and waifu1-4",
)
plt.show()

ããªã䌌ãŠããŸããããããã¡ãããïŒ ãšããæããªã®ã§ãå ç®ããŠã¿ãŸãããã
ãªããWaifu Diffusion V1.4ã®VAEã¯Stable Diffusionã®ããããã¢ãã¡èª¿ã«é©ããŠãããããããã ãã¯å ç®ã§ã¯ãªãã眮ãæããŸãã
import copy
import torch
base = torch.load("./512-base-ema.ckpt")
depth = torch.load("./512-depth-ema.ckpt")
waifu1_4 = torch.load("./wd-1-4-anime_e2.ckpt")
# make task vector
task_waifu = copy.deepcopy(waifu1_4)
for key in set(waifu1_4["state_dict"].keys()):
if "model.diffusion_model" in key:
task_waifu["state_dict"][key] = waifu1_4["state_dict"][key] - base["state_dict"][key]
for key in set(depth["state_dict"].keys()) & set(task_waifu["state_dict"].keys()):
# replace weight of VAE with waifu's weight
if "first_stage_model" in key:
depth["state_dict"][key] = task_waifu["state_dict"][key]
# don't replace weight of an input block whose dimension is different from each other
elif "model.diffusion_model.input_blocks.0.0" in key:
pass
# add task weight to unet layers
elif "model.diffusion_model" in key:
task_depth = depth["state_dict"][key] - base["state_dict"][key]
depth["state_dict"][key] = base["state_dict"][key] + (task_depth * 0.5 + task_waifu["state_dict"][key] * 0.5)
torch.save(depth, "./depth_waifu_1_4.ckpt")
ããã§ã¯ãæ©éãã®ã¢ãã«ã§depth2imgããã£ãŠã¿ãŸãã
å
¥åã¯äžã®ãã®ãšåãç»åã»ããã¹ãã䜿ã£ãŠã¿ãŸãããã
(åæ²)å
¥åç»å

以äžãdepth2imgã®çµæã§ãã
倿åŸ

ã©ãã§ããããïŒ å®åâã¢ãã¡èª¿ãžã®å€æã¯ããããªã§ãããç»åå ã®ãªããžã§ã¯ããäœãå šãå¥ã®ã¢ããšè§£éãããŠçœ®ãæãã£ãŠãããšããããšããããŸããïŒïŒäººç©ã䌌ãŠãªãã®ã¯å ¥åããã¹ãã®ã»ãã§äŒŒããåªåãããŠããªãããïŒ
ç¹ã«å€§æãããªããšãããã§ããªããdepthã¢ãã«ãšWaifu Diffusion V1.4ããããããåŸæãªã¿ã¹ã¯ãäž¡æ¹ãšãããªããã¢ãã«ãçã¿åºãããšãã§ããŸããã
ã¡ãªã¿ã«ããã®ãã¡ã©ã¢ãã«ãå¿çšããŠãããªããšãã§ããŸãïŒ1æç®ãå ¥åã2æç®ãåºåïŒã
ãªããå ç®ã®ãšãã®ä¿æ°ã§ãã0.5, 0.5ã«ã¯äœã®æ ¹æ ããªããããæ°ã«ãªãæ¹ããããè²ã
æ°å€ãå€ããŠè©ŠããŠã¿ãŠãã ããã
.
.
.
ãããã«
depthã¢ãã«ã¯Stable Diffusion V2.0ã®ãã¡ã€ã³ãã¥ãŒãã³ã°ã¢ãã«ã§ããããªãªãžãã«ããæ§é ãå°ãå€ãã£ãŠããã®ã§ããŸããTask Arithmeticã䜿ãããšã¯æ£çŽæã£ãŠããªããŠããã¡å ã§é©åœã«ãã£ããäžæããã£ãŠãããæ£ããããŸããã
åœç¶ã§ãããWaifu Diffusion V1.4ã«éããStable Diffusion V2.0ãã远å ã§åŠç¿ãããã¢ãã«ã§ããã°ãåæ§ã®ææ³ã§çµã¿åãããããšãåºæ¥ããšæããŸããæ¥œããã§ããã
ãã²è©ŠããŠã¿ãŠãã ãããæªçšã¯ð
Discussion