[WebUI] Stable DiffusionããŒã¹ã¢ãã«ã®CLIPã®éã¿ãè¯ããã€ã«å€æŽãã
ç°¡åãªèšäºãªã®ã§å眮ãã¯çç¥ããŸãã
CLIPã«ã€ããŠ
Stable DiffusionããŒã¹ã®ã¢ãã«ã¯ãç»åçæã«éããŠããã¹ãã§æ¡ä»¶ã¥ããè¡ãïŒa.k.a. txt2imgïŒå ŽåãCLIPãšããåºç€ã¢ãã«ã§ããã¹ãããã¯ãã«ã«å€æããçæåŠçã«æž¡ããŠããŸãã
ã€ãŸããå ¥åããããã¹ããããã«ç確ã«çæåŠçã®ã¢ãã«ã«äŒããããã®ãã¯CLIPïŒã®ãããªããã¹ããšã³ã³ãŒããŒïŒã®æ§èœã«ããã£ãŠããããã§ãããå®ã¯Stable Diffusionã«çµã¿èŸŒãŸããŠããCLIPã¯ãCLIPã®äžã§ãæåŒ·æ§èœã®ãã®ã§ã¯ãããŸããã

é©åœã«ãç»åã®ã°ã©ãã«ç»å Žããç¹ã¯ããããCLIPã®ããªã¢ã³ãã§ãäžã«ããã¢ãã«ã»ã©æ§èœããããšèããŠãã ããã
ããã§ãStable Diffusion V1ç³»ã«çµã¿èŸŒãŸããŠããCLIPã¯L/14ã§ãããCLIPã®äžã§æãæ§èœã®è¯ãL/14@336pxã«äžæ©é
ãããšã£ãŠããããšãåãããŸãã
For the ViT-L/14 we also pre-train at a higher 336
pixel resolution for one additional epoch to boost performance similar to FixRes (Touvron et al., 2019). We denote
this model as ViT-L/14@336px. - CLIPå è«æãã
ã€ãŸããL/14@336pxã䜿ã£ãæ¹ãStable Diffusionã®æ§èœäžããããããšããããšã§ããã®èšäºã§ã¯Stable Diffusion WebUIäžã§CLIPã®éã¿ãäžçªãããã€ã«çœ®ãæããæ¹æ³ã玹ä»ããŸãã
[2023/01/20远èš]
Stable Diffusion WebUIã®æŽæ°ã«ããããã®èšäºã®ããæ¹ã¯out of dateã«ãªããŸããã
ããè¯ãæ¹æ³ã§ã¢ãã«ã®CLIPã眮ãæããã¹ã¯ãªãããæå¿ã®æ¹ãäœæããããããªã®ã§ããã¡ãã䜿çšããŠã¿ãŠãã ããã
以äžã®æç« ã¯å€ãããæ¹ã®ç޹ä»ãšããã®æ€èšŒçµæã§ãã
.
.
.
æ¬é¡
Stable Diffusion WebUIã®ãã©ã«ããéããmodules/sd_hijack.pyã®83-90è¡ç®ãæ¢ããŠãã ããã
ãããŠããã®ããã«å€æŽããŸãã
def hijack(self, m, use_improved_clip=True):
if type(m.cond_stage_model) == ldm.modules.encoders.modules.FrozenCLIPEmbedder:
if use_improved_clip:
from transformers import CLIPTextModel, CLIPTokenizer
device = "cuda" if torch.cuda.is_available() else "cpu"
m.cond_stage_model.transformer = CLIPTextModel.from_pretrained("openai/clip-vit-large-patch14-336").to(device)
m.cond_stage_model.tokenizer = CLIPTokenizer.from_pretrained("openai/clip-vit-large-patch14-336")
model_embeddings = m.cond_stage_model.transformer.text_model.embeddings
model_embeddings.token_embedding = EmbeddingsWithFixes(model_embeddings.token_embedding, self)
m.cond_stage_model = sd_hijack_clip.FrozenCLIPEmbedderWithCustomWords(m.cond_stage_model, self)
elif type(m.cond_stage_model) == ldm.modules.encoders.modules.FrozenOpenCLIPEmbedder:
m.cond_stage_model.model.token_embedding = EmbeddingsWithFixes(m.cond_stage_model.model.token_embedding, self)
m.cond_stage_model = sd_hijack_open_clip.FrozenOpenCLIPEmbedderWithCustomWords(m.cond_stage_model, self)
çµããã§ãã
ãã®èšäºãæžãçŽåãŸã§ã¯ã ãã¶æéããããŠé£ããããšããã£ãŠããŸãããããã£ããããã ãã§äžæãããããšã«æ°ã¥ããŸããã
ãã®å€æŽã«ãã£ãŠåæã«å
éšã§äœ¿çšããCLIPãL/14@336pxã«åãæ¿ãããŸããäœ¿çšæã¯å€ãããŸãããã[]ã()ã«ããããŒã¯ã³ã®åŒ·èª¿æ©èœãåé¡ãªã䜿ããŸãã
ããäœãäžå
·åãããå Žåããããã¯Dreamboothãè¡ã£ãã¢ãã«ã䜿çšããå Žåã¯ãdef hijack(...)ã®äžã®TrueãFalseã«æžãæããŠãã ãããæåãå
ã«æ»ããã¢ãã«ãckptå
ã«ããCLIPã䜿çšããããã«ãªããŸãã
æ¯èŒ
以äžã®ããã«ããªããé£ããããªããã³ããã§çæããŸãã
ã·ãŒãå€ã®ãããªãã®ã¯å
šãŠæããŠããŸãã
a digital painting of kawaii 1girl school student, focus_on_face, black hair, twintails, blue eyes, white shirt, yellow headband, evening, perfect symmetrical pretty face, bangs, extremely detailed, best quality, 8k uhd
L/14 (ããã©ã«ã)
æªããããŸããããçœã·ã£ããæå®ããã®ã«ã»ãŒã©ãŒæãçãŠããŸãã

L/14@336px (眮ãæãåŸ)
äžæããã£ãŠããŸãã

ãã®æ¯èŒã§ã¯ããŸãåšåãæããªããããããŸããããå人çã«æ§å³æå®ã®ããã³ããã远å ãããšãã®çæã«å¯Ÿããå¹ããç®ã«èŠããŠè¯ããªã£ããšæããŸããã
ãªããCLIPèªäœã®éçããããããè¯ããã€ã«å€æŽãããšããŠãäžæããããªãå Žåã¯ãŸããŸããããŸãã
ãããã«
å²ãšããªãŒã©ã³ãã§Stable DiffusionããŒã¹ã®ç»åçæã¢ãã«ãæ¹åã§ããã®ã§ããã²ã
.
.
.
[2022-12-06 远èš] BONUS: 2ã€ã®CLIPã¢ãã«ã®å¹³åããšã
- 泚æïŒè©Šã䟡å€ã¯ãããŸããããŸãã
ãã®æãDisco Diffusionãšãããä»ãšã¯å¥ã®ææ³ã§ããã¹ãã«ããæ¡ä»¶ä»ããè¡ããDiffusion Modelã§txt2imgãããPlaygroundããããŸããã
ããã§ã¯è€æ°ã®CLIPã¢ãã«ãã¢ã³ãµã³ãã«ããŠäœ¿ãããšãåœããåã«ãªã£ãŠããŠãäŸãã°CLIPã®Resnet-50ãšViT-L/14@336pxãçµã¿åãããŠäœ¿ããšãããããçæ¹1ã€ã ãã§äœ¿çšãããããçæçµæãè¯ããªãããšãç¥ãããŠããŸããã
Stable Diffusionã§ã®ããã¹ãæ¡ä»¶ä»ãã«ãããŠãåãããšãèšãããã¯åãããŸãããããšãããã詊ããŠã¿ãã®ã§ãã®æ¹æ³ã远èšããŸãã
ãŸããäžã®ã³ãŒãã倿ŽããŸãã
倿Žå
倿ŽåŸ
def hijack(self, m, use_improved_clip=True):
if type(m.cond_stage_model) == ldm.modules.encoders.modules.FrozenCLIPEmbedder:
if use_improved_clip:
from transformers import CLIPTextModel, CLIPTokenizer
device = "cuda" if torch.cuda.is_available() else "cpu"
m.cond_stage_model.tokenizer = CLIPTokenizer.from_pretrained("openai/clip-vit-large-patch14-336")
m.cond_stage_model.transformer = CLIPTextModel.from_pretrained("openai/clip-vit-large-patch14-336").to(device)
model_embeddings = m.cond_stage_model.transformer.text_model.embeddings
model_embeddings.token_embedding = EmbeddingsWithFixes(model_embeddings.token_embedding, self)
###################################################################################################################
# 倿Žç¹1
transformer_2 = CLIPTextModel.from_pretrained("openai/clip-vit-large-patch14").to(device)
model_embeddings_2 = transformer_2.text_model.embeddings
model_embeddings_2.token_embedding = EmbeddingsWithFixes(model_embeddings_2.token_embedding, self)
m.cond_stage_model = sd_hijack_clip.FrozenCLIPEmbedderWithCustomWords_2(m.cond_stage_model, self, transformer_2)
###################################################################################################################
else:
model_embeddings = m.cond_stage_model.transformer.text_model.embeddings
model_embeddings.token_embedding = EmbeddingsWithFixes(model_embeddings.token_embedding, self)
m.cond_stage_model = sd_hijack_clip.FrozenCLIPEmbedderWithCustomWords(m.cond_stage_model, self)
elif type(m.cond_stage_model) == ldm.modules.encoders.modules.FrozenOpenCLIPEmbedder:
m.cond_stage_model.model.token_embedding = EmbeddingsWithFixes(m.cond_stage_model.model.token_embedding, self)
m.cond_stage_model = sd_hijack_open_clip.FrozenOpenCLIPEmbedderWithCustomWords(m.cond_stage_model, self)
ãããŠæ¬¡ã«ãmodules/sd_hijack_clip.pyãéãã以äžã®ã³ãŒãã远å ããŠãã ããã
倿ŽåŸ
class FrozenCLIPEmbedderWithCustomWords_2(FrozenCLIPEmbedderWithCustomWordsBase):
def __init__(self, wrapped, hijack, additional_transformer):
super().__init__(wrapped, hijack)
self.tokenizer = wrapped.tokenizer
self.comma_token = [v for k, v in self.tokenizer.get_vocab().items() if k == ',</w>'][0]
self.token_mults = {}
tokens_with_parens = [(k, v) for k, v in self.tokenizer.get_vocab().items() if '(' in k or ')' in k or '[' in k or ']' in k]
for text, ident in tokens_with_parens:
mult = 1.0
for c in text:
if c == '[':
mult /= 1.1
if c == ']':
mult *= 1.1
if c == '(':
mult *= 1.1
if c == ')':
mult /= 1.1
if mult != 1.0:
self.token_mults[ident] = mult
self.id_start = self.wrapped.tokenizer.bos_token_id
self.id_end = self.wrapped.tokenizer.eos_token_id
self.id_pad = self.id_end
self.additional_transformer = additional_transformer
def tokenize(self, texts):
tokenized = self.wrapped.tokenizer(texts, truncation=False, add_special_tokens=False)["input_ids"]
return tokenized
def encode_with_transformers(self, tokens):
outputs = self.wrapped.transformer(input_ids=tokens, output_hidden_states=-opts.CLIP_stop_at_last_layers)
outputs_2 = self.additional_transformer(input_ids=tokens, output_hidden_states=-opts.CLIP_stop_at_last_layers)
if opts.CLIP_stop_at_last_layers > 1:
z = outputs.hidden_states[-opts.CLIP_stop_at_last_layers]
z = self.wrapped.transformer.text_model.final_layer_norm(z)
z_2 = outputs_2.hidden_states[-opts.CLIP_stop_at_last_layers]
z_2 = self.additional_transformer.text_model.final_layer_norm(z_2)
z = (z + z_2) / 2
else:
z = outputs.last_hidden_state
return z
def encode_embedding_init_text(self, init_text, nvpt):
embedding_layer = self.wrapped.transformer.text_model.embeddings
ids = self.wrapped.tokenizer(init_text, max_length=nvpt, return_tensors="pt", add_special_tokens=False)["input_ids"]
embedded = embedding_layer.token_embedding.wrapped(ids.to(devices.device)).squeeze(0)
return embedded
ããã§ã2ã€ã®CLIP(ViT-L/14 & ViT-L/14@336px)ããåºãŠããåã蟌ã¿è¡šçŸã®å¹³åå€ã«ãã£ãŠStable Diffusionã®æ¡ä»¶ä»ããè¡ãããããã«ãªããŸãã
ãããŠãå®éšçµæã¯ä»¥äžã

远å å®éšã»åäžããã³ããã§ã·ãŒããæããŠ6æãã€

ããŒãããŸããŸããšãããããªããéè¡ããŠãªãâŠïŒ
æ¹åãããŠããã©ããããåãããªãäžã远å ã§VRAMãå æãããã®ã§ãçŸæç¹ã§ãããŸããªã¹ã¹ã¡ã¯ã§ããŸããã
ViT-L/14ããViT-L/14@336pxãžã®å€æŽã«ããæ¹åã®ã»ããæç¶éç«ã€å®éšçµæã«ãªã£ãŠããŸããâŠã
Discussion
æçšãªèšäºæé£ãæèŠãããŠããã£ãŠããŸãã
çŽ äººè³ªåã§æçž®ãªã®ã§ããCLIPãäžè𿹿³ã§å€æŽããå ŽåãDreamBoothãªã©ã§ã®è¿œå åŠç¿ã«è¯ã圱é¿ããã£ããããã®ã§ããããïŒ
ãªã§ãŠãã€ãããããã«ã¡ã¯ã
èšäºãã芧ããã ãããããšãããããŸãïŒ
Dreamboothãªã©ã®è¿œå åŠç¿ã«ãããCLIP倿Žã®åœ±é¿ã«ã€ããŠãç³ãèš³ãããŸãããèªåã®ç¥èã§ã¯æèšã§ããŸããâŠã
æšæž¬ã®ç¯å²ã§è¿°ã¹ãããŠããã ããŸãã
ãµãã£ãšããåçã«ãªã£ãŠããŸã£ãŠå€§å€æçž®ã§ããããã1ããªã§ãåèã«ãªãã°å¹žãã§ãã
詳ãããçãé ãããããšãããããŸãïŒ