tags: 第 16 屆 iThome 鐵人賽 (2023)

{%hackmd BJrTq20hE %}

HED Edge模型是使用3M個邊緣圖像和標題對進行訓練的，並使用Nvidia A100 80G進行了600 GPU小時的訓練，基於Stable Diffusion 1.5作為基礎模型。

模型下載

https://github.com/patrickvonplaten/controlnet_aux

pip install controlnet_aux

下載對應的transformers

pip install diffusers transformers accelerate

from PIL import Image
from diffusers import StableDiffusionControlNetPipeline, ControlNetModel, UniPCMultistepScheduler
import torch
from controlnet_aux import HEDdetector
from diffusers.utils import load_image

hed = HEDdetector.from_pretrained('lllyasviel/ControlNet')

image = load_image("https://huggingface.co/lllyasviel/sd-controlnet-hed/resolve/main/images/man.png")


image = hed(image)

controlnet = ControlNetModel.from_pretrained(
    "lllyasviel/sd-controlnet-hed", torch_dtype=torch.float16
)

pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5", controlnet=controlnet, safety_checker=None, torch_dtype=torch.float16
)

pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)

# Remove if you do not have xformers installed
# see https://huggingface.co/docs/diffusers/v0.13.0/en/optimization/xformers#installing-xformers
# for installation instructions
pipe.enable_xformers_memory_efficient_attention()

pipe.enable_model_cpu_offload()

image = pipe("oil painting of handsome old man, masterpiece", image, num_inference_steps=20).images[0]

image.save('images/man_hed_out.png')

運行效果

再生成後的結果

訓練結論

HED模型擁有最好的圖片還原程度，可以將同一張圖片的特徵擷取並再生成，算是想要重鑄圖片的好模型

參考網址

https://huggingface.co/lllyasviel/sd-controlnet-hed