【Video2Video】Diffusers で AnimateDiffVideoToVideoPipeline というのが公開されていたので AnimateDiff v2 と v3 で実行してみました。

PC環境

Windows 11
CUDA 11.8
Python 3.11

Python環境構築

pip install torch==2.1.2+cu118 --index-url https://download.pytorch.org/whl/cu118
pip install git+https://github.com/huggingface/diffusers
pip install accelerate transformers

準備

こちらから「animatediff-vid2vid-input-1.gif」をダウンロードしました。

アライグマがギターを弾くGIF動画です。

結果

Video2Videoでアライグマをパンダに変えています。

結果はGoogle Bloggerに載せておきます。
support-touchsp.blogspot.com

Pythonスクリプト

import torch
from diffusers import AnimateDiffVideoToVideoPipeline, DDIMScheduler, MotionAdapter
from diffusers.utils import export_to_gif
from PIL import Image

adapter = MotionAdapter.from_pretrained(
    "animatediff-motion-adapter-v1-5-2",
    #"animatediff-motion-module-v1-5-3",
    torch_dtype=torch.float16
)
model_id = "model/Realistic_Vision_V5.1_noVAE"
pipe = AnimateDiffVideoToVideoPipeline.from_pretrained(
    model_id,
    motion_adapter=adapter,
    torch_dtype=torch.float16
)
pipe.scheduler = DDIMScheduler.from_config(
    pipe.scheduler.config,
    clip_sample=False,
    timestep_spacing="linspace",
    beta_schedule="linear",
    steps_offset=1
)
pipe.to("cuda")

'''
pipe.load_lora_weights("lora/v3_sd15_adapter.ckpt", adapter_name="domain")
pipe.set_adapters(["domain"], adapter_weights=[1.0])
'''

images_from_gif = []
gif_images = Image.open("animatediff-vid2vid-input-1.gif")
for i in range(gif_images.n_frames):
    gif_images.seek(i)
    image = gif_images.copy()
    images_from_gif.append(image.convert("RGB"))

output = pipe(
    video = images_from_gif,
    prompt="panda playing a guitar, on a boat, in the ocean, high quality",
    negative_prompt="bad quality, worse quality",
    guidance_scale=7.5,
    num_inference_steps=35,
    strength=0.5,
    generator=torch.manual_seed(20240126)
)
frames = output.frames[0]
export_to_gif(frames, "animation.gif")