twodgirl / stable-diffusion-3-medium-gguf

Stable Diffusion 3 Medium

Distribute SD3 in Q8 file format, but run the model in regular fp16.

Gerganov's gguf scripts have been updated, this is just the result.

Download

Download the t5xxl_fp8_e4m3fn.safetensors from camenduru .

You should have:

sd3_medium.Q8.gguf
t5xxl_fp8_e4m3fn.safetensors

Setup

pip install diffusers numpy safetensors sentencepiece transformers
pip install "gguf @ git+https://github.com/ggerganov/llama.cpp.git@master#subdirectory=gguf-py"

Inference

from diffusers import StableDiffusion3Pipeline, SD3Transformer2DModel
import gguf
from gguf.quants import dequantize
from safetensors.torch import load_file
from transformers import T5Config, T5EncoderModel
import torch


def load_transformer_from_gguf(filepath):
    sd = {}
    reader = gguf.GGUFReader(filepath)
    for item in reader.tensors:
        xs = dequantize(item.data, item.tensor_type)
        tensor = torch.tensor(xs)
        sd[item.name] = tensor.to(dtype=torch.float16)
    config = SD3Transformer2DModel.load_config('./transformer')
    transformer = SD3Transformer2DModel.from_config(config).to(dtype=torch.float16)
    transformer.load_state_dict(sd)
    transformer.eval()

    return transformer

transformer = load_transformer_from_gguf('sd3_medium.Q8.gguf')
# Load T5 encoder in F8_E4M3 format.
# t5_config = T5Config.from_pretrained('text_encoder_3')
# t5 = T5EncoderModel(t5_config).to(dtype=torch.float16)
# t5_sd = load_file('t5xxl_fp8_e4m3fn.safetensors')
# t5.load_state_dict(t5_sd)
# t5.eval()
pipe = StableDiffusion3Pipeline.from_pretrained('stabilityai/stable-diffusion-3-medium-diffusers',
                                                transformer=transformer,
                                                # Either disable the T5 encoder.
                                                text_encoder_3=None,
                                                tokenizer_3=None,
                                                # Or enable it.
                                                # text_encoder_3=t5,
                                                torch_dtype=torch.float16)
pipe.enable_model_cpu_offload()
image = pipe('cat playing piano', num_inference_steps=20).images[0]
image.save('cat.png')

Disclaimer

Use of this code and the copy of documentation requires citation and attribution to the author via a link to their Hugging Face profile in all resulting work.