briaai / BRIA-2.3-ControlNet-BG-Gen

BRIA-2.3-ControlNet-Background-Generation, Model Card

BRIA 2.3 ControlNet-Background Generation, trained on the foundation of BRIA 2.3 Text-to-Image , enables the generation of high-quality images guided by a textual prompt and the extracted background mask estimation from an input image. This allows for the creation of different background variations of an image, all sharing the same foreground.

BRIA 2.3 was trained from scratch exclusively on licensed data from our esteemed data partners. Therefore, they are safe for commercial use and provide full legal liability coverage for copyright and privacy infringement, as well as harmful content mitigation. That is, our dataset does not contain copyrighted materials, such as fictional characters, logos, trademarks, public figures, harmful content, or privacy-infringing content.

Model Description

• Developed by: BRIA AI

• Model type: ControlNet for Latent diffusion

• License: bria-2.3

• Model Description: ControlNet Background-Generation for BRIA 2.3 Text-to-Image model. The model generates images guided by text and the background mask.

• Resources for more information: BRIA AI

Get Access

BRIA 2.3 ControlNet-Background Generation requires access to BRIA 2.3 Text-to-Image. For more information, click here .

Usage

Installation

pip install -qr https://huggingface.co/briaai/BRIA-2.3-ControlNet-BG-Gen/resolve/main/requirements.txt
huggingface-cli download briaai/BRIA-2.3-ControlNet-BG-Gen --include replace_bg/* --local-dir . --quiet

import torch
from diffusers import (
    AutoencoderKL,
    EulerAncestralDiscreteScheduler,
)
from diffusers.utils import load_image
from replace_bg.model.pipeline_controlnet_sd_xl import StableDiffusionXLControlNetPipeline
from replace_bg.model.controlnet import ControlNetModel
from replace_bg.utilities import resize_image, remove_bg_from_image, paste_fg_over_image, get_control_image_tensor
controlnet = ControlNetModel.from_pretrained("briaai/BRIA-2.3-ControlNet-BG-Gen", torch_dtype=torch.float16) 
vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
pipe = StableDiffusionXLControlNetPipeline.from_pretrained("briaai/BRIA-2.3", controlnet=controlnet, torch_dtype=torch.float16, vae=vae).to('cuda:0')
pipe.scheduler = EulerAncestralDiscreteScheduler(
    beta_start=0.00085,
    beta_end=0.012,
    beta_schedule="scaled_linear",
    num_train_timesteps=1000,
    steps_offset=1
)

image_path = "https://farm5.staticflickr.com/4007/4322154488_997e69e4cf_z.jpg"
image = load_image(image_path)
image = resize_image(image)
mask = remove_bg_from_image(image_path)
control_tensor = get_control_image_tensor(pipe.vae, image, mask)

prompt = "in a zoo"
negative_prompt = "Logo,Watermark,Text,Ugly,Bad proportions,Bad quality,Out of frame,Mutation"
generator = torch.Generator(device="cuda:0").manual_seed(0)

gen_img = pipe(
    negative_prompt=negative_prompt, 
    prompt=prompt,     
    controlnet_conditioning_scale=1.0, 
    num_inference_steps=50,
    image = control_tensor,
    generator=generator
).images[0]

result_image = paste_fg_over_image(gen_img, image, mask)