thedeoxen / refcontrol-FLUX.2-klein-9B-reference-canny-lora

RefControl FLUX.2 Klein 9B – Reference Canny LoRA

📝 Short description

A LoRA for FLUX.2 Klein 9B Base that fuses a reference image (identity) with a canny edge map (structure / contours).
It preserves identity and style from the reference while following the shape and composition from the canny control map.

Trigger word: refcontrol

📊 Examples

Each preview is a single combined image from ComfyUI: Canny → Reference → Result (left to right).

Canny → Reference → Result

📖 Extended description

This LoRA was primarily trained on humans , but it also works with stylized characters and some objects.
Its main purpose is to preserve identity — facial features, hairstyle, clothing, or object details — from the reference image, while adapting the subject to the structure and contours defined by the canny edge map.

FLUX.2 Klein 9B Base already handles reference + edge-guided transfer reasonably well with the right prompt alone. This LoRA builds on that capability — it improves consistency and better preserves character identity and edge fidelity than the base model without LoRA.

Part of the RefControl family: reference + control fusion for consistent, controllable generation on FLUX.2 Klein 9B Base .

⚙️ How to use

1. Use the first image as the canny edge map (structure / contours).
2. Use the second image as the reference (character, person, or object).
3. Add the trigger word refcontrol in your prompt.
4. Adjust LoRA weight (recommended 0.8–1.0 ) depending on how strongly you want to preserve identity.

ComfyUI requirenments

Canny extraction in the included workflow uses CannyEdgePreprocessor (via comfyui_controlnet_aux ):

https://github.com/Fannovel16/comfyui_controlnet_aux

You can disable the built-in canny preprocessor if your input is already a canny/edge image.

Base model

Trained on and recommended with black-forest-labs/FLUX.2-klein-base-9B .
The undistilled Base variant is intended for LoRA training and custom pipelines (~50 inference steps, guidance_scale ~4.0).

The LoRA also works with the 4-step distilled black-forest-labs/FLUX.2-klein-9B for faster inference (~4 steps, guidance_scale ~1.0), but quality may be slightly lower — especially for identity and edge fidelity — compared to the Base model.

✅ Example prompt

refcontrol

🎯 What it does

• Preserves character or object identity across generations.
• Adapts the subject to a new structure or composition defined by the canny map.
• Works best when the edge map has similar proportions and scale to the reference.

⚡ Tips

• Best results when the canny map is not drastically different in body scale or framing from the reference.
• Combine with text prompts to refine background, lighting, or mood.
• Canny edge maps on a black background work well as control input.

📌 Use cases

• Character restyling while keeping identity and edge-defined structure.
• Consistent character design across different compositions.
• Illustration and storyboard generation with edge-guided layout.
• Object transformations with contour-guided placement.

📦 Files

• Weights: flux2_klein_9b_refcontrol_canny.safetensors
• ComfyUI workflow: refcontrol_canny_flux_klein_9b.json
• Repo: thedeoxen/refcontrol-FLUX.2-klein-9B-reference-canny-lora