BiliSakura / HSIGene

BiliSakura/HSIGene

Hyperspectral image generation — HSIGene converted to diffusers format. Conditional generation with local controls (HED, MLSD, sketch, segmentation), global controls (content, text), and metadata embeddings. Outputs 48-band hyperspectral images (256×256 pixels).

Source: HSIGene . Converted to diffusers format; model dir is self-contained (no external project for inference).

Conversion

The main diffusion checkpoint ( last.ckpt ) must be downloaded from GoogleDrive and placed in projects/HSIGene-Diffusers/checkpoints/ .

Note: models/raw/HSIGene contains annotator/auxiliary models (body pose, depth, SAM, etc.) only — not the main diffusion checkpoint.

cd projects/HSIGene-Diffusers
python convert_to_diffusers.py \
    --config_path configs/inference.yaml \
    --ckpt_path checkpoints/last.ckpt \
    --output_dir /root/worksapce/models/BiliSakura/HSIGene

Repository Structure (after conversion)

Usage

Option 1 – No sys.path.insert (AeroGen-style): Load the pipeline from the model path via importlib ; the model dir is added to the path automatically.

import importlib.util
import sys

model_path = "/path/to/HSIGene"  # or "BiliSakura/HSIGene" for Hub
spec = importlib.util.spec_from_file_location("pipeline_hsigene", f"{model_path}/pipeline_hsigene.py")
mod = importlib.util.module_from_spec(spec)
sys.modules["pipeline_hsigene"] = mod
spec.loader.exec_module(mod)

pipe = mod.HSIGenePipeline.from_pretrained(model_path)
pipe = pipe.to("cuda")

Option 2 – With sys.path.insert : Simpler if you are fine adding the model dir to the path once.

import sys
sys.path.insert(0, "/path/to/HSIGene")
from pipeline_hsigene import HSIGenePipeline

pipe = HSIGenePipeline.from_pretrained("/path/to/HSIGene")
pipe = pipe.to("cuda")

Option 3 – DiffusionPipeline.from_pretrained : May work with trust_remote_code=True . If you see "raw config (list)" errors (e.g. when loading from cache), use Option 1 or 2 instead.

from diffusers import DiffusionPipeline
pipe = DiffusionPipeline.from_pretrained("/path/to/HSIGene", trust_remote_code=True)
pipe = pipe.to("cuda")

Dependencies: pip install diffusers transformers torch einops safetensors

# Conditional generation
output = pipe(
    prompt="Wasteland",
    num_samples=1,
    height=256,
    width=256,
    num_inference_steps=50,
    local_conditions=local_tensor,   # (B, 18, H, W) or None
    global_conditions=global_tensor, # (B, 768) or None
    metadata=metadata_tensor,        # (7,) or (B, 7) or None
    guidance_scale=1.0,
)
images = output.images  # (B, H, W, 48) in [0, 1]

Conditioning

• Local : 18-channel maps (HED, MLSD, sketch, segmentation, etc.) at 512×512 default.
• Global : 768-dim CLIP features from reference images.
• Metadata : 7-dim vector.
• Text : Via prompt ; use text_strength to scale.

Model Sources

• Paper : HSIGene: A Foundation Model For Hyperspectral Image Generation
• Checkpoint : GoogleDrive
• Annotators : BaiduNetdisk (code: 98f1) → data_prepare/annotator/ckpts

Citation

@misc{pang2024hsigenefoundationmodelhyperspectral,
  title={HSIGene: A Foundation Model For Hyperspectral Image Generation},
  author={Li Pang and Datao Tang and Shuang Xu and Deyu Meng and Xiangyong Cao},
  year={2024},
  eprint={2409.12470},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
}