Total runs:
40.1K
Run Growth:
32.0K
Growth Rate:
79.77%
Sponsored by
Zawa
KBlueLeaf / TIPO-200M
Model's Last Updated: November 29 2024
text-generation
Introduction of TIPO-200M
Model Details of TIPO-200M
TIPO: Text to Image with text presampling for Prompt Optimization
200M LLaMA arch model trained for TIPO.
Tech Report:
https://hackmd.io/@KBlueLeaf/BJULOQBR0
Introduction
In this project, we introduce "TIPO" ( T ext to I mage with text presampling for P rompt O ptimization), an innovative framework designed to significantly enhance the quality and usability of Text-to-Image (T2I) generative models. TIPO utilizes the Large Language Models (LLMs) to perform "Text Presampling" within the inference pipeline of text-to-image generative modeling. By refining and extending user input prompts, TIPO enables generative models to produce superior results with minimal user effort, making T2I systems more accessible and effective for a wider range of users.
Usage
Use updated version of DTG extension (renamed to z-tipo-extension), current version of z-tipo-extension support stable-diffusion-webui, stable-diffusion-webui-forge and ComfyUI. SD-Next haven't been tested. https://github.com/KohakuBlueleaf/z-tipo-extension
Model arch and Training
This model is LLaMA arch with 200M parameters, the training data is combined version of Danbooru2023, GBC10M and Coyo-HD-11M.
The total token seen is around 40B tokens.
For more information please refer to the tech report and following table.
| TIPO-200M | TIPO-500M | |
|---|---|---|
| Arch | LLaMA | LLaMA |
| Max ctx length | 1024 | 1024 |
| Batch Size | 2048 | 3584 |
| Training dataset |
Danbooru, GBC10M, 5epoch
Danbooru, GBC10M, Coyo11M, 3epoch |
Danbooru, GBC10M, Coyo11M, 5epoch |
| Real Token Seen* | 40B token | 30B token |
| Training Hardware | RTX 3090 x 4 | H100 x 8 |
| Training Time | 420 hour` | 100 hour` |
| URL | KBlueLeaf/TIPO-200M · Hugging Face | KBlueLeaf/TIPO-500M · Hugging Face |
*: We only count "non-padding token" in the token seen, since all the training data have very large length range
`: Since the training data is pretty short, it cost more time to reach same token seen than general LLM pretraining.
As reference, with 4096 as max ctx length and almost all the data have reach that length, you may only need 2days to reach 10B token seen on RTX 3090 x 4 with 200M model.
Evaluation
We have tested TIPO in several metric:
1. Aesthetic Score (Higher is Better)
We compute the Aesthetic Score using the Aesthetic Predictor V2.5 . This metric is calculated on the short/truncated long test.
Figure 1: Aesthetic Score distribution.
2. AI Corrupt Score (Higher is Better)
The AI Corrupt Score is obtained from the AICorruptMetrics in sdeval .
This metric is calculated on the short/truncated long test.
Figure 2: AI Corrupt Score distribution.
3. Frechet Dino Distance (FDD) on Scenery Tag Test
We use FDD on the Scenery Tag Test to demonstrate that when input prompts address a smaller distribution, the model struggles to generate images that reflect the true distribution. However, with TIPO , this issue is mitigated.
| FDD Model |
<meta> scenery
only
|
<meta> scenery
+ TIPO
|
|---|---|---|
| DinoV2 ViT-S | 0.1917 | 0.1786 |
| DinoV2 ViT-B | 0.2002 | 0.1755 |
| DinoV2 ViT-L | 0.2017 | 0.1863 |
| DinoV2 ViT-G | 0.2359 | 0.2096 |
Table 1: Frechet Dino Distance (FDD) on Scenery Tag Test.
LICENSE
This model is released under
Kohaku License 1.0
You can check the above provided URL or check the LICENSE file in this repo.
Citation
@misc{yeh2024tipo,
title = {TIPO: Text to Image with text presampling for Prompt Optimization},
author = {Yeh, Shih-Ying},
year = {2024},
month = {9},
day = {29},
note = {Technical report available at \url{https://hackmd.io/@KBlueLeaf/BJULOQBR0}.
Model available at \url{https://huggingface.co/KBlueLeaf/TIPO-500M}.
Source code available at \url{https://github.com/KohakuBlueleaf/KGen}},
}
Runs of KBlueLeaf TIPO-200M on huggingface.co
1.6K
Total runs
0
24-hour runs
48
3-day runs
57
7-day runs
-1.7K
30-day runs
More Information About TIPO-200M huggingface.co Model
Url of TIPO-200M
TIPO-200M huggingface.co Url
Provider of TIPO-200M huggingface.co
Other API from KBlueLeaf
Updated:November 29 2024
Total runs:
26.9K
Run Growth:
-10.3K
Growth Rate:
-38.08%
Total runs:
11.4K
Run Growth:
-3.5K
Growth Rate:
-29.21%
Total runs:
6.2K
Run Growth:
1.4K
Growth Rate:
21.25%
Total runs:
6.1K
Run Growth:
1.3K
Growth Rate:
20.79%
Total runs:
3.4K
Run Growth:
558
Growth Rate:
16.57%
Total runs:
1.6K
Run Growth:
720
Growth Rate:
44.55%
Total runs:
518
Run Growth:
-50
Growth Rate:
-9.43%
Total runs:
355
Run Growth:
40
Growth Rate:
11.27%
Total runs:
198
Run Growth:
143
Growth Rate:
72.22%
Total runs:
181
Run Growth:
0
Growth Rate:
0.00%
Total runs:
117
Run Growth:
90
Growth Rate:
80.36%
Total runs:
83
Run Growth:
36
Growth Rate:
45.00%
Total runs:
46
Run Growth:
-94
Growth Rate:
-200.00%
Total runs:
44
Run Growth:
15
Growth Rate:
34.88%
Total runs:
29
Run Growth:
14
Growth Rate:
46.67%
Total runs:
20
Run Growth:
11
Growth Rate:
55.00%
Total runs:
12
Run Growth:
4
Growth Rate:
33.33%
Total runs:
8
Run Growth:
5
Growth Rate:
62.50%
Total runs:
8
Run Growth:
-27
Growth Rate:
-337.50%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%