Total runs:
7.8K
Run Growth:
0
Growth Rate:
0.00%
Sponsored by
APIMart
Rocketknight1 / evo-1k-test
Model's Last Updated: April 23 2024
text-generation
Introduction of evo-1k-test
Model Details of evo-1k-test
Evo-1 (Phase 1)
About
Evo is a biological foundation model capable of long-context modeling and design.
Evo uses the StripedHyena architecture to enable modeling of sequences at a single-nucleotide, byte-level resolution with near-linear scaling of compute and memory relative to context length. Evo has 7 billion parameters and is trained on OpenGenome, a prokaryotic whole-genome dataset containing ~300 billion tokens.
Technical details about Evo can be found in our preprint and our accompanying blog posts. Evo was collaboratively developed by the Arc Institute and TogetherAI.
As part of our commitment to open science, we release weights of 15 intermediate pretraining checkpoints for phase 1 and phase 2 of pretraining. The checkpoints are available as branches of the corresponding HuggingFace repository.
Evo-1 (Phase 1) is our first model in the Evo family, trained at a context length of 8k.
| Checkpoint Name | Description |
|---|---|
evo-1-8k-base
|
A model pretrained with 8,192 context. We use this model as the base model for molecular-scale finetuning tasks. |
evo-1-131k-base
|
A model pretrained with 131,072 context using
evo-1-8k-base
as the initialization. We use this model to reason about and generate sequences at the genome scale.
|
Model Architecture
StripedHyena is a deep signal processing, hybrid architecture composed of multi-head attention and gated convolutions arranged in Hyena blocks, improving over decoder-only Transformers.
StripedHyena is designed to leverage the specialization of each of its layer classes, with Hyena layers implementing the bulk of the computation required for sequence processing and attention layers supplementing the ability to perform targeted pattern recall.
Some highlights of the architecture:
- Efficient autoregressive generation via a recurrent mode (>500k generation with a single 80GB GPU)
- Significantly faster training and finetuning at long context (>3x at 131k)
- Improved scaling laws over state-of-the-art architectures (e.g., Transformer++) on both natural language and biological sequences.
- Robust to training beyond the compute-optimal frontier e.g., training way beyond Chinchilla-optimal token amounts (see preprint for details -- more details to come)
How to use Evo
Example usage is provided in the standalone repo .
Parametrization for Inference and Finetuning
One of the advantages of deep signal processing models is their flexibility. Different parametrizations of convolutions can be used depending on the memory, expressivity and causality requirements of pretraining, finetuning or inference workloads.
The main classes are:
- Modal canonical: unconstrained poles ( reference , reference ), or constrained poles ( reference , reference ).
- Companion canonical / rational: TBA.
- Hypernetworks: hypernetwork ( reference ), modulated hypernetwork ( reference ).
- Explicit: modulated explicit ( reference ).
StripedHyena is a mixed precision model. Make sure to keep your
poles
and
residues
in
float32
precision, especially for longer prompts or training.
Disclaimer
To use StripedHyena outside of the playground, you will need to install custom kernels. Please follow the instructions from the standalone repository .
Cite
@article{nguyen2024sequence,
author = {Eric Nguyen and Michael Poli and Matthew G. Durrant and Armin W. Thomas and Brian Kang and Jeremy Sullivan and Madelena Y. Ng and Ashley Lewis and Aman Patel and Aaron Lou and Stefano Ermon and Stephen A. Baccus and Tina Hernandez-Boussard and Christopher Ré and Patrick D. Hsu and Brian L. Hie},
journal = {Arc Institute manuscripts},
title = {Sequence modeling and design from molecular to genome scale with Evo},
url = {https://arcinstitute.org/manuscripts/Evo},
year = {2024},
}
Runs of Rocketknight1 evo-1k-test on huggingface.co
29
Total runs
0
24-hour runs
-1
3-day runs
0
7-day runs
-1
30-day runs
More Information About evo-1k-test huggingface.co Model
Url of evo-1k-test
evo-1k-test huggingface.co Url
Provider of evo-1k-test huggingface.co
Other API from Rocketknight1
Updated:February 09 2024
Total runs:
1.2K
Run Growth:
-58
Growth Rate:
-4.64%
Total runs:
967
Run Growth:
158
Growth Rate:
16.14%
Total runs:
468
Run Growth:
-514
Growth Rate:
-117.35%
Total runs:
195
Run Growth:
-167
Growth Rate:
-85.64%
Total runs:
96
Run Growth:
-19
Growth Rate:
-23.46%
Total runs:
90
Run Growth:
57
Growth Rate:
63.33%
Total runs:
89
Run Growth:
-1
Growth Rate:
-1.20%
Total runs:
67
Run Growth:
-13
Growth Rate:
-18.57%
Total runs:
63
Run Growth:
-3
Growth Rate:
-4.41%
Total runs:
62
Run Growth:
-40
Growth Rate:
-57.97%
Total runs:
52
Run Growth:
11
Growth Rate:
21.15%
Total runs:
47
Run Growth:
0
Growth Rate:
0.00%
Total runs:
46
Run Growth:
6
Growth Rate:
13.04%
Total runs:
41
Run Growth:
18
Growth Rate:
43.90%
Total runs:
39
Run Growth:
9
Growth Rate:
23.68%
Total runs:
37
Run Growth:
-105
Growth Rate:
-291.67%
Total runs:
36
Run Growth:
-37
Growth Rate:
-97.37%
Total runs:
32
Run Growth:
19
Growth Rate:
61.29%
Total runs:
29
Run Growth:
-59
Growth Rate:
-196.67%
Total runs:
26
Run Growth:
-20
Growth Rate:
-71.43%
Total runs:
26
Run Growth:
-6
Growth Rate:
-23.08%
Total runs:
25
Run Growth:
11
Growth Rate:
45.83%
Total runs:
24
Run Growth:
-105
Growth Rate:
-437.50%
Total runs:
24
Run Growth:
9
Growth Rate:
36.00%
Total runs:
24
Run Growth:
6
Growth Rate:
25.00%
Total runs:
21
Run Growth:
-230
Growth Rate:
-1000.00%
Total runs:
21
Run Growth:
8
Growth Rate:
38.10%
Total runs:
21
Run Growth:
-9
Growth Rate:
-50.00%
Total runs:
20
Run Growth:
10
Growth Rate:
50.00%
Total runs:
19
Run Growth:
6
Growth Rate:
30.00%
Total runs:
19
Run Growth:
9
Growth Rate:
47.37%
Total runs:
19
Run Growth:
1
Growth Rate:
4.76%
Total runs:
19
Run Growth:
-1
Growth Rate:
-5.26%
Total runs:
16
Run Growth:
9
Growth Rate:
56.25%
Total runs:
15
Run Growth:
-2
Growth Rate:
-11.11%
Total runs:
15
Run Growth:
0
Growth Rate:
0.00%
Total runs:
15
Run Growth:
4
Growth Rate:
30.77%
Total runs:
13
Run Growth:
3
Growth Rate:
18.75%
Total runs:
13
Run Growth:
2
Growth Rate:
15.38%
Total runs:
13
Run Growth:
-1
Growth Rate:
-9.09%
Total runs:
13
Run Growth:
-3
Growth Rate:
-23.08%
Total runs:
13
Run Growth:
-1
Growth Rate:
-9.09%
Total runs:
12
Run Growth:
-7
Growth Rate:
-58.33%
Total runs:
12
Run Growth:
-2
Growth Rate:
-13.33%
Total runs:
12
Run Growth:
0
Growth Rate:
0.00%
Total runs:
12
Run Growth:
2
Growth Rate:
13.33%
Total runs:
12
Run Growth:
3
Growth Rate:
25.00%
Total runs:
11
Run Growth:
-1
Growth Rate:
-8.33%
Total runs:
11
Run Growth:
7
Growth Rate:
63.64%
Total runs:
11
Run Growth:
-1
Growth Rate:
-8.33%
Total runs:
11
Run Growth:
-1
Growth Rate:
-9.09%
Total runs:
11
Run Growth:
-56
Growth Rate:
-509.09%
Total runs:
11
Run Growth:
-10
Growth Rate:
-90.91%
Total runs:
10
Run Growth:
-5
Growth Rate:
-50.00%
Total runs:
10
Run Growth:
1
Growth Rate:
12.50%
Total runs:
9
Run Growth:
-8
Growth Rate:
-88.89%
Total runs:
9
Run Growth:
-3
Growth Rate:
-33.33%
Total runs:
9
Run Growth:
0
Growth Rate:
0.00%
Total runs:
9
Run Growth:
-3
Growth Rate:
-27.27%
Total runs:
8
Run Growth:
-16
Growth Rate:
-200.00%
Total runs:
8
Run Growth:
-13
Growth Rate:
-185.71%
Total runs:
8
Run Growth:
-8
Growth Rate:
-100.00%
Total runs:
8
Run Growth:
-14
Growth Rate:
-200.00%
Total runs:
8
Run Growth:
-4
Growth Rate:
-50.00%
Total runs:
8
Run Growth:
-1
Growth Rate:
-12.50%
Total runs:
7
Run Growth:
-26
Growth Rate:
-433.33%
Total runs:
7
Run Growth:
0
Growth Rate:
0.00%
Total runs:
6
Run Growth:
2
Growth Rate:
33.33%
Total runs:
5
Run Growth:
-3
Growth Rate:
-75.00%
Total runs:
4
Run Growth:
2
Growth Rate:
50.00%
Total runs:
3
Run Growth:
1
Growth Rate:
33.33%
Total runs:
2
Run Growth:
-1
Growth Rate:
-50.00%
Total runs:
1
Run Growth:
1
Growth Rate:
100.00%
Total runs:
1
Run Growth:
-5
Growth Rate:
-500.00%
Total runs:
1
Run Growth:
-1
Growth Rate:
-100.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%