Nous-Yarn-Llama-2-13b-64k is a state-of-the-art language model for long context, further pretrained on long context data for 400 steps.
This model is the Flash Attention 2 patched version of the original model:
https://huggingface.co/conceptofmind/Yarn-Llama-2-13b-64k
Note that this model
requires
the
Flash Attention library
in order to function correctly, see the Model Usage section for installation instructions.
Model Training
Starting from the base Llama 2 models, this model was further pretrained on a subset of the PG19 dataset, allowing it to effectively utilize up to 64k tokens of context.
The authors would like to thank Stability AI, Carper AI, and Eleuther AI for their generous support of significant computing resources that enabled the training of these models and the completion of this research. We would also like to thank Jonathan Tow and Dakota Mahan directly for their help in advising on the use of the Stability AI compute cluster. Additionally, we would like to thank a16z, and PygmalionAI, for providing resources to run evaluations and experiments on the models.
There are no specific prompt formats as this is a pretrained base model.
Benchmark Results
TODO
Future Plans
We plan to continue training when we have more compute and to improve the dataset and/or instruct tune the models in order to improve the long context performance even further.
Model Usage
The model is available for download on HuggingFace.
Runs of NousResearch Yarn-Llama-2-13b-64k on huggingface.co
309
Total runs
0
24-hour runs
0
3-day runs
286
7-day runs
287
30-day runs
More Information About Yarn-Llama-2-13b-64k huggingface.co Model
Yarn-Llama-2-13b-64k huggingface.co
Yarn-Llama-2-13b-64k huggingface.co is an AI model on huggingface.co that provides Yarn-Llama-2-13b-64k's model effect (), which can be used instantly with this NousResearch Yarn-Llama-2-13b-64k model. huggingface.co supports a free trial of the Yarn-Llama-2-13b-64k model, and also provides paid use of the Yarn-Llama-2-13b-64k. Support call Yarn-Llama-2-13b-64k model through api, including Node.js, Python, http.
Yarn-Llama-2-13b-64k huggingface.co is an online trial and call api platform, which integrates Yarn-Llama-2-13b-64k's modeling effects, including api services, and provides a free online trial of Yarn-Llama-2-13b-64k, you can try Yarn-Llama-2-13b-64k online for free by clicking the link below.
NousResearch Yarn-Llama-2-13b-64k online free url in huggingface.co:
Yarn-Llama-2-13b-64k is an open source model from GitHub that offers a free installation service, and any user can find Yarn-Llama-2-13b-64k on GitHub to install. At the same time, huggingface.co provides the effect of Yarn-Llama-2-13b-64k install, users can directly use Yarn-Llama-2-13b-64k installed effect in huggingface.co for debugging and trial. It also supports api for free installation.
Yarn-Llama-2-13b-64k install url in huggingface.co: