princeton-nlp / AutoCompressor-1.3b-30k

license: apache-2.0

Paper : Adapting Language Models to Compress Contexts

Code : https://github.com/princeton-nlp/AutoCompressors

Models :

• Llama-2-7b fine-tuned models: AutoCompressor-Llama-2-7b-6k , FullAttention-Llama-2-7b-6k
• OPT-2.7b fine-tuned models: AutoCompressor-2.7b-6k , AutoCompressor-2.7b-30k , RMT-2.7b-8k , FullAttention-2.7b-4k
• OPT-1.3b fine-tuned models: AutoCompressor-1.3b-30k , RMT-1.3b-30k

AutoCompressor-1.3b-30k is a model fine-tuned from facebook/opt-1.3b following the AutoCompressor method in Adapting Language Models to Compress Contexts . This model is fine-tuned on 2B tokens from Books3 in The Pile . The pre-trained OPT-1.3b model is fine-tuned on sequences of 30,720 tokens with 50 summary vectors, summary accumulation, randomized segmenting, and stop-gradients.

To get started, download the AutoCompressor repository and load the model as follows:

from auto_compressor import AutoCompressorModel

model = AutoCompressorModel.from_pretrained("princeton-nlp/AutoCompressor-1.3b-30k")

Evaluation

We record the perplexity achieved by our 30k-fine-tuned OPT models on segments of 2,048 tokens sampled from Books3 and ArXiv in The Pile, conditioned on different amounts of context.

Bibtex

@misc{chevalier2023adapting,
      title={Adapting Language Models to Compress Contexts}, 
      author={Alexis Chevalier and Alexander Wettig and Anirudh Ajith and Danqi Chen},
      year={2023},
      eprint={2305.14788},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}