BAAI / Aquila2-34B

English | 简体中文 |

We opensource our Aquila2 series, now including Aquila2 , the base language models, namely Aquila2-7B and Aquila2-34B , as well as AquilaChat2 , the chat models, namely AquilaChat2-7B and AquilaChat2-34B , as well as the long-text chat models, namely AquilaChat2-7B-16k and AquilaChat2-34B-16k

The additional details of the Aquila model will be presented in the official technical report. Please stay tuned for updates on official channels.

Updates 2024.6.6

We have updated the basic language model Aquila2-34B , which has the following advantages compared to the previous model:

• Replaced tokenizer with higher compression ratio:

• The maximum processing length supported by the model has increased from 2048 to 8192

Quick Start Aquila2-34B

1. Inference

Aquila2-34B is a base model that can be used for continuation.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers import BitsAndBytesConfig

device= "cuda:0"

# Model Name
model_name = 'BAAI/Aquila2-34B'

# load model and tokenizer
quantization_config=BitsAndBytesConfig(
                        load_in_4bit=True,
                        bnb_4bit_use_double_quant=True,
                        bnb_4bit_quant_type="nf4",
                        bnb_4bit_compute_dtype=torch.bfloat16,
                    )
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, trust_remote_code=True,
                        # quantization_config=quantization_config # Uncomment this one for 4-bit quantization
                        )

tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True)

model.eval()

model.to(device)

# Example
text = "The meaning of life is"
tokens = tokenizer.encode_plus(text)['input_ids']
tokens = torch.tensor(tokens)[None,].to(device)

with torch.no_grad():
        out = model.generate(tokens, do_sample=False, max_length=128, eos_token_id=tokenizer.eos_token_id)[0]
        out = tokenizer.decode(out.cpu().numpy().tolist())
        print(out)

License

Aquila2 series open-source model is licensed under BAAI Aquila Model Licence Agreement