from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer, TextStreamer
model_path = "solidrust/TinyLlama-1.1B-32k-Instruct-AWQ"
system_message = "You are TinyLlama-1.1B-32k-Instruct, incarnated as a powerful AI. You were created by Doctor-Shotgun."# Load model
model = AutoAWQForCausalLM.from_quantized(model_path,
fuse_layers=True)
tokenizer = AutoTokenizer.from_pretrained(model_path,
trust_remote_code=True)
streamer = TextStreamer(tokenizer,
skip_prompt=True,
skip_special_tokens=True)
# Convert prompt to tokens
prompt_template = """\<|im_start|>system{system_message}<|im_end|><|im_start|>user{prompt}<|im_end|><|im_start|>assistant"""
prompt = "You're standing on the surface of the Earth. "\
"You walk one mile south, one mile west and one mile north. "\
"You end up exactly where you started. Where are you?"
tokens = tokenizer(prompt_template.format(system_message=system_message,prompt=prompt),
return_tensors='pt').input_ids.cuda()
# Generate output
generation_output = model.generate(tokens,
streamer=streamer,
max_new_tokens=512)
About AWQ
AWQ is an efficient, accurate and blazing-fast low-bit weight quantization method, currently supporting 4-bit quantization. Compared to GPTQ, it offers faster Transformers-based inference with equivalent or better quality compared to the most commonly used GPTQ settings.
AWQ models are currently supported on Linux and Windows, with NVidia GPUs only. macOS users: please use GGUF models instead.
Runs of solidrust TinyLlama-1.1B-32k-Instruct-AWQ on huggingface.co
30
Total runs
-1
24-hour runs
-1
3-day runs
4
7-day runs
17
30-day runs
More Information About TinyLlama-1.1B-32k-Instruct-AWQ huggingface.co Model
TinyLlama-1.1B-32k-Instruct-AWQ huggingface.co
TinyLlama-1.1B-32k-Instruct-AWQ huggingface.co is an AI model on huggingface.co that provides TinyLlama-1.1B-32k-Instruct-AWQ's model effect (), which can be used instantly with this solidrust TinyLlama-1.1B-32k-Instruct-AWQ model. huggingface.co supports a free trial of the TinyLlama-1.1B-32k-Instruct-AWQ model, and also provides paid use of the TinyLlama-1.1B-32k-Instruct-AWQ. Support call TinyLlama-1.1B-32k-Instruct-AWQ model through api, including Node.js, Python, http.
TinyLlama-1.1B-32k-Instruct-AWQ huggingface.co is an online trial and call api platform, which integrates TinyLlama-1.1B-32k-Instruct-AWQ's modeling effects, including api services, and provides a free online trial of TinyLlama-1.1B-32k-Instruct-AWQ, you can try TinyLlama-1.1B-32k-Instruct-AWQ online for free by clicking the link below.
solidrust TinyLlama-1.1B-32k-Instruct-AWQ online free url in huggingface.co:
TinyLlama-1.1B-32k-Instruct-AWQ is an open source model from GitHub that offers a free installation service, and any user can find TinyLlama-1.1B-32k-Instruct-AWQ on GitHub to install. At the same time, huggingface.co provides the effect of TinyLlama-1.1B-32k-Instruct-AWQ install, users can directly use TinyLlama-1.1B-32k-Instruct-AWQ installed effect in huggingface.co for debugging and trial. It also supports api for free installation.
TinyLlama-1.1B-32k-Instruct-AWQ install url in huggingface.co: