crumb / nano-mistral

Model Card for Model ID

Model Details

Model Description

This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.

• Developed by: me
• Model type: Mistral
• Language(s) (NLP): en
• License: apache

Uses

general web text completions at extremely low resource use

Out-of-Scope Use

not an instruct model

Bias, Risks, and Limitations

trained on web text, though filtered no guarantees theres not toxic stuff in there

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("crumb/nano-mistral")
tokenizer = AutoTokenizer.from_pretrained("crumb/nano-mistral")

inputs = tokenizer(["Once upon a time,"], return_tensors="pt")
inputs = {k:v.to(model.device) for k,v in dict(inputs).items()}
outputs = model.generate(inputs, max_new_tokens=128, temperature=0.7, top_k=20, do_sample=True)
outputs = tokenizer.batch_decode(outputs)
for i in outputs:
  print(i)

Training Details

Training Data

crumb/askmistral-pile-2-15

Training Procedure

Preprocessing [optional]

[More Information Needed]

Training Hyperparameters

• Training regime: bf16 non-mixed precision

Speeds, Sizes, Times [optional]

train_runtime 62541.9424

train_samples_per_second 26.557

[More Information Needed]

Evaluation

Testing Data, Factors & Metrics

Testing Data

held out set of crumb/askmistral-pile-2-15

Factors

[More Information Needed]

Metrics

open llm leaderboard eval datasets and settings

Results

OpenLLM Leaderboard Mean Score + Stderr: (29.30, 0.42)

MMLU

value, stderr = (0.253980701754386, 0.004428598058450528)

Summary

Model Examination [optional]

its ok

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019) .

• Hardware Type: A6000
• Hours used: 34.74
• Cloud Provider: n/a
• Compute Region iowa
• Carbon Emitted: 4.5kg CO2eq.

Technical Specifications [optional]

Model Architecture and Objective

mistral, causal language modelling

Compute Infrastructure

what

Hardware

lambda vector 2xA6000

Software

huggingface transformers / pytorch / custom trainer

Citation [optional]

BibTeX:

[More Information Needed]

APA:

[More Information Needed]

Glossary [optional]

[More Information Needed]

More Information [optional]

[More Information Needed]

Model Card Authors [optional]

[More Information Needed]

Model Card Contact

[More Information Needed]