AlexHung29629 / Draft1

huggingface.co
Total runs: 2
24-hour runs: -1
7-day runs: -1
30-day runs: -9
Model's Last Updated: August 06 2025
text-generation

Introduction of Draft1

Model Details of Draft1

Built with Axolotl

See axolotl config

axolotl version: 0.12.0.dev0

base_model: out_khanacademy
remove_unused_columns: true
auto_resume_from_checkpoints: true
plugins:
  - axolotl.integrations.liger.LigerPlugin
  #- axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin

liger_rope: true
liger_rms_norm: true
liger_glu_activation: true
liger_fused_linear_cross_entropy: true

unfrozen_parameters:
  - ^\S+layers\S+$
  - ^\S+norm\S+$


datasets:
  - path: AlexHung29629/nllb_processed
    split: train[:1_000_000]
    type: chat_template
    chat_template: jinja
    chat_template_jinja: "{{ bos_token }}{% for message in messages %}{% if (message['role'] == 'system') %}{{message['content'] + '\n'}}{% elif (message['role'] == 'user') %}{{'Source: ' + '\n' + message['content'] + '\n' + '\nTarget:\n'}}{% elif message['role'] == 'assistant' %}{{message['content'] + '</s>' + '\n'}}{% endif %}{% endfor %}"
    roles_to_train: ['user', 'assistant']

#test_datasets:
#  - path: HuggingFaceTB/cosmopedia
#    name: khanacademy
#    split: train[-100:]
#    type:
#      system_prompt: ""
#      field_system: 
#      field_instruction: prompt
#      field_output: text
#      format: "User: {instruction}\n\nAssistant: "
#      no_input_format: "User: {instruction}\n\nAssistant: "
      
sample_packing_bin_size: 500
dataset_prepared_path: data_prep_nllb
output_dir: ./out_nllb
dataloader_num_workers: 1
dataloader_pin_memory: true
shuffle_merged_datasets: false

sequence_len: 8192
eval_sequence_len: 2048
sample_packing: true
eval_sample_packing: true
pad_to_sequence_len: true

use_tensorboard: true
use_wandb: true
# Set the name of your wandb run
wandb_name: nllb
# Your wandb project name
wandb_project: Draft_Tiny

gradient_accumulation_steps: 1

micro_batch_size: 1
num_epochs: 1
#eval_steps: 500
save_steps: 1000
save_total_limit: 1
save_only_model: false
optimizer: adamw_8bit
adam_beta1: 0.9
adam_beta2: 0.95
adam_epsilon: 1e-6
lr_scheduler: constant_with_warmup
learning_rate: 0.0003
max_grad_norm: 1.0

bf16: true

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false

torch_compile: true
torch_compile_backend: inductor
torch_compile_mode: default
#flash_attention: true
#sdp_attention: true
#xformers_attention: true
flex_attention: true
flex_attn_compile_kwargs:
  dynamic: false
  mode: max-autotune-no-cudagraphs

warmup_steps: 1
logging_steps: 1
weight_decay: 0.001


special_tokens:
  bos_token: <s>
  eos_token: </s>
  pad_token: <pad>
  unk_token: <unk>

out_nllb

This model was trained from scratch on the AlexHung29629/nllb_processed dataset.

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure
Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0003
  • train_batch_size: 1
  • eval_batch_size: 1
  • seed: 42
  • optimizer: Use OptimizerNames.ADAMW_8BIT with betas=(0.9,0.95) and epsilon=1e-06 and optimizer_args=No additional optimizer arguments
  • lr_scheduler_type: constant_with_warmup
  • lr_scheduler_warmup_steps: 2
  • training_steps: 13786
Training results
Framework versions
  • Transformers 4.54.1
  • Pytorch 2.7.1+cu128
  • Datasets 4.0.0
  • Tokenizers 0.21.4

Runs of AlexHung29629 Draft1 on huggingface.co

2
Total runs
-1
24-hour runs
0
3-day runs
-1
7-day runs
-9
30-day runs

More Information About Draft1 huggingface.co Model

Draft1 huggingface.co

Draft1 huggingface.co is an AI model on huggingface.co that provides Draft1's model effect (), which can be used instantly with this AlexHung29629 Draft1 model. huggingface.co supports a free trial of the Draft1 model, and also provides paid use of the Draft1. Support call Draft1 model through api, including Node.js, Python, http.

AlexHung29629 Draft1 online free

Draft1 huggingface.co is an online trial and call api platform, which integrates Draft1's modeling effects, including api services, and provides a free online trial of Draft1, you can try Draft1 online for free by clicking the link below.

AlexHung29629 Draft1 online free url in huggingface.co:

https://huggingface.co/AlexHung29629/Draft1

Draft1 install

Draft1 is an open source model from GitHub that offers a free installation service, and any user can find Draft1 on GitHub to install. At the same time, huggingface.co provides the effect of Draft1 install, users can directly use Draft1 installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

Draft1 install url in huggingface.co:

https://huggingface.co/AlexHung29629/Draft1

Url of Draft1

Provider of Draft1 huggingface.co

AlexHung29629
ORGANIZATIONS

Other API from AlexHung29629