ducanhdinh / Macro-Nano-Instruct-MidAlign

Macro-Nano-Instruct-MidAlign

LoRA adapter finetune tu ATH-MaaS/Marco-Nano-Instruct , mot mo hinh Mixture-of-Experts, theo baseline MidAlign (Middle-Layer Representation Alignment, Liu & Niehues 2025) — Alternate Training giua task objective (causal LM tren target language) va alignment objective (contrastive loss tai 1 middle layer) — adapt sang backbone MoE.

Cau hinh LoRA / Alignment

• LoRA + trich xuat hidden state cho contrastive loss CHI ap dung tai layer thu 16 (0-indexed block = 15) trong tong so 28 layer.
• Module duoc gan LoRA: attention , router , experts tai layer tren.
• r = 16, alpha = 32, dropout = 0.05
• Nhiet do contrastive tau = 0.1

Loss (Alternate Training — moi step chi 1 trong 2)

• Task step : L_task = L_LM + lb_loss_coef * L_LB
  • L_LM : causal LM loss tren cau TARGET LANGUAGE (phia "other" trong cap english-other).
  • L_LB : load balancing loss chuan cua MoE tai router trong layer duoc finetune.
  • lb_loss_coef = None, num_experts = 232, top_k = 8
• Align step : L_align = symmetric InfoNCE / contrastive loss (in-batch negatives) giua mean-pooled hidden state cua cau tieng Anh va cau target tai layer 16.

Du lieu

Cap bitext english-other duoc sample tu cac bo du lieu multiway-parallel: flores.json, bible.json, ntrex.json . Voi moi record, cau eng_Latn duoc ghep voi tung ngon ngu khac trong cung record de tao 1 cap bitext rieng.

Training

• 3 epoch, batch_size = 64 (per-process).
• Multi-GPU: DistributedDataParallel (torchrun), checkpoint chi giu ban moi nhat.

Diagnostics

Xem diagnostics/loss_log.jsonl (log theo tung step, phan biet step_type=task/align) va diagnostics/loss_curve.png .