cloudyu / Llama-3-70B-instruct-DPO-1M

This is a merge of cloudyu/Meta-Llama-3-70B-Instruct-DPO and cognitivecomputations/Llama-3-70B-Gradient-1048k-adapter.

try to build a 1048K context Llama-3-70B-Instruct model, but as H100 limit, it's only 4bit.

example code

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
import math

## v2 models
model_path = "cloudyu/Llama-3-70B-instruct-DPO-1M"

tokenizer = AutoTokenizer.from_pretrained(model_path, use_default_system_prompt=False)
model = AutoModelForCausalLM.from_pretrained(
        model_path, torch_dtype=torch.float16, device_map='auto',local_files_only=False, load_in_4bit=True
)
print(model)
prompt = input("please input prompt:")
while len(prompt) > 0:
  input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to("cuda")

  generation_output = model.generate(
    input_ids=input_ids, max_new_tokens=500,repetition_penalty=1.2
  )
  print(tokenizer.decode(generation_output[0]))
  prompt = input("please input prompt:")

example output

<|begin_of_text|>write me a story about yosemite. would love it if there was some sort of magical realism in the story, (but not too over-the-top). here's a prompt: two friends go to yosemite for a camping trip and one friend has been struggling with anxiety/depression lately. could be really beautiful.
I'd be delighted to craft a tale that weaves together the majesty of Yosemite with a touch of magic! Here's my attempt:

As they wound their way through the Sierra foothills,, Rachel couldn't help but notice how her best friend,. It had only been a few months since Emily's mom passed away, A cloud of sadness seemed to follow Emily everywhere,, a constant reminder of what she'd lost. The once-luminous spark in her eyes now flickered like a dying ember.

Rachel hoped this camping trip would bring back some of that sparkle. She packed up her car with all the essentials—tent, sleeping bags,,, more coffee than necessary—and off they went into the heart of Yosemite National Park. As they drove deeper into the park, I'm sure the grandeur of El Capitan will lift Em's spirits, Rachel thought. But as they set up camp near Tenaya Lake,. They spent most of the first day huddled around the campfire,, silence punctuated by occasional small talk. Even the usually majestic scenery seemed muted, filtered through the lens of Emily's sorrow.

That night, under a star-studded sky, Rachel pulled out her guitar and began strumming softly. The music wove itself into the forest sounds—the rustling leaves, chirping crickets, a distant waterfall. As she played on, something peculiar happened. The stars above them started to twinkle in time with the rhythm, casting an otherworldly glow across the clearing. Emily looked up from her reverie, of grief,,,, and for the first time in weeks, a faint smile crept onto her face. In that moment,, the weight of her loss still present,, yet somehow less crushing.

The next morning, after a restless sleep, Emily woke before dawn. She slipped out of the tent, leaving Rachel snoring peacefully, and wandered toward the lake. Mist swirled above its surface,,,, imbuing the air with an ethereal quality. As she approached the water's edge,, a family of river otters emerged from the fog, playing and chasing each other along the shore. Their carefree antics brought