cloudyu / Llama-3-8Bx2-MOE-DPO

Apple MPS code example

import transformers
import torch

model_id = "cloudyu/Llama-3-8Bx2-MOE-DPO"

pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.float16},
    device_map="mps",
)

prompt = "what is biggest animal in earth?"
while len(prompt)>0:
  messages = [
    {"role": "system", "content": "You are a nice chatbot who always responds in kindly speak!"},
    {"role": "user", "content": prompt},
  ]

  prompt = pipeline.tokenizer.apply_chat_template(
        messages, 
        tokenize=False, 
        add_generation_prompt=True
  )

  terminators = [
    pipeline.tokenizer.eos_token_id,
    pipeline.tokenizer.convert_tokens_to_ids("<|eot_id|>")
  ]

  outputs = pipeline(
    prompt,
    max_new_tokens=1024,
    eos_token_id=terminators,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
  )
  print(outputs[0]["generated_text"][len(prompt):])

  prompt=input("please input prompt:\n")

example output

write me a story about yosemite.
Dear friend, I'd be delighted to spin a tale about the breathtaking beauty of Yosemite National Park!

In the heart of California's Sierra Nevada mountains, where the granite walls rise high and the valleys stretch far, there's a place that's been a treasured haven for nature lovers and adventure seekers alike. Yosemite, with its majestic waterfalls, towering sequoias, and serene lakes, is a haven that's sure to capture the hearts of all who visit.

Once upon a time, a young explorer named Lily set out to discover the wonders of Yosemite. As she wandered through the park's lush meadows, she stumbled upon a hidden clearing, surrounded by towering trees that seemed to whisper secrets to the wind. The air was filled with the sweet scent of blooming wildflowers, and the gentle chirping of birds accompanied her every step.

As she explored further, Lily chanced upon the mighty El Capitan, its rugged face a testament to the power of nature. She watched in awe as the sun began to set, casting a golden glow over the landscape, and the granite monolith seemed to come alive, its shadows dancing across the valley floor.

As night began to fall, Lily settled in at a cozy campsite, surrounded by the soothing sounds of the forest. She gazed up at the star-studded sky, feeling as small yet connected to the vast expanse of the universe. The night air was filled with the scent of campfires and s'mores, and she felt grateful to be a part of this magical place.

The next morning, Lily set out to explore the park's iconic waterfalls. She hiked through the misty veil of Vernal Falls, feeling the cool spray on her face, and marveled at the sheer force of Bridalveil Fall, its delicate veil of water suspended high above the valley floor.

As the sun began to set once more, Lily made her way to the park's scenic overlook, where she beheld the breathtaking view of Yosemite Valley. The towering cliffs, the serene lakes, and the lush meadows all blended together in a tapestry of natural beauty, a true masterpiece of creation.

And so, dear friend, I hope you've enjoyed this tale of Yosemite's wonders. May it inspire you to visit this enchanted land, where the beauty of nature is sure to leave you in awe.