deepcogito / cogito-671b-v2.1-FP8

Cogito v2.1 - 671B MoE

Blog Post , GitHub

The Cogito v2.1 LLMs are instruction tuned generative models. All models are released under an open license for commercial use.

• Cogito v2.1 models are hybrid reasoning models. Each model can answer directly (standard LLM), or self-reflect before answering (like reasoning models).
• The LLMs are trained using Iterated Distillation and Amplification (IDA) - an scalable and efficient alignment strategy for superintelligence using iterative self-improvement.
• The models have been optimized for coding, STEM, instruction following, general helpfulness and tool calling capabilities.
• This model is trained in over 30 languages and supports a context length of 128k.

Evaluations

Here is the model performance on some standard industry benchmarks:

For detailed evaluations, please refer to the Blog Post .

Usage

This checkpoint is a quantized version of deepcogito/cogito-671b-v2.1 and can be served on 8 H200s or 8 B200s.

To download and cache the model:

pip install transformers hf_transfer accelerate vllm
pip install sglang[all]
hf download deepcogito/cogito-671b-v2.1-FP8

With vLLM

from transformers import AutoTokenizer
from vllm import SamplingParams, LLM

model_id = "deepcogito/cogito-671b-v2.1-FP8"
tokenizer = AutoTokenizer.from_pretrained(model_id)
llm = LLM(model=model_id, tensor_parallel_size=8, gpu_memory_utilization=0.95, max_model_len=16384)
sampling_params = SamplingParams(temperature=0.6, max_tokens=8192)

prompts = ["who created you?", "how are you doing?"]

prompts = [
    tokenizer.apply_chat_template(
        [{"role": "system", "content": "Always respond in 1-2 words."}, {"role": "user", "content": prompt}],
        tokenize=False,
        add_generation_prompt=True,
        enable_thinking=False,
    )
    for prompt in prompts
]
# To enable reasoning, set `enable_thinking=True` above.

out = llm.generate(prompts, sampling_params=sampling_params)
print([res.outputs[0].text for res in out])

Tool Calling

from vllm import LLM, SamplingParams


def get_current_temperature(location: str) -> float:
    """
    Get the current temperature at a location.
    
    Args:
        location: The location to get the temperature for, in the format "City, Country"
    Returns:
        The current temperature at the specified location in the specified units, as a float.
    """
    return 22. # A real function should probably actually get the temperature!


model_id = "deepcogito/cogito-671b-v2.1-FP8"

llm = LLM(model=model_id, gpu_memory_utilization=0.9, tensor_parallel_size=8, max_model_len=16384)
sampling_params = SamplingParams(temperature=0.6, max_tokens=512)

tokenizer = llm.get_tokenizer()

def generate_output(messages):
    global tokenizer, llm, sampling_params
    prompt = tokenizer.apply_chat_template(
        messages,
        tools=[get_current_temperature],
        tokenize=False,
        add_generation_prompt=True,
        enable_thinking=False,
    )
    response = llm.generate(prompt, sampling_params)
    return response[0].outputs[0].text

messages = [{"role": "user", "content": "whats the temperature today?"}]
response = generate_output(messages)
print(response)
# 'I\'d be happy to check the temperature for you. Could you please let me know which location you\'re interested in? Please provide the city and country (e.g., "New York, USA").'

messages.append({"role": "assistant", "content": 'I\'d be happy to check the temperature for you. Could you please let me know which location you\'re interested in? Please provide the city and country (e.g., "New York, USA").'})
messages.append({"role": "user", "content": "I live in San Francisco."})

response = generate_output(messages)
print(response)
# '<|tool▁calls▁begin|><|tool▁call▁begin|>function<|tool▁sep|>get_current_temperature<|tool▁sep|>{"location": "San Francisco, USA"}<|tool▁call▁end|><|tool▁calls▁end|>'

tool_calls = [{"type": "function", "function": {"name": "get_current_temperature", "arguments": {"location": "San Francisco, USA"}}}]
messages.append({"role": "assistant", "tool_calls": tool_calls})

messages.append({"role": "tool", "name": "get_current_temperature", "content": "22.0"})
response = generate_output(messages)
print(response)
# The current temperature in San Francisco, USA is 22°C.

With SGLang

Start the local endpoint with:

# H200s
python3 -m sglang.launch_server --model deepcogito/cogito-671b-v2.1-FP8 --tp 8

# B200s
python3 -m sglang.launch_server --model deepcogito/cogito-671b-v2.1-FP8 --tp 8 --quantization compressed-tensors --moe-runner-backend triton

Then query the model:

import openai
client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="default",
    messages=[
        {"role": "system", "content": "Always respond in 1-2 words."},
        {"role": "user", "content": "Who created you?"},
    ],
    temperature=0.6,
    max_tokens=8192,
    extra_body = {"chat_template_kwargs": {"enable_thinking": False}}
)
# To enable reasoning, set `enable_thinking=True` above.
print(response.choices[0].message.content)

NOTE: We initiate the response with "<think>\n" at the beginning of every output when thinking is enabled. This is because hybrid models can be brittle at times, and adding a "<think>\n" ensures that the model does indeed respect thinking.

License

This repository and the model weights are licensed under MIT License .

Contact

If you would like to reach out to our team, send an email to [email protected] .