abenzerps / Hy-Memory-30B-A3B-Quantized

Hy-Memory-30B-A3B Quantized

GGUF, Safetensors, and MLX quantized formats of tencent/Hy-Memory-30B-A3B for local memory operations and agent execution.

Tencent's Hy-Memory-30B-A3B is a specialized Mixture-of-Experts (MoE) model (30B total parameters, 3B active per token) trained specifically as a high-performance memory writer and agent-memory operator. It excels at session summarization, context compaction, entity memory extraction, and agent memory management.

Benchmark

Quantized Files

Q4_K_M is recommended for the best balance of size and quality for GGUF. NVFP4 and FP8 offer high performance on supported NVIDIA GPUs, while MLX formats are optimized for Apple Silicon.

Usage

1. llama.cpp / llama-server (GGUF)

Interactive Chat via CLI

Run multi-turn interactive conversation directly in your terminal:

./llama-cli -m Hy-Memory-30B-A3B-Q4_K_M.gguf \
    -cnv \
    -c 8192 \
    -n 512 \
    --temp 0.6 \
    -t 16

OpenAI-Compatible API Server

Start llama-server to serve OpenAI-compatible /v1/chat/completions endpoints:

./llama-server -m Hy-Memory-30B-A3B-Q4_K_M.gguf \
    -c 16384 \
    --port 8080

Query via curl:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {
        "role": "user",
        "content": "Extract key entities and memory facts from this text: Alice is a software architect living in Berlin who prefers dark mode and Python."
      }
    ],
    "temperature": 0.6
  }'

2. Ollama (GGUF)

Create a Modelfile :

FROM ./Hy-Memory-30B-A3B-Q4_K_M.gguf

TEMPLATE """{{- if .System }}<｜hy_User｜>{{ .System }}
{{ end }}{{- range .Messages }}<｜hy_{{ if eq .Role "user" }}User{{ else }}Assistant{{ end }}｜>{{ .Content }}{{ if eq .Role "assistant" }}<eos:6124c78e>{{ end }}{{- end }}{{- if .Prompt }}<｜hy_User｜>{{ .Prompt }}<｜hy_Assistant｜>{{- end }}"""

PARAMETER stop "<eos:6124c78e>"
PARAMETER stop "<｜hy_User｜>"
PARAMETER stop "<｜hy_Assistant｜>"
PARAMETER stop "<｜hy_begin▁of▁sentence｜>"
PARAMETER temperature 0.6

Build and run:

ollama create hy-memory -f Modelfile
ollama run hy-memory

3. Apple Silicon (MLX Safetensors)

For Apple Silicon Macs, load MLX quantized weights ( 4-bit , 6-bit , or 8-bit ):

import mlx.core as mx
from safetensors.numpy import load_file

# Load MLX-quantized weights
weights = load_file("hy-memory-30b-a3b-MLX-4bit.safetensors")

# Convert to native MLX array dictionary
mlx_model_weights = {k: mx.array(v) for k, v in weights.items()}
print(f"Loaded {len(mlx_model_weights)} MLX tensor parameters successfully.")

4. NVIDIA FP8 & NVFP4 (Safetensors)

Load the quantized weights directly via safetensors for TensorCore-accelerated pipelines (such as comfy-kitchen , vLLM , or custom inference):

import safetensors.torch
import torch

# Load FP8 (torch.float8_e4m3fn with per-tensor scales)
fp8_weights = safetensors.torch.load_file("hy-memory-30b-a3b-fp8.safetensors")

# Load NVFP4 (packed 4-bit with block-wise scales)
nvfp4_weights = safetensors.torch.load_file("hy-memory-30b-a3b-NVFP4.safetensors")

Source and Build

• Source model: tencent/Hy-Memory-30B-A3B
• Architecture: HYV3ForCausalLM (MoE: 128 experts, 8 active per token, 30B total / 3B active)
• Conversion & Quantization: llama.cpp & comfy-kitchen
• License: Apache 2.0
• Checksums: SHA256SUMS