EvalCore is an open-source test runner for LLM apps and agents that records behavior and replays it in CI.

How does EvalCore avoid flaky AI tests?

It records model or judge calls to a local SQLite cassette and replays the same requests offline with identical results.

What kinds of targets does EvalCore support?

It supports OpenAI-compatible models, REST APIs, shell commands, and OTel or OpenInference traces.

Do I need API keys in CI?

No. Replay mode runs offline, so it does not need network access or API keys.

Can EvalCore compare models and enforce gates?

Yes. It supports model comparisons, baselines, pass-rate gates, and regression checks.

EvalCore: Offline snapshot testing for AI behavior

Name: EvalCore
Rating: 5 (80 reviews)
Author: toolify

Sponsored by SocQ - AI-Agent-Ready Social API. Simple. Reliable. Scalable.

Free Tools Category Jobs

AI Ad Library

Home

AI Testing

EvalCore

Open site

0 Reviews

0 Saved

Introduction:

Offline snapshot testing for AI behavior

Added on:

Jul 24 2026

Monthly Visitors:

Social & Email:

Free

Website

AI Testing AI Developer Tools Large Language Models (LLMs)AI Agent

Advertise this tool Update this tool

SocQ

AI-Agent-Ready Social API. Simple. Reliable. Scalable.

Tripo AI

AI-powered 3D model generator from images and text.

freebeat AI

Turn Music & Ideas into Viral Videos In One Click

Tokenhot

Unified LLM API gateway for 100+ models with up to 90% cost savings.

Chatbot App

Multi-Model AI Chat Platform that lets you switch between 30+ leading AI models instantly or run them side by side, including ChatGPT, Claude, Gemini, and more, all in one place.

EvalCore Product Information

What is EvalCore?

EvalCore is a single-binary test runner for LLM apps and agents. It lets you define test cases and scorers in YAML, record live behavior, and replay it offline in CI to catch regressions.

How to use EvalCore?

Install the EvalCore binary, create an evals.yaml file and a JSONL dataset, then point it at an OpenAI-compatible, REST, or shell target. Run it once against the live system to record a cassette, and then replay the suite in CI with baselines, trials, and scorers.

EvalCore's Core Features

Single-binary runner for LLM apps and agents

YAML-based eval definitions and scorers

Record and replay model calls with offline deterministic CI runs

Support for OpenAI-compatible, REST, shell, and trace-based targets

Baseline checks and regression gating

Model comparisons, trials, and reporting

Local SQLite cassette storage

OpenTelemetry and OpenInference trace support

EvalCore's Use Cases

Run AI regression tests on every pull request

Replay recorded LLM outputs offline in CI without API keys

Compare two model versions before shipping a prompt or dependency change

Snapshot-test agent behavior and gate releases on pass rate

FAQ from EvalCore

What is EvalCore?
How does EvalCore avoid flaky AI tests?
What kinds of targets does EvalCore support?
Do I need API keys in CI?
Can EvalCore compare models and enforce gates?

Articos

Articos is a fast, recruitment free user research platform that helps you validate product ideas, test UX flows, and understand customer needs without waiting weeks to find real participants. Instead of booking calls and chasing no shows, you run AI moderated interviews with realistic synthetic users that match your target personas. In a short time, you get clear feedback on what people understand, what confuses them, what they would pay for, and what would stop them from using your product. It is built for founders, product managers, designers, and agencies who need quick direction before they commit time and budget to building the wrong thing.

Try Now