yasserrmd / enterprise-reflex-v1

Enterprise Reflex V1

Model: yasserrmd/enterprise-reflex-v1
Base Model: answerdotai/ModernBERT-base
Language: English
Architecture: Dynamic enterprise action scorer
Status: Research Prototype / Pre-Production Candidate

Enterprise Reflex V1 is a lightweight enterprise decision model designed to operate as a System-1 action-ranking layer in front of larger reasoning models, agent frameworks, and enterprise automation systems.

Given a request, structured enterprise state, optional context, and a dynamic runtime set of candidate actions, the model ranks the available actions, can abstain with NO_ACTION , and uses calibrated confidence to decide whether the request should stay in the fast System-1 path or be escalated to a more capable System-2 model or a human.

Enterprise Reflex is not a generative LLM and not a fixed intent classifier . Candidate actions are supplied dynamically at inference time, allowing the model to score actions it was not explicitly trained to recognize by identifier alone.

---

Why Enterprise Reflex

Enterprise agent systems increasingly expose large action spaces: create or update records, approve or reject workflow steps, search internal systems, trigger notifications, route work, invoke APIs, execute business operations, or abstain when no safe action is available.

Sending every request directly to a large reasoning model can be unnecessarily expensive and slow. Enterprise Reflex is designed to act as a compact decision layer:

Request + Enterprise State + Context + Candidate Actions
                         |
                         v
                 Enterprise Reflex
                         |
             +-----------+-----------+
             |                       |
             v                       v
     Confident System-1      Low confidence / NO_ACTION
          decision                    |
             |                        v
             v                System-2 / Human Review
          Execute

The objective is not to replace larger reasoning models. It is to reduce how often they are required.

---

What Changed in V1

V1 extends the original Enterprise Reflex prototype with targeted hard-training data focused on failure modes observed in V0.

The V1 hard-training set emphasizes:

• same-domain sibling-action discrimination,
• counterfactual state changes,
• policy and workflow constraints,
• semantic cross-domain collisions,
• improved NO_ACTION boundaries,
• unseen and renamed action identifiers.

The training strategy retained broad V0 enterprise coverage while adding targeted V1 hard examples so the model could improve difficult routing behavior without losing general enterprise performance.

Training Configuration

---

Input Representation

Enterprise Reflex scores each candidate action against a compact serialized request representation.

Request Side

{
  "request": "Release the approved supplier payment",
  "domain": "Finance",
  "state": {
    "payment_approved": true,
    "invoice_matched": true
  },
  "context": {}
}

Candidate Action Side

{
  "name": "finance.release_payment",
  "description": "Release an approved and validated supplier payment.",
  "family": "EXECUTE",
  "domain": "Finance"
}

Each candidate is scored independently. Compatibility margins are then calibrated and normalized across the runtime candidate set. NO_ACTION is added as an explicit abstention candidate.

---

Evaluation

V1 was evaluated at three levels:

1. pairwise request-action classification,
2. grouped candidate ranking,
3. a manually designed 100-case hard stress test.

These evaluations represent different levels of difficulty and should be interpreted separately.

---

Pairwise Test Results

V0 Test Distribution

V1 Hard Test Distribution

Combined Test Distribution

---

Grouped Action-Ranking Results

Grouped evaluation measures whether the correct action is ranked highest within the full runtime candidate set.

V0 Grouped Test

V1 Hard Grouped Test

Combined Grouped Test

The V1 hard grouped split is intentionally more difficult than the broad V0 evaluation and should not be treated as the same distribution.

---

Selective System-1 / System-2 Routing

V1 uses calibrated confidence to determine whether a decision should remain in System-1 or be escalated.

For the reported run:

• Selected System-2 threshold: 0.87
• Validation System-1 coverage: 73.71%
• Validation System-1 accuracy: 99.02%

NO_ACTION is always treated as a System-2 route.

This threshold is calibrated on the validation distribution and should be recalibrated for any materially different deployment domain.

---

100-Case Manual Hard Stress Test

A separate manual suite of 100 hard enterprise cases was used to stress behavior outside the easier validation distribution.

Overall Results

Performance by Category

The manual stress test is intentionally adversarial and significantly harder than the standard grouped benchmark.

---

What V1 Improved

Compared with V0 hard-test behavior, V1 improved both hard-decision accuracy and autonomous coverage.

Observed improvements include:

• hard Top-1 accuracy increased from approximately 80% to 85% ,
• System-1 coverage increased from approximately 53% to 66% ,
• state-sensitive decisions improved substantially,
• unseen action-name generalization remained strong,
• NO_ACTION behavior improved on the manual hard suite,
• broad V0 enterprise ranking performance remained largely intact.

The result supports the core Enterprise Reflex design: a lightweight model can perform useful dynamic enterprise action ranking while routing uncertain cases to a larger reasoner.

---

Current Limitation: Policy-Constrained Execution

The dominant V1 weakness is policy-sensitive action validity.

Several hard cases were semantically understood but executed incorrectly because state or policy should have blocked the action.

Observed failure patterns include:

• releasing a payment without required approval,
• provisioning privileged access without security approval,
• deleting logs under legal or retention hold,
• cancelling an order after a workflow state that prohibits cancellation,
• granting physical access before mandatory induction is complete.

This indicates that V1 is currently stronger at answering:

Which action best matches this request?

than:

Is this action actually permitted under the current enterprise state and policy?

For this reason, V1 should not be used as the sole authority for autonomous high-impact enterprise execution.

---

Intended Use

Enterprise Reflex V1 is suitable for research and controlled enterprise-agent experiments such as:

• action ranking,
• dynamic tool selection,
• top-k tool narrowing,
• System-1 / System-2 routing,
• agent handoff,
• workflow recommendation,
• shadow-mode decision analysis,
• human-in-the-loop action suggestions,
• enterprise action-space reduction before LLM reasoning.

---

Not Recommended For

V1 is not recommended as the sole decision layer for:

• autonomous financial transactions,
• privileged-access provisioning,
• destructive security actions,
• compliance-sensitive deletion,
• irreversible workflow actions,
• legal or regulatory decisions,
• production execution without deterministic authorization and policy enforcement.

---

Recommended Production Architecture

Enterprise Reflex should be combined with deterministic controls.

Request + State + Candidate Actions
                |
                v
       Enterprise Reflex
                |
                v
        Ranked Action
                |
                v
 Policy / Authorization Engine
         /              \
      Allowed          Blocked
        |                 |
        v                 v
     Execute        System-2 / Human

The learned model provides decision intelligence. Authorization, policy, entitlement, retention, approval, and other hard enterprise controls should remain deterministic whenever possible.

---

Example Usage

import json
import torch
import numpy as np

from transformers import (
    AutoTokenizer,
    AutoModelForSequenceClassification,
)

MODEL_ID = "yasserrmd/enterprise-reflex-v1"
SYSTEM2_THRESHOLD = 0.87
MAX_LENGTH = 384

# Load the runtime calibration values saved with the model if available.
# The reported experiment used a calibrated temperature determined from
# the combined validation distribution.
TEMPERATURE = 1.0

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForSequenceClassification.from_pretrained(MODEL_ID)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
model.eval()


def softmax_np(x):
    x = np.asarray(x, dtype=np.float64)
    x = x - np.max(x)
    e = np.exp(x)
    return e / e.sum()


def request_text(request, domain="enterprise", state=None, context=None):
    return json.dumps(
        {
            "request": request,
            "domain": domain,
            "state": state or {},
            "context": context or {},
        },
        ensure_ascii=False,
        sort_keys=True,
    )


def action_text(action, default_domain="enterprise"):
    return json.dumps(
        {
            "name": action.get("name", ""),
            "description": action.get("description", ""),
            "family": action.get("family", "OTHER"),
            "domain": action.get("domain", default_domain),
        },
        ensure_ascii=False,
        sort_keys=True,
    )


def rank_actions(request, candidate_actions, state=None, context=None, domain="enterprise"):
    actions = list(candidate_actions)

    actions.append(
        {
            "name": "NO_ACTION",
            "description": "None of the available actions safely or correctly satisfy the request.",
            "family": "ABSTAIN",
            "domain": domain,
        }
    )

    left = request_text(
        request,
        domain=domain,
        state=state,
        context=context,
    )

    enc = tokenizer(
        [left] * len(actions),
        [action_text(a, default_domain=domain) for a in actions],
        padding=True,
        truncation=True,
        max_length=MAX_LENGTH,
        return_tensors="pt",
    ).to(device)

    with torch.no_grad():
        logits = model(**enc).logits

    margins = (logits[:, 1] - logits[:, 0]).float().cpu().numpy()
    probabilities = softmax_np(margins / TEMPERATURE)
    order = np.argsort(-probabilities)

    ranked = [
        {
            "action": actions[int(i)]["name"],
            "probability": float(probabilities[int(i)]),
        }
        for i in order
    ]

    top = ranked[0]

    system2_required = (
        top["action"] == "NO_ACTION"
        or top["probability"] < SYSTEM2_THRESHOLD
    )

    return {
        "decision": top["action"],
        "confidence": top["probability"],
        "system2_required": system2_required,
        "ranked_actions": ranked,
    }

Example

result = rank_actions(
    request="Release the supplier payment",
    domain="Finance",
    state={
        "invoice_matched": True,
        "payment_approved": True,
    },
    candidate_actions=[
        {
            "name": "finance.release_payment",
            "description": "Release an approved and validated supplier payment.",
            "family": "EXECUTE",
            "domain": "Finance",
        },
        {
            "name": "finance.create_invoice",
            "description": "Create an invoice record.",
            "family": "CREATE",
            "domain": "Finance",
        },
    ],
)

print(result)

---

Calibration Note

The reported confidence threshold was selected for the reported V1 experiment.

For a new deployment:

1. collect domain-specific validation data,
2. calibrate temperature,
3. determine an acceptable System-1 error rate,
4. select the confidence threshold for that environment,
5. validate policy-sensitive and destructive actions separately.

Do not assume that 0.87 is appropriate for every enterprise domain.

---

Research Status

Enterprise Reflex V1 should currently be considered a:

Research Prototype / Pre-Production Candidate

The model has demonstrated:

• strong broad enterprise action ranking,
• useful dynamic action selection,
• effective abstention,
• high Top-3 retrieval quality,
• improved state-sensitive behavior,
• improved System-1 coverage,
• promising generalization to unseen action identifiers.

It has not yet demonstrated sufficient reliability for unrestricted autonomous enterprise execution.

The primary V2 research target is policy-aware action validity and confident wrong-action suppression .

---

V2 Direction

The next iteration should focus less on generic enterprise volume and more on targeted safety and state-validity examples:

• policy counterfactuals,
• approval-sensitive actions,
• retention and legal-hold constraints,
• authorization and entitlement state,
• workflow-state legality,
• semantic domain collisions,
• hard negatives mined from confident V1 failures,
• valid-action vs NO_ACTION boundary cases.

A likely architectural extension is to separate:

1. semantic action suitability,
2. state/policy validity,

before producing the final action confidence.

---

Datasets

Enterprise Reflex Dataset

yasserrmd/enterprise-reflex-dataset

Broad enterprise action-ranking data used for V0 and retained in V1 training.

Enterprise Reflex V1 Hard Dataset

yasserrmd/enterprise-reflex-v1-hard-dataset

Targeted hard-training examples covering sibling actions, counterfactual state, policy constraints, cross-domain collisions, NO_ACTION , and unseen action names.

---

Limitations

• English only in V1.
• Text and structured state only.
• No multimodal input.
• No deterministic policy engine is embedded in the model.
• Confidence calibration is distribution-dependent.
• The hard-test suite is manually constructed and relatively small.
• Policy-sensitive action validity remains the main weakness.
• The model may still produce high-confidence incorrect actions.
• Reported metrics should not be interpreted as production-safety guarantees.

---

Responsible Use

Enterprise Reflex is intended to assist enterprise decision routing, not replace enterprise authorization, policy, compliance, or human accountability.

High-impact actions should remain subject to deterministic controls and appropriate human or System-2 review.

---

Author

Mohamed Yasser
Hugging Face: yasserrmd
GitHub: yasserrmd

---

Version

Enterprise Reflex V1
Research Prototype / Pre-Production Candidate