Benchmarking System — Technical Reference

This page covers the internal architecture, data model, API surface, and development configuration of the benchmarking system. For a usage-focused guide, see the Benchmarking Guide.

Table of Contents

• Architecture Overview
• Data Model
  • Dataset & DatasetVersion
  • Split
  • BenchmarkProject
  • BenchmarkDefinition
  • BenchmarkRun
  • BenchmarkAuditLog
  • DatasetGroundTruthJob
  • Enums
• API Endpoints
  • Dataset Management
  • HITL Dataset Creation
  • Ground Truth Generation
  • Projects
  • Definitions
  • Runs
• Temporal Workflow Engine
  • Orchestrator Workflow
  • Activities
  • Configuration & Timeouts
• Evaluator System
  • Evaluator Interface
  • Schema-Aware Evaluator
  • Black-Box Evaluator
  • Evaluator Registry
• Metrics Pipeline
  • Per-Sample Metrics
  • Aggregation
  • Storage Format
  • Sliced Metrics
  • Failure Analysis
• Baseline Comparison
• Object Storage
• Environment Variables
• Key Files Reference

Architecture Overview

The benchmarking system is built on three infrastructure components that integrate with the existing NestJS backend and Temporal workflow engine:

Data Model

All benchmark entities are defined in the Prisma schema at apps/shared/prisma/schema.prisma.

Dataset & DatasetVersion

Split

BenchmarkProject

BenchmarkDefinition

BenchmarkRun

BenchmarkAuditLog

DatasetGroundTruthJob

Tracks the lifecycle of ground truth generation for individual samples. Each job links a dataset sample to a Document record that is processed through an OCR workflow and then reviewed via a dataset-scoped HITL queue.

Indices: datasetVersionId, documentId, status. The unique constraint on documentId is used by the production HITL queue to exclude ground truth documents via groundTruthJob: { is: null }.

Enums

API Endpoints

Dataset Management

Controller: apps/backend-services/src/benchmark/dataset.controller.ts

HITL Dataset Creation

Controller: apps/backend-services/src/benchmark/hitl-dataset.controller.ts

These endpoints allow creating datasets from documents that have been verified through the Human-In-The-Loop review interface. The corrected OCR data becomes ground truth in the standard flat key-value format.

Eligible Documents Query Parameters

Ground Truth Construction

For each selected document, the service takes the most recent approved review session and applies its field corrections to the OCR results. Corrections are applied as follows: confirmed fields keep their original value, corrected fields use the reviewer's value, deleted fields are removed, and flagged fields are kept as-is. The output is a flat key-value JSON file (e.g., {"vendor_name": "Acme Corp", "total_amount": 1250.75}) using the same field value extraction logic as the benchmark workflow (extractFieldValue). Provenance metadata (source document ID, review session, reviewer) is stored in the manifest sample's metadata field.

Ground Truth Generation

Controller: apps/backend-services/src/benchmark/ground-truth-generation.controller.ts

These endpoints manage the ground truth generation workflow — running samples without ground truth through an OCR workflow and providing a dataset-scoped HITL review queue.

Jobs Query Parameters

Review Queue Query Parameters

Queue Separation

Production and dataset HITL queues are separated at the database query level. The production queue (GET /api/hitl/queue) filters documents with groundTruthJob: { is: null }, excluding any document linked to a ground truth generation job. The dataset queue queries DatasetGroundTruthJob records directly. Both queues use the same ReviewSession and FieldCorrection models for the actual review process.

Post-Approval Hook

When a review session is approved via POST /api/hitl/sessions/:id/submit, the HITL service checks if the document is linked to a DatasetGroundTruthJob. If so, it triggers ground truth extraction: the OCR results are combined with the reviewer's corrections using buildGroundTruth(), the resulting JSON is written to dataset storage, and the manifest is updated. This hook is non-blocking — if it fails, the session approval still succeeds.

Projects

Controller: apps/backend-services/src/benchmark/benchmark-project.controller.ts

Definitions

Controller: apps/backend-services/src/benchmark/benchmark-definition.controller.ts

Runs

Controller: apps/backend-services/src/benchmark/benchmark-run.controller.ts

Per-Sample Results Query Parameters

Temporal Workflow Engine

Orchestrator Workflow

The benchmarkRunWorkflow function in apps/temporal/src/benchmark-workflow.ts orchestrates the full run lifecycle:

1. Materialize dataset — downloads files from object storage to local cache.
2. Load manifest — parses dataset-manifest.json and filters by split.
3. Execute & evaluate (fan-out) — processes samples in batches:
   • Executes the graph workflow as a child Temporal workflow per sample.
   • Extracts predictions from the workflow context.
   • Writes predictions to disk and runs the evaluator.
4. Aggregate — computes run-level statistics from all evaluation results.
5. Update run status — writes metrics and status to the database.
6. Compare against baseline — if a baseline exists, runs threshold checks.
7. Cleanup — removes temporary outputs (cached datasets are preserved).

The workflow supports:

• Progress queries via getProgress — returns current phase, sample counts, and percent complete.
• Cancellation via the cancel signal — gracefully stops processing and marks the run as cancelled.

Progress Query Response

Activities

The child workflow benchmarkExecuteWorkflow (in activities/benchmark-execute.ts) wraps the existing graphWorkflow and executes it for a single sample. It extracts predictions from the workflow context using a field extraction function that handles Azure Document Intelligence field types (typed values like valueNumber, valueCurrency, valueDate, etc.).

Configuration & Timeouts

All defaults can be overridden per definition via runtimeSettings.

Evaluator System

Evaluator Interface

Defined in apps/temporal/src/benchmark-types.ts:

interface BenchmarkEvaluator {
  type: string;
  evaluate(input: EvaluationInput): Promise<EvaluationResult>;
}

interface EvaluationInput {
  sampleId: string;
  inputPaths: string[];
  predictionPaths: string[];
  groundTruthPaths: string[];
  metadata: Record<string, unknown>;
  evaluatorConfig: Record<string, unknown>;
}

interface EvaluationResult {
  sampleId: string;
  metrics: Record<string, number>;
  diagnostics: Record<string, unknown>;
  pass: boolean;
  artifacts?: EvaluationArtifact[];
  groundTruth?: unknown;
  prediction?: unknown;
  evaluationDetails?: unknown;
}

Schema-Aware Evaluator

File: apps/temporal/src/evaluators/schema-aware-evaluator.ts

Performs field-by-field comparison between predicted and ground truth JSON objects. For each field, applies a configurable matching rule:

• exact — string equality (default).
• fuzzy — Levenshtein similarity with configurable threshold.
• numeric — numeric comparison with absolute or relative tolerance.
• date — date normalization to YYYY-MM-DD before comparison.
• boolean — boolean-like value parsing (true/yes/1).

Produces metrics: precision, recall, f1, truePositives, falsePositives, falseNegatives, totalGroundTruthFields, matchedFields, checkboxAccuracy.

Pass condition: f1 ≥ passThreshold (default: 1.0).

Black-Box Evaluator

File: apps/temporal/src/evaluators/black-box-evaluator.ts

Treats outputs as opaque. Supports two modes:

• JSON mode — deep equality check with field-level diff generation. Produces exact_match, field_overlap, and diff_count. Creates a JSON diff artifact file.
• Raw mode — byte-level string comparison. Produces exact_match and byte length metrics.

Pass condition: exact match only.

Evaluator Registry

File: apps/temporal/src/evaluator-registry.ts

A simple Map-based registry that stores evaluator instances keyed by their type string. Both built-in evaluators are registered at module load time. Custom evaluators can be added via the registerEvaluator() function.

Metrics Pipeline

Per-Sample Metrics

Each evaluator returns a Record<string, number> of named metrics for each sample. The specific metrics depend on the evaluator (see Per-Sample Metrics in the user guide).

Aggregation

File: apps/temporal/src/benchmark-aggregation.ts

The aggregateResults() function collects all metric names across samples, then computes the following statistics for each metric:

interface MetricStatistics {
  name: string;
  mean: number;     // Arithmetic mean
  median: number;   // 50th percentile
  stdDev: number;   // Population standard deviation
  p5: number;       // 5th percentile
  p25: number;      // 25th percentile (Q1)
  p75: number;      // 75th percentile (Q3)
  p95: number;      // 95th percentile
  min: number;      // Minimum value
  max: number;      // Maximum value
}

Percentiles are calculated using linear interpolation: for percentile P across N sorted values, the index is P/100 × (N-1). If the index falls between two values, a weighted average is used.

The aggregation also computes overall counts: totalSamples, passingsSamples, failingSamples, and passRate.

Storage Format

The BenchmarkRun.metrics JSON field stores a combined structure:

{
  // Flat metrics (for baseline comparison and quick access)
  "total_samples": 100,
  "passing_samples": 95,
  "failing_samples": 5,
  "pass_rate": 0.95,
  "f1.mean": 0.92,
  "f1.median": 0.93,
  "f1.stdDev": 0.05,
  "f1.p5": 0.81,
  "f1.p25": 0.88,
  "f1.p75": 0.96,
  "f1.p95": 0.99,
  "f1.min": 0.65,
  "f1.max": 0.99,

  // Structured aggregate (for drill-down)
  "_aggregate": {
    "overall": { ... },
    "sliced": [ ... ],
    "failureAnalysis": { ... }
  },

  // Per-sample results (for sample browsing)
  "perSampleResults": [
    {
      "sampleId": "sample-1",
      "metrics": { "f1": 0.92, "precision": 0.95, "recall": 0.89 },
      "diagnostics": { ... },
      "pass": true,
      "groundTruth": { ... },
      "prediction": { ... }
    }
  ]
}

The flat keys (like f1.mean) are produced by the flattenMetrics() helper in the workflow, which converts AggregatedMetrics into a flat Record<string, number>. These are the keys used by baseline comparison.

Sliced Metrics

When sliceDimensions is configured, the aggregation groups samples by metadata dimension values and computes separate AggregatedMetrics for each group:

interface SlicedMetrics {
  dimension: string;                          // e.g. "docType"
  slices: Record<string, AggregatedMetrics>;  // e.g. { "invoice": {...}, "receipt": {...} }
}

Metadata is read from the sample's diagnostics.metadata field. Samples with missing dimension values are grouped under "unknown".

Failure Analysis

The performFailureAnalysis() function produces two outputs:

• Worst samples — the N samples with the lowest value for a given metric (default: bottom 10 by f1), sorted ascending. Each entry includes the sample ID, metric value, all metrics, and diagnostics.
• Per-field error breakdown — (schema-aware evaluator only) aggregates comparison results across all samples to compute, for each field: total occurrences, match count, missing count, mismatch count, and error rate. Sorted by error rate descending.

Baseline Comparison

File: apps/temporal/src/activities/benchmark-baseline-comparison.ts

The comparison activity runs after every completed run. It:

1. Looks up the baseline run for the same definition (the run with isBaseline: true).
2. Iterates over all numeric metrics in both runs.
3. For each metric, computes the absolute delta and percentage change.
4. Checks the metric against any matching threshold:
   • Absolute: currentValue ≥ threshold.value
   • Relative: currentValue ≥ baselineValue × threshold.value
5. Stores the BaselineComparison result on the run record.
6. Tags the run with regression: "true" if any metric failed its threshold.

interface BaselineComparison {
  baselineRunId: string;
  overallPassed: boolean;
  metricComparisons: MetricComparison[];
  regressedMetrics: string[];
}

interface MetricComparison {
  metricName: string;
  currentValue: number;
  baselineValue: number;
  delta: number;
  deltaPercent: number;
  passed: boolean;
  threshold?: MetricThreshold;
}

interface MetricThreshold {
  metricName: string;
  type: "absolute" | "relative";
  value: number;
}

Threshold Types

Object Storage

The benchmarking system accesses object storage through a BlobStorageInterface abstraction, supporting both MinIO (S3-compatible, default for local development) and Azure Blob Storage (for cloud deployments). The active provider is selected via the BLOB_STORAGE_PROVIDER environment variable ("minio" or "azure").

Datasets are stored with the following structure:

datasets/{datasetId}/{version}/
  ├── dataset-manifest.json
  ├── inputs/
  │   ├── sample-1.pdf
  │   └── sample-2.pdf
  └── ground_truth/
      ├── sample-1.json
      └── sample-2.json

Dataset Materialization

The materialization activity downloads all files under the storagePrefix to a local cache directory:

• Cache base: BENCHMARK_CACHE_DIR environment variable (default: /tmp/benchmark-cache).
• Cache key: {datasetId}-{datasetVersionId}.
• On cache hit (manifest file exists locally), the download is skipped.
• On failure, the partial cache directory is removed.

Environment Variables

Key Files Reference

Backend Services

Temporal Workflows & Activities

Data Model