Benchmarking System Guide

The benchmarking system evaluates your document processing workflows by running them against curated datasets with ground truth, then measuring accuracy, detecting regressions, and comparing results across workflow versions.

Table of Contents

• Overview
• Core Concepts
  • Projects and Definitions
  • Datasets and Versions
  • Splits
  • Evaluators
• Understanding Per-Sample Metrics
  • Schema-Aware Evaluator Metrics
  • Black-Box Evaluator Metrics
  • Pass / Fail Determination
• Understanding Aggregate (Run-Level) Metrics
  • Summary Counts
  • Statistical Measures
  • Reading Aggregate Metrics
• Setting Up a Benchmark
  • Creating Datasets from Verified Documents
  • Generating Ground Truth via HITL Review
• Running Benchmarks
• Baseline Management & Regression Detection
• Drill-Down & Failure Analysis
• Comparing Runs
• Sliced Metrics
• Deleting Resources
• Scheduled Benchmarks

Overview

The benchmarking system answers one question: "How well does my workflow perform, and has it gotten worse?" It is workflow-agnostic — it can evaluate any graph workflow configuration. The system wraps your existing workflow execution with dataset management, automated evaluation, metric tracking, and regression detection.

Core Concepts

Projects and Definitions

The hierarchy is: Project → Definition → Run.

Datasets and Versions

A Dataset is a named collection of documents. Each dataset has one or more Dataset Versions. A version starts as mutable — you can upload files and delete samples freely. When a benchmark run is started against a version, it is automatically frozen, preventing any further modifications. This ensures that benchmark results are reproducible. A version consists of:

• A dataset manifest (dataset-manifest.json) listing every sample, its input files, ground truth files, and metadata.
• Input files — the documents to process (e.g. PDFs, images).
• Ground truth files — JSON files containing the expected output for each sample.
• An optional ground truth schema used for validation.

All files are stored in object storage (MinIO or Azure Blob Storage) under a per-version storage prefix.

{
  "schemaVersion": "1.0",
  "samples": [
    {
      "id": "sample-1",
      "inputs": [
        { "path": "inputs/invoice-1.pdf", "mimeType": "application/pdf" }
      ],
      "groundTruth": [
        { "path": "ground_truth/invoice-1.json", "format": "json" }
      ],
      "metadata": {
        "docType": "invoice",
        "language": "en",
        "source": "vendor-A"
      }
    }
  ],
  "splits": {
    "train": ["sample-1", "sample-2"],
    "test": ["sample-10", "sample-11"]
  }
}

Splits

A Split is a named subset of samples within a dataset version. Splits partition samples for different purposes:

When a benchmark definition references a split, only the samples in that split are processed during a run. If no split is specified, all samples in the dataset version are used.

Evaluators

An Evaluator is a pluggable component that compares workflow predictions against ground truth and produces per-sample metrics. Two evaluators are built in:

Understanding Per-Sample Metrics

Each sample in a benchmark run is evaluated individually. The evaluator compares the workflow's prediction against the ground truth and produces a set of numeric metrics. Which metrics you get depends on the evaluator type.

Schema-Aware Evaluator Metrics

The schema-aware evaluator performs field-level comparison between the predicted JSON and the ground truth JSON. It categorises each field into one of three outcomes:

From these counts the evaluator calculates:

In addition to these primary metrics, the evaluator also emits the underlying count metrics that feed the formulas above:

Matching Rules

The schema-aware evaluator supports configurable matching rules per field. The default rule is exact match (string equality). You can override this in the evaluator configuration:

Evaluator Config Reference

When creating a benchmark definition, you set evaluatorType and evaluatorConfig as separate fields. The evaluatorConfig JSON is passed directly to the evaluator. Here is the complete set of available options for the schema-aware evaluator:

Each FieldMatchingRule object supports:

{
  "passThreshold": 0.9,
  "defaultRule": { "rule": "exact" },
  "fieldRules": {
    "total_amount": {
      "rule": "numeric",
      "numericAbsoluteTolerance": 0.01
    },
    "vendor_name": {
      "rule": "fuzzy",
      "fuzzyThreshold": 0.85
    },
    "invoice_date": {
      "rule": "date",
      "dateFormats": ["YYYY-MM-DD", "MM/DD/YYYY"]
    },
    "is_taxable": {
      "rule": "boolean"
    }
  }
}

{}

Black-Box Evaluator Metrics

The black-box evaluator treats outputs as opaque. It operates in two modes depending on the content type:

JSON Mode

Used when both prediction and ground truth are valid JSON objects. Produces:

JSON mode also produces a diff artifact — a JSON file listing every difference path, type (added/deleted/changed), and the expected vs. actual values.

Raw Mode

Used when content is not JSON (e.g. plain text, raw bytes). Produces:

Pass / Fail Determination

Each sample is marked pass or fail based on evaluator-specific rules:

• Schema-aware: A sample passes when its F1 score ≥ passThreshold (default: 1.0, i.e., every field must match).
• Black-box: A sample passes only on exact match.

Understanding Aggregate (Run-Level) Metrics

After all samples in a run are evaluated, the system aggregates the individual results into run-level statistics. These give you a bird's-eye view of how the workflow performed across the entire dataset.

Summary Counts

Statistical Measures

For each per-sample metric (e.g. f1, precision, recall, exact_match), the system computes the following statistics across all samples:

Reading Aggregate Metrics

Here is a practical guide to interpreting a run's aggregate metrics:

Setting Up a Benchmark

The end-to-end process to benchmark a workflow:

Step 1: Create a Dataset

1. Navigate to Benchmarking > Datasets and click "Create Dataset". Enter a name, optional description, and optional metadata key-value pairs.
2. On the Dataset Detail page, click "New Version" to create a version (optionally give it a name such as "Q4 invoices").
3. A file upload dialog opens automatically — drag and drop or select input documents and their corresponding ground truth files.
4. Review uploaded samples in the Sample Preview tab.
5. Optionally validate the dataset using the version's three-dot menu > "Validate", which opens a validation report.
6. Optionally freeze the version via three-dot menu > "Freeze Version". Note: versions are also automatically frozen when a benchmark run starts against them.

Alternative: Create a Dataset from Verified Documents

If your documents have already been processed through the OCR pipeline and verified via the Human-In-The-Loop (HITL) review interface, you can create a benchmark dataset directly from them — no separate ground truth files required. The corrected OCR data from approved review sessions becomes the ground truth.

1. Click "From Verified Documents" on the Dataset List page (to create a new dataset) or on a Dataset Detail page (to add a version to an existing dataset).
2. Enter a dataset name and description (when creating a new dataset).
3. Select eligible documents from the table. You can search by filename and paginate through the list. The table shows each document's filename, file type, approval date, reviewer, field count, and correction count.
4. Confirm and submit. The system will:
   • Copy the original document files into the dataset's inputs/ directory.
   • Build ground truth by applying the reviewer's corrections to the OCR results, producing flat key-value JSON files in the ground-truth/ directory.
   • Generate a dataset-manifest.json with provenance metadata (source document ID, review session, reviewer) for each sample.

The resulting dataset is identical in format to one created via file upload and can be used with any evaluator.

Alternative: Generate Ground Truth via HITL Review

If you have documents but no ground truth, you can generate it by running the documents through an OCR workflow and reviewing the results in a dedicated, dataset-scoped HITL queue. This is the third pathway for creating dataset ground truth.

1. Upload documents without ground truth files to a dataset version (input files only).
2. Navigate to the dataset detail page and select the version. Click the "Ground Truth" tab.
3. Select a workflow template from the dropdown (e.g., your standard OCR workflow) and click "Start Generation".
4. The system creates a ground truth generation job for each sample that lacks ground truth. Each job:
   • Creates a Document record and copies the input file to document storage.
   • Starts the selected OCR workflow with the model ID from the workflow configuration.
   • The workflow runs to completion (the confidence gate is bypassed so the workflow does not pause for human review during processing).
5. Monitor progress on the Ground Truth tab — a progress bar and job table show the status of each sample (Pending → Processing → Awaiting Review).
6. Once jobs reach "Awaiting Review", click "Open Review Queue" to open the dataset-scoped HITL review queue.
7. Review each document using the same HITL review interface used in production — view OCR results, make corrections, and approve.
8. On approval, the system automatically extracts ground truth from the OCR results plus your corrections and writes it to the dataset storage. The dataset manifest is updated with the new ground truth entry.

Step 2: Create a Project and Definition

1. Navigate to Benchmarking > Projects and click "Create Project". Enter a name and optional description.
2. Open the project and click "Create Definition". Fill in the dialog:
   • Name — a descriptive name for this test specification.
   • Dataset Version — select from available versions (grouped by dataset, showing document count).
   • Split (optional) — choose a split to evaluate a subset of the dataset, or leave as "All samples".
   • Workflow — select the workflow to benchmark.
   • Evaluator Type — choose Schema-Aware or Black-Box.
   • Evaluator Config (optional) — JSON configuration for the evaluator (see Evaluator Config Reference).
   • Max Parallel Documents — number of samples processed concurrently (default: 10).
   • Per Document Timeout (ms) — timeout for each sample (default: 300,000 ms / 5 minutes).
3. Click "Create".

Step 3: Run the Benchmark

1. On the Project Detail page, click a definition row to open the Definition Details dialog.
2. Click "Start Run". The system creates a new run and navigates to the Run Detail page.
3. Watch real-time progress as the run moves through its phases (see Running Benchmarks below). You can cancel at any time using the "Cancel" button.

Running Benchmarks

When a benchmark run is started, the system automatically progresses through six phases:

1. Dataset Preparation — downloads the dataset version to the worker (cached for subsequent runs).
2. Execution — runs the configured workflow against each sample, processing them in configurable parallel batches.
3. Evaluation — compares each sample's predicted output against the ground truth using the configured evaluator.
4. Aggregation — combines all per-sample results into run-level statistics (see Understanding Aggregate Metrics).
5. Baseline Comparison — if a baseline exists, checks the current run's metrics against the baseline thresholds (see Baseline Management).
6. Cleanup — removes temporary output files while preserving cached datasets for reuse.

You can monitor progress in real time on the Run Detail page, which shows the current phase, sample counts, and percent complete. If needed, click "Cancel" to stop a running benchmark.

Run Statuses

Baseline Management & Regression Detection

Promoting a Baseline

Any completed run can be promoted to baseline. Only one run per definition can be the baseline — promoting a new one automatically demotes the previous one.

1. Open a completed run on the Run Detail page.
2. Click "Promote to Baseline" in the top-right actions.
3. In the Configure Baseline Thresholds dialog, set a threshold for each metric:
   • Choose Relative (%) or Absolute from the dropdown.
   • Enter the threshold value.
4. Click "Promote to Baseline" to confirm.

After promotion, the run shows a Baseline badge and its thresholds appear in the Definition Details dialog. You can update thresholds later using the "Edit Thresholds" button on the baseline run.

Threshold Types

Automatic Comparison

Every completed run is automatically compared against the baseline for its definition (if one exists). The comparison produces:

• A per-metric delta showing the absolute and percentage change from baseline.
• A pass/fail determination for each threshold.
• A list of regressed metrics — metrics that failed their threshold check.
• An overallPassed flag — true only if no metrics regressed.

If any metric regresses, the run is tagged with regression: "true" for easy filtering.

Drill-Down & Failure Analysis

After a run completes, the Run Detail page provides several tools for inspecting results:

Run Detail Page

• Baseline Comparison banner — a green (passed) or orange (regression detected) alert at the top of the page. If no baseline is set, a blue prompt offers to promote the current run.
• Aggregated Metrics — an expandable accordion listing every metric name and its aggregated value.
• Per-Field Error Breakdown (schema-aware evaluator only) — a table showing each field's error count and error rate. Click a field row to open a drawer with the specific samples where that field had errors, including expected and predicted values.
• Artifacts — browse all run artifacts, filtered by type (per-document output, intermediate node output, diff report, evaluation report, error log). Click an artifact row to view its contents.

Per-Sample Drill-Down

Click "View All Samples" on the Run Detail page to open the full drill-down page with paginated, filterable per-sample results:

• Filter by metadata dimensions — the filter panel dynamically shows dropdowns for each available metadata dimension (e.g. document type, language). Apply one or more filters to narrow results.
• Results table — shows each sample's ID, pass/fail status, metadata values, and key metrics.
• Sample detail drawer — click a sample row to open a side panel showing the full metrics, metadata, ground truth (expected) vs prediction (actual) side by side, evaluation details, and diagnostics.

Sliced Metrics

Sliced metrics break down the aggregate statistics by metadata dimensions. For example, if your samples have docType metadata, you can see separate aggregate metrics for invoices, receipts, and contracts.

Slicing is configured via the sliceDimensions option in the aggregation. Each dimension produces a separate set of AggregatedMetrics per unique value.

This is useful for identifying if performance varies by document type, language, source, or any other metadata attribute.

Comparing Runs

You can compare metrics across multiple runs side by side:

1. On the Project Detail page, use the checkboxes in the Recent Runs table to select 2 to 5 completed runs.
2. Click the "Compare" button that appears above the table.
3. The Run Comparison page displays:
   • Run Information — status, definition, and start time for each run.
   • Metrics Comparison — a table with one column per run, plus delta and delta-percentage columns. Values are colour-coded: green for improvements, red for regressions.
   • Parameters Comparison — highlights parameters that differ across runs with a "Changed" badge.
   • Tags Comparison — same format as parameters.

You can export the comparison data as CSV or JSON using the buttons in the top-right corner.

Deleting Resources

The following resources can be deleted through the interface:

Scheduled Benchmarks

Benchmark definitions support scheduled execution. You can configure a cron schedule so that benchmark runs are triggered automatically — useful for continuous quality monitoring such as nightly regression checks against a golden dataset.

1. On the Project Detail page, click a definition row to open the Definition Details dialog.
2. Scroll to the Schedule Configuration card.
3. Toggle "Enable automatic scheduled runs" on.
4. Enter a cron expression (e.g. 0 2 * * * for daily at 2 AM).
5. Click "Save Schedule".

Once active, the schedule status panel displays the schedule ID, cron pattern, next run time, and last run time, along with an Active or Paused badge.

Scheduled runs use the same workflow as manual runs and produce identical metrics and baseline comparisons.