nemo-evaluator-plugin

Evaluator Plugin

The Plugin CLI entrypoint is uv run nemo evaluator.

Purpose

Use this skill to choose an evaluation interface and metric, validate a minimal example, submit a NeMo Platform evaluation job, and retrieve its results.

Inputs

Establish these inputs before building an evaluation:

• Evaluation interface: dataset-driven vs. task-driven agentic evaluation
• Execution interface: standalone SDK evaluation or a durable NeMo Platform job.
• Pass/fail dataset examples: the smallest representative pass and failure cases.
• Metrics: the behaviors to score and the template fields they consume.
• Target: no target for offline scoring, or the model, agent, runner, or precomputed trials that produce outputs.

Instructions

1. Clarify whether the input is dataset-driven rows or task-driven agent work.
2. Choose the simplest metric that measures the requested behavior. Prefer deterministic metrics when possible.
3. Build a tiny smoke case with one expected pass and one expected failure.
4. Validate metric behavior with the standalone SDK and inspect row-level output plus aggregates.
5. Fix field mappings, prompts, parsers, or task definitions before scaling.
6. Submit the platform job only after the input and scoring shape works.

Read Metric Selection before choosing a metric for a rubric, RAG workflow, or tool-calling evaluation.

Choose the execution interface

Default to submit for every plugin evaluation. The plugin's local execution path — client.evaluator.run() and the nemo evaluator ... run CLI verb — is being retired, so do not build on it even though --help still lists it. For fast metric iteration without the platform, use the standalone nemo_evaluator_sdk.Evaluator instead.

• Read SDK Execution for datasets, targets, configuration, field mapping, job lifecycle, and custom metric packaging.
• Read Stored Resources for persisted definitions and result queries.

Limitations

• api_key_secret is an environment-variable name standalone but a NeMo Platform secret name on submit. See API Auth.
• HTTP 409 from a submission often means a referenced platform secret is missing, not a duplicate job. Read the response body.
• intent is grader metadata and is never shown to the agent; only inputs reaches it.
• Metric templates use item.* for dataset rows but reference.*, sample.*, and inputs.* in agent evaluation.
• Metric progress can reach 100 percent before the platform job is terminal. Always call job.wait_until_done() before retrieving results or downloading artifacts.

CLI Interface

Prerequisites

All commands in this file assume that the shell's working directory is the root of the NVIDIA-NeMo/nemo-platform repository.

In a NeMo Platform repository checkout, run commands through the workspace:

# confirms plugin readiness and lists the registered evaluator jobs.
uv run nemo evaluator info
# lists available metric names; add a metric name to print its schema.
uv run nemo evaluator metric-types
# next two commands print the dataset-driven and task-driven job input and
# output schemas - can be very large, use with caution to avoid filling up the context window.
uv run nemo evaluator evaluate explain
uv run nemo evaluator agent-evaluate explain

When the skill and plugin are installed, use the installed nemo command without assuming a repository root or manually activating .venv.

Resolve bundled assets relative to this skill directory. In this repository the canonical path is skills/nemo-evaluator-plugin; an installed skill may live under a different skills root.

Bundled assets

Available Scripts

In this repository, NeMo uses the displayed workspace command:

uv run --frozen python skills/nemo-evaluator-plugin/scripts/generate_example_specs.py --check

Do not assume a client-specific run_script() helper; use the displayed uv run command.

Examples

Dataset-driven evaluation examples

• Follow Validate standalone, then submit to the platform. for the two-row pass/fail smoke test and its CLI submission.
• Follow Map noncanonical fields when dataset columns need field_mapping.
• Follow Getting job results for submission, terminal waiting, result retrieval, and artifact download.
• Follow Store a metric, task, and taskset for reusable definitions, and Query persisted results for result lookup.

Task-driven agent evaluation examples

Standalone SDK evaluation

Use AgentEvaluator().run(...) for standalone task-driven SDK evaluation. Its target can be a Model, a GenericAgent, or a direct AgentTaskRunner.

Platform job evaluation

Use the plugin agent-evaluate submit job for platform task evaluation. Its target is a ModelTarget, AgentTarget, CodexRunnerTarget, FabricRunnerTarget, or HarborRunnerTarget; alternatively provide precomputed trials. Provide exactly one of target or trials.

Submission accepts inline tasks or a stored TasksetRef. Stored tasksets are resolved in the target workspace.

Read Agent Evaluation for inline tasks, TasksetRef, concurrency, fail-fast behavior, result artifacts, and runner configuration.

Prepare Fabric in a repository checkout

Fabric runner examples and tests need the optional harness adapters and the matching Relay gateway:

uv sync --frozen --package nemo-evaluator-sdk --extra fabric --inexact
script/dev-install-fabric.sh

The install script downloads the checksum-verified nemo-relay binary that matches the locked Python bindings. Add its reported directory to PATH, then use uv run --frozen --no-sync ... for Fabric checks so uv does not remove the optional adapters.

Output Format

Report a completed platform evaluation in this form:

Job: <job-name>
Status: <terminal-status>
Metrics: <metric-names>
Mean: <aggregate-mean>
Artifacts: <downloaded result or artifact location>
Errors: <error messages>

Read specialized references

• Read Evaluator API Auth before using a model, agent, remote metric, or durable submission.
• Read LLM Judge before writing judge scores, prompts, or parsers.

Troubleshooting

Read Evaluator troubleshooting when schema, authentication, job, result, or runner behavior fails.

Follow security best practices

Never print, serialize, or commit secret values. Store only environment-variable names or platform secret references in specs and examples.