Multi-Agent Workflow Lab

An open-source testing and observability framework for multi-agent delegation, tool execution, MCP workflows, permissions, sandboxed actions, prompts, and runtime behavior.

Status: Experimental — v0.1.0 release candidate. Suitable for local development, evaluation, and policy testing; not production-hardened infrastructure.

Why this exists

Most model evaluation stops at input → model → output. Multi-agent systems add behavior that a final answer cannot explain:

• Which agent delegated a task, to whom, and why?
• What context, permissions, tools, MCP servers, and budget did the child receive?
• Did an agent attempt privilege escalation, repeat work, or enter a loop?
• Was the delegation efficient, and was the child result actually integrated?
• Can the run be inspected, compared, or replayed without repeating side effects?

MAWL makes those decisions explicit, policy-controlled, and traceable. Prompts are versioned assets; delegation is a first-class runtime event; deterministic rules evaluate behavior independently from optional model judges.

Key capabilities

Architecture

flowchart TD
  Input["User or test input"] --> Parser["Input parsers"]
  Parser --> Runtime["Agent runtime + task graph"]
  Runtime --> Delegation["Delegation engine"]
  Delegation --> Guards["Identity + permission + policy"]
  Guards --> Agents["Child agents"]
  Guards --> Tools["Tool executor"]
  Tools --> MCP["MCP connectors"]
  Tools --> Sandbox["Sandbox providers"]
  Agents --> Events["Runtime events"]
  MCP --> Events
  Sandbox --> Events
  Events --> Observe["Trace + metrics + monitors"]
  Events --> Evaluate["Rules + optional judge + replay"]

Quick Start

Requirements: Node.js 22 or newer and pnpm 11.

From a clone or an extracted release archive:

cd multi-agent-workflow-lab
pnpm install --frozen-lockfile
pnpm demo

The demo requires no API key or paid service. A public clone URL can be added after the maintainer chooses and publishes the repository location.

What the demo shows

pnpm demo runs a deterministic review chain through the CLI:

status: completed

orchestrator
  → researcher
    → analyst
      → reviewer
        → evaluator

delegation score: 88.75/100
workflow completed

The actual CLI output is JSON and includes the run ID, every task/agent/status, all eight evaluation dimensions, derived metrics, and the JSONL trace path.

Delegation observability

A trace records behavior rather than only the final answer:

orchestrator
  delegation.requested → researcher
  delegation.created   context + permissions + budget + depth

researcher
  tool.requested       → mcp.mock.search
  policy.allowed       explicit decision evidence
  tool.completed       size + redaction metadata

orchestrator
  delegation.result.accepted → child task integrated

Use these commands after a run:

pnpm mawl trace show <run-id>
pnpm mawl evaluate <run-id>
pnpm mawl graph <run-id>
pnpm mawl replay <run-id> dry-run

Evaluation scores are normalized diagnostics, not mathematical truth or a security guarantee.

Example workflows

Run any positive example with:

pnpm mawl run workflows/02-parallel-research.yaml

The negative examples intentionally return a non-zero exit code when the requested action is blocked.

Good vs. bad orchestrator

The lab compares a bounded, least-privilege orchestration trace with a synthetic bad trace containing duplicate work, leaked context, escalated permission, and an unauthorized tool request. RunComparator and DelegationEvaluator report changes in:

• agent selection and delegation edges;
• task decomposition and depth;
• context and permission minimization;
• tool use, retries, tokens, runtime, and failures;
• result integration and overall diagnostic score.

The executable coverage is in tests/lab.test.ts, with scenario inputs under scenarios/.

CLI

In this workspace, prefix CLI arguments with pnpm mawl:

pnpm mawl run <workflow.yaml> [json-input]
pnpm mawl agents list
pnpm mawl workflow inspect <run-id>
pnpm mawl task inspect <task-id>
pnpm mawl inspect agent permissions <agent-id>
pnpm mawl inspect mcp server [server-id]
pnpm mawl trace show <run-id>
pnpm mawl evaluate <run-id>
pnpm mawl graph <run-id> [output.mmd]
pnpm mawl replay <run-id> [exact|model-rerun|tool-rerun|dry-run]
pnpm mawl compare <run-a> <run-b>
pnpm mawl test <workflow-test.yaml>
pnpm mawl prompts list
pnpm mawl prompts inspect <prompt-id> [version]
pnpm mawl permissions inspect <agent-id>
pnpm mawl doctor

tool-rerun blocks tool names or manifests classified as external side effects unless the embedding application supplies explicit permission for that exact tool. The CLI supplies no side-effect permissions.

Prompts are first-class assets

This is not a directory of free-form prompt snippets. YAML assets under prompts/ include:

• stable ID and semantic version;
• owner, type, purpose, and known risks;
• declared input schema and expected output;
• allowed actions and recommended capabilities;
• content hash verified by PromptRegistry;
• provenance and trust classification when assembled at runtime.

Runtime and security policy layers remain separate from untrusted user, tool, MCP, and child-agent content. Prompt quality tests detect duplicate IDs, invalid versions, missing metadata, variable/placeholder errors, unexpected hashes, and undeclared references.

Security model

• Permissions and contextual policies deny unmatched requests by default.
• Agent execution identities are issued by the runtime and linked to workflow/task/session IDs.
• Child context, authority, and budget are reduced from the parent.
• MCP responses and model/tool/child output are untrusted data, not authority.
• Secret references and output redaction limit accidental exposure.
• Tool requests pass manifes