UnrealClientProtocol

UnrealClientProtocol (UCP) is an atomic client communication protocol designed for AI Agents. Its core design philosophy is:

• Don't make decisions for the Agent — give it capability.

Traditional UE automation requires writing dedicated interfaces or scripts for every operation. UCP takes the opposite approach — it exposes only the engine's atomic capabilities (call any UFunction, read/write any UPROPERTY, find objects, introspect metadata), then trusts the AI Agent's own understanding of the Unreal Engine API to compose these primitives into arbitrarily complex tasks.

This means:

• You don't need to predefine "what it can do." The Agent isn't limited to a fixed set of predefined commands — it has access to every function and property exposed by the engine's reflection system. If the engine can do it, the Agent can do it.
• You can shape Agent behavior with Skills. By authoring custom Skill files, you can inject domain knowledge into specific workflows — level design conventions, asset naming rules, material authoring strategies — and the Agent will combine this knowledge with the UCP protocol to work the way you define.
• Capabilities grow as models evolve. The UCP protocol layer is stable, while AI comprehension is continuously improving. Today the Agent might need describe to explore an unfamiliar class; tomorrow it may already know it by heart. You don't need to change a single line of code to benefit from AI progress.

Why AI Agent + Atomic Protocol + Domain Skills Changes Everything

LLMs Already Understand Unreal Engine

Ask any large language model — even one not specialized for coding — "How do you get all Actors in UE via Blueprint? What's the function signature?" Most LLMs can answer accurately: UGameplayStatics::GetAllActorsOfClass, parameters are WorldContextObject and ActorClass, returns TArray<AActor*>.

This demonstrates that LLMs already possess deep knowledge of Unreal Engine. They understand the reflection system, know the APIs of common modules, and can map Blueprint nodes to their underlying C++ functions. This knowledge isn't exclusive to any particular model — it's a consensus across the entire LLM ecosystem.

But knowledge alone isn't capability. The "balanced" nature of LLMs means they tend toward generic, conservative responses without guidance — they know how to do it, but won't proactively act. What UCP does is simple: give them hands (the protocol) plus a behavioral framework (Skills), turning existing knowledge into actual action.

From ShaderToy to Production Materials — Skill-Driven Continuous Evolution

Take material authoring as an example. UCP's current material Skill can read and write complete material node graphs, meaning the Agent is fully capable of translating ShaderToy code into corresponding UE materials — decomposing logic, building nodes, wiring pins, compiling and verifying, all in one seamless flow.

But the distance from a demo-level effect to a production-grade engineered material isn't about protocol capability — it's about domain experience:

• Workflow experience: Repeated logic should be abstracted into Material Functions; prefer a single parent material covering more variants over creating separate materials for each effect.
• Craft experience: Scene, character, VFX, and UI each have different material strategies; classic techniques (FlowMap, parallax mapping, distance field blending) have proven node composition patterns.
• Self-feedback loops: The Agent should be able to validate its own output — compile the material, check instruction count, compare against expected results — then distill successful experiences into new Skill knowledge.

This is the evolutionary direction of Skills: not a one-time instruction manual, but a continuously accumulating, ever-refining experience base. Every successful material creation can feed back into more precise Skill descriptions, making the Agent better next time.

Skill Memory Layering — A Sustainably Evolving Architecture

The Agent community is adopting a memory layer strategy called "Skills" or "Memory" that dramatically improves AI performance in specific domains by structuring experience into retrievable knowledge fragments.

For Unreal Engine, this strategy has a natural fit. UE itself is highly modular — rendering, physics, animation, UI, networking, audio — each subsystem has independent API hierarchies and best practices. This means Skill layering can naturally align with UE's module structure:

Skills/
├── unreal-client-protocol/            # Transport: protocol spec, invocation patterns
├── unreal-object-operation/           # Foundation: property R/W, object search, metadata
├── unreal-actor-editing/              # Scene: Actor lifecycle management
├── unreal-asset-operation/            # Assets: search, dependencies, CRUD
├── unreal-material-editing/           # Materials: node graphs, HLSL, material instances
├── unreal-blueprint-editing/          # Blueprints: node graph R/W, function/event/variable editing
├── unreal-large-blueprint-analysis/   # BP Analysis: systematic large Blueprint analysis & C++ translation
├── unreal-animation/                  # (future) Animation layer
└── ...

As functionality grows, the Skill system will become increasingly extensive. But this isn't a problem — based on UE's own code structure and LLMs' understanding of UE modules, Skill memory layering can be systematically constructed. When executing tasks, the Agent only loads the relevant Skill subset rather than reading all knowledge at once.

More importantly, this is an open, community-collaborative architecture. Any developer can write Skills for their workflow — a Technical Artist can contribute material authoring experience, a Level Designer can contribute scene building conventions, a Tools Programmer can contribute engine extension patterns — and this knowledge ultimately flows into a shared Skill ecosystem that benefits everyone.

Features

• Zero Intrusion — Pure plugin architecture; drop into Plugins/ and go, no engine source changes required
• Reflection-Driven — Leverages UE's native reflection system to automatically discover all UFunction and UPROPERTY fields
• Minimal Protocol — Only call command, covering every reflectable operation in the engine
• Editor Integration — Property writes are automatically registered with the Undo/Redo system
• WorldContext Auto-Injection — No need to manually pass WorldContext parameters
• Security Controls — Loopback-only binding, class path allowlists, and function blocklists
• Batteries-Included Python Client — Lightweight CLI script to talk to the engine in one line
• Domain Skill Ecosystem — 7 specialized Skills covering objects, Actors, assets, materials, and Blueprints; Agent loads domain knowledge on demand
• Multi-Tool Compatible — Skill descriptors work with Cursor / Claude Code / OpenCode and other mainstream AI coding tools

Skill Ecosystem

The UCP protocol layer provides "atomic capabilities" — calling functions, reading/writing properties. Domain Skills organize these atomic capabilities into domain-specific work patterns, letting the Agent load the right expertise for each task scenario.

Currently available Skills:

| Skill | Layer | Capabilities | |-------|-------|-------------| | unreal-client-protocol | Transport | TCP/JSON communication, call protocol, error handling & self-correction | | unreal-object-operation | Foundation | Any UObject property R/W, metadata introspection, instance search, derived class discovery, Undo/Redo | | unreal-actor-editing | Scene | Actor spawn / delete / duplicate / move / select, level queries, viewport control | | unreal-asset-operation | Assets | AssetRegistry search, dependency & reference analysis, asset CRUD, editor operations | | unreal-material-editing | Materials | Text-based material node graph R/W, Custom HLSL, material instance editing, compilation & stats | | unreal-blueprint-editing | Blueprints | Text-based Blueprint node graph R/W, function / event / macro editing, variable creation, auto-compilation | | unreal-large-blueprint-analysis | BP Analysis | Systematic large Blueprint (10+ functions) analysis, C++ translation, logic auditing |

Each Skill is a standalone SKILL.md file that the Agent reads on demand when receiving a task. Skills naturally chain through the UCP protocol layer — for example, creating a material requires unreal-asset-operation's asset creation + unreal-material-editing's node graph editing + unreal-object-operation's property setting; the Agent automatically combines multiple Skills to complete composite tasks.

How It Works

Architecture Overview

flowchart TD
    User["User"]
    Agent["AI Agent"]
    
    subgraph skills ["Domain Skills"]
        Proto["unreal-client-protocol<br/>Transport"]
        Obj["unreal-object-operation<br/>Foundation"]
        ActorSkill["unreal-actor-editing<br/>Scene"]
        Asset["unreal-asset-operation<br/>Assets"]
        Mat["unreal-material-editing<br/>Materials"]