Agent Framework
Building Intelligent Agent Orchestration at Rox
At Rox, we're building agents that provide intelligence and take actions over complex data layers. Our agents have access to meeting transcripts, emails, notes, public data, and artifacts—all critical context for delivering value to sellers. But having access to data is just the beginning. These agents need to take meaningful actions: sending emails, scheduling meetings, writing reports, and orchestrating complex workflows.
To deliver these capabilities effectively, we've built an agent orchestration system that dynamically manages pipelines of specialized worker agents. This post dives into the technical architecture that makes this possible.
The Core Architecture
Our system consists of three fundamental components:
Worker Agents - Specialized LLMs with domain-specific tools
Orchestrator - Dynamically constructs execution pipelines based on user queries
Pipeline Engine - Manages the flow of data and execution between agents
Let's explore each component in detail.
Worker Agents: The Specialized Executors
A worker agent is an LLM equipped with a focused system prompt and a curated set of tools. Unlike generic LLMs, each worker agent is optimized for specific tasks with access to precisely the tools it needs.
Tool Categories
Worker agents leverage three primary categories of tools:
Data Tools — Fetch data from our knowledge graph using intelligent, scoped queries. These tools maximize relevant context by providing agents with access to meetings, emails, and other critical data sources.
Verifier Tools — Validate temporary data before allowing agents to continue reasoning. These tools ensure data correctness at each step, with only verified outputs used for subsequent reasoning and user-facing results.
Instruction Fetching Tools — Retrieve specialized instructions based on context. For instance, a report writing agent can fetch specific report templates relevant to the current task.
Example: Email Composer Agent
Consider our email composer agent, which has tools to:
Fetch contacts associated with a company
Retrieve email templates based on context
Verify composed emails (validating to, cc, bcc, subject, body, and signature fields)
This isn't just a simple LLM call with structured output. The agent can iteratively call tools, compose multiple emails, and revise its output based on accumulated context—all while streaming results to the user.
The Orchestrator: Dynamic Pipeline Construction
The orchestrator serves as the brain of our system. It takes user queries and dynamically constructs appropriate pipelines of worker agents. With high-level knowledge of each worker agent's capabilities (but not implementation details), the orchestrator transforms user intent into executable todo items.
This architecture delivers two critical benefits:
Query Refinement — Underspecified user queries are automatically refined based on Rox's capabilities, allowing worker agents to focus on what they do best while collaborating to deliver meaningful results.
User Training — Queries outside Rox's feature set are identified and returned with guidance, helping users learn to leverage the agent system most effectively.
Pipelines: Composing Complex Workflows
Complex tasks require sophisticated coordination. Our pipeline system breaks these tasks into logical components, each handled by a specialized worker agent.
A pipeline consists of:
A sequence of worker agents
Post-hooks that process results between agents
Todo items that define each agent's specific task
Each worker agent receives:
Its specific todo item (the task prompt)
Context from previous agent executions via post-hooks
Access to its specialized tools
Pipeline Design Philosophy
Breaking tasks into smaller, focused components allows us to:
Optimize each agent's prompt and tools for its specific task
Avoid polluting context with irrelevant metadata
Create reusable, composable agent workflows
For example, our email composer agent has a detailed system prompt, template fetching capabilities, and verification tools—all optimized for email composition without burdening the overall pipeline with this specialized context.
While we've found that simple linear pipelines perform best in practice, our architecture supports parallelization when beneficial. In principle, an agent pipeline can be any directed acyclic graph with a single source and sink.
The Power of Specialization
By separating concerns across orchestration, pipeline management, and specialized worker agents, we've created a system where:
Each component excels at its specific role
Complex workflows emerge from simple, composable parts
The system scales with new capabilities through additional worker agents
Users receive intelligent, context-aware assistance without needing to understand the underlying complexity
This architecture enables Rox to handle everything from simple data queries to complex multi-step workflows like managing sales opportunities or orchestrating outreach campaigns—all while maintaining the flexibility to evolve with our users' needs.
Last updated