Copyright thenewstack
Orchestrators make sure that AI agents don’t work in a vacuum; they organize the components, processes and information exchange to get work done. Understanding orchestrators will give developers more fine-grained control over the efficiency of AI agents. “Understanding the tools that you can use to solve problems is always a helpful skillset,” said Zachary Hanif, head of AI, machine learning and data at Twilio. That includes finding the most efficient AI model for specific workloads, fine-tuned system-level prompting, and putting the right context and guardrails in place. First and foremost, orchestrators make sure agents are meeting performance expectations and costs. With AI security a major concern, orchestrators also fend off threats. “That stops prompt injection or stops it from swearing or whatever it is that you want to prohibit,” said Michael Atkisson, the AI product management director at Epicor, an enterprise software company. “Any process, AI included, definitely depends on a great deal of orchestration.” – Zachary Hanif, Twilio Hanif compared orchestrators to the real-world example of McDonald’s, which has “mastered the concept of orchestration” in its kitchen. Everything in the McDonald’s kitchen is coordinated — from the layout, to the size of machines, the space, the number of people and how everything works within business constraints. That extends to assembling the sandwich, preparing fries, bagging the order correctly and delivering it within a certain timeframe. “Any process, AI included, definitely depends on a great deal of orchestration,” Hanif said. The Need for Upskilling in AI Orchestration The orchestration of AI agents will fit into traditional software development, with some changes in the typical CI/CD pipeline. “We will live in a world where the traditional and core systems are still there, and new things are coming in,” said Jinsook Han, chief strategy and agentic AI officer at Genpact, a technology consultant. Orchestration of AI agents is directly tied to automating complex business processes, and developers need to adapt to these new areas. That includes understanding workflows and business process management. “I’ve been telling our agent operations that, look, you are the last generation of managers who manage humans alone and you’re the first generation of managers who manage human and AI agents,” Han said. Developers also need to develop domain expertise, or talk to subject matter experts regularly. It is analogous to developers training AI agents — aka digital workers — to do their job more efficiently, which is like training human workers to work effectively. Understanding Orchestration Layers and Agentic Life Cycles A typical orchestration layer involves lining up lots of agents into an action plan of tasks to follow and produce some larger set of tasks. The layer interprets the question, understands its guardrails and context, and then gives an answer. Han specified three layers for orchestration: The first layer determines which steps need to be orchestrated. The second makes sure the audits are in place to guarantee that the AI train is moving in the right direction. The third layer focuses on alerting humans to get involved and provide supervision and oversight. “Let the machines do what machines do best and let the humans do what humans do best,” he said. “But how do you make sure where humans get inserted? How does the handoff happen?” Genpact has implemented what Han calls ADLC (Agentic Life Cycle Development). The AI agent development framework tracks how AI agents are created, deployed, used and improved within orchestrated systems, providing visibility into which agents are being called, how often, and whether they need to be tweaked or optimized. “Over time, you can train a fine-tuned model using supervised fine-tuning reinforcement learning,” Han said. Eventually, the model learns the business process, making workflows unnecessary. What Makes a Well-Orchestrated AI System? A well-orchestrated system creates AI applications where agents are a cohesive set of microservices that perform specific business processes with desired outcomes. Early versions of orchestration had limitations, as machine learning models took a long time to train for very specific, narrow and rigid use cases. Orchestrators function as intelligent coordinators rather than rigid workflow engines. “It’s like putting a rocket on train tracks,” Atkisson said. “It can’t left or right, but it’s very good at going in one direction.” Further training of large language models (LLMs) on diverse domains provided the breakthrough: orchestrators function as intelligent coordinators rather than rigid workflow engines. “It made it so it could go from one thing to the other,” Atkisson said. “It enabled this capability of having kind of a mini-judge in a box.” Key Components and Skills for Building Orchestrators A key component of fine-grained orchestration involves selecting the right AI model for agents, understanding data retrieval, securing the model and prompt engineering. Developers need to know Python, since “most of the tooling for development in agentic AI is done in Python,” Atkisson said. Developers need to know how these models work, and figure out where things may be going wrong. “You need to understand how a transformer works, how tokens are turned into embeddings and embedding models,” Atkisson said. Understanding context is critical for orchestration, he added: “Context is like a fancy way of saying ‘search.'” That means developers need to master data retrieval from vector databases, keyword indexes and hybrid search. AI models are now multimodal, which can process audio, video, image and text input. “Vector databases are very good at understanding a broader context of things, but they’re not very good at very fine distinctions — like the names of column headers in a database,” Atkisson said. That’s where developers need to pull the data to get their answers. Then there’s a security layer that runs audits and defines access to the data. This is where guardrails are inserted to prevent prompt injection and prohibit agents from specific tasks and rogue outputs. Red flags will invoke human involvement. “Evaluation is probably the most important skill right now.” – Michael Atkisson, Epicor Building a good orchestrator requires evaluating output through prompt engineering on different models. There are many agent development environments on Microsoft Azure and OpenAI that offer a range of models to evaluate and test agents against. “Evaluation is probably the most important skill right now,” Atkisson said. “These are essentially unit tests for … LLM real-world use cases. You need a pairing between … here is a prompt, and here’s what good looks like as an output.” “You have to quickly estimate what level of quality you’re getting for this cost,” he continued. “There are those trade-offs. You look at what your per-unit action cost is.” Moreover, large language models are changing every day, so the responses of a specific model may change quickly. Typical quality assurance involves developers going back and forth on prompts against AI models to define the highest quality of responses. “The models are changing all the time, right?” Atkisson said. “And so suddenly it’s maybe a lot better and your test set is now too easy.” The Rise of Pre-Packaged Orchestration Solutions Many service providers plug orchestrators into their pre-packaged AI agent offerings. Box has AI agents that extract context and intelligence from documents, which helps businesses make better decisions. But that requires what the company’s CTO Ben Kus called “continuous orchestration.” “Whether it’s scanning for threat detection, whether it’s classification, whether it’s access controls, whether it’s retention and disposition, all those continue to work with our new age of AI applications,” Kus said. DevRev is providing a complex AI system with an orchestration layer that extracts intelligence from data in disparate systems. The company has created a knowledge graph that orchestrates data access across enterprise systems. “There’s a search engine, a SQL engine and MCP gateway as well.” – Dheeraj Pandey, DevRev For customer-facing applications, DevRev’s system can be deployed with just a few lines of code. MCP is a part of its technology. “There’s a search engine, a SQL engine and MCP gateway as well,” said Dheeraj Pandey, CEO of DevRev. “Workflows are also an engine that’s being published through [the Model Context Protocol] and can be invoked by users via LLMs.” The technology takes automations, annotates and labels them, and publishes them through the MCP protocol, said Pandey, who was previously CEO of Nutanix. “What you build on top of all of this is basically agents and scales,” Pandey said. “The same stack that drives AI agents for us delivers all these.” Good orchestration is important to a good AI implementation. Said Atkisson, “I think what’s more important about AI is very much a secular trend.”
