A multi-agent AI system is several model-driven agents working on one problem, often by debating or dividing the work. Building one taught us that agreement between agents proves little, that every conclusion should be a recorded commitment graded against what happened later, and that calibration matters more than confident wording.
SignalDesk is a lab project of ours: a market research terminal where five AI analysts argue, and every call gets graded later. It ingests public filings and news, has a committee of agents debate each signal, commits to a call, and then scores itself over time.
It is not for sale, it is not a trading tool, and nothing in this post is investment advice. We built it to learn how to make AI systems that know when they are wrong. Markets were a convenient subject for one reason: the world eventually tells you whether a call was right.
What follows is what the project taught us about multi-agent design. We are not publishing its scores. The lessons are about how to build such a system, not about how well this one predicts anything.
What is a multi-agent AI system?
A multi-agent AI system is software in which several AI agents, each a language model with its own instructions and tools, work on the same problem and exchange their output. They may split a task into parts, check each other, or argue opposing positions before the system settles on a result.
A single agent is a model taking steps toward a goal. We explain that in what are AI agents. A multi-agent system adds a second layer: some structure that decides who speaks, in what order, and how a final answer is reached.
In SignalDesk that structure is a committee. The agents debate each signal drawn from public filings and news, and the system commits to one call.
Why use several AI agents instead of one?
Several AI agents are used instead of one when a problem benefits from separate perspectives or from having work checked by something other than its author. One agent asked to produce an answer and critique it tends to defend its first idea. Separate agents can be instructed to disagree.
That is the theory, and it holds only under conditions. Agents that share the same underlying model, the same instructions, and the same information will mostly agree with each other. Their agreement then tells you nothing new.
| Single agent | Multi-agent | |
|---|---|---|
| Cost per task | Lower | Higher, one or more calls per agent |
| Speed | Faster | Slower |
| Easy to debug | Yes | Harder, more moving parts |
| Built-in challenge to the answer | No | Yes, if agents differ |
| Good for | Clear tasks with a checkable result | Ambiguous questions where a second view helps |
For most business tasks, the left column is the right choice.
What did we learn building a multi-agent system?
Building a multi-agent system taught us that the debate is the least important part. What made the system useful was forcing a commitment, recording it, and grading it later. Without a score, a committee of agents is only a longer way to produce a confident paragraph.
- Force a commitment. A debate that ends in "it could go either way" cannot be graded. SignalDesk commits to a call each time, so there is something definite to check.
- Grade every call later. The score comes from what happened, not from how persuasive the reasoning was. This is the part most AI systems skip.
- Track calibration, not confidence. A model's confident tone says nothing about whether it is right. What matters is whether calls made with high stated confidence turn out right more often than calls made with low confidence.
- Agreement is not evidence. Agents built from the same model share the same blind spots. Five of them agreeing is closer to one opinion repeated than to five independent ones.
- Give agents real reasons to differ. Disagreement has to be designed in through different instructions or different information. Otherwise the debate is theater.
- Keep the record. Store what each agent said and what the system decided. When a call is graded wrong, the record is how you find out why.
- Count the cost. Each agent is a separate set of model calls. A committee multiplies the usage bill and the time to an answer.
- Start with one agent. Add a second only when you can name the specific error it would catch.
What is calibration and why does it matter more than confidence?
Calibration is how well a system's stated confidence matches how often it is right. A well calibrated system that says it is 70 percent sure is correct about 70 percent of the time on such calls. It matters more than confidence because language models sound sure regardless.
This is the idea the whole experiment rests on. A model writes fluent, assured prose by default. If you take that tone as a signal, you will trust it most in exactly the cases where you should not.
Calibration can only be measured against outcomes, which is why SignalDesk scores itself over time. A system that never learns whether it was right has no way to know when to be cautious.
The same principle applies outside research. An AI feature in a business product should have some way to find out it was wrong: a reviewer's corrections, a customer's reply, a later fact. We discuss the reviewer's role in human in the loop AI.
When is a multi-agent system worth building?
A multi-agent system is worth building when the question is ambiguous, a wrong answer is costly, results can be graded afterward, and a single agent has already been tried and found lacking. If any of those is missing, the extra agents add expense and complexity without adding reliability.
Most business problems do not meet that bar. Drafting a document, summarizing records, or answering a visitor's question is a single-agent job, often with plain code around it. We lay out that choice in when to use AI and when plain code is better.
SignalDesk is in our lab section because it is an experiment. We would not propose its architecture to a client who needed a chat or a drafting tool.
Frequently asked questions
Do multiple AI agents make fewer mistakes than one?
Not automatically. If the agents share a model and the same information, they tend to share mistakes. Extra agents help when they are set up to examine a problem differently and when something checks the final result.
How many agents should a system have?
As few as the problem requires. SignalDesk has five, and it is an experiment. Each added agent raises cost and makes failures harder to trace, so the default for a business tool is one.
Can AI agents grade their own work?
They can compare an earlier call with a later recorded outcome, which is what scoring over time means. Having a model judge its own reasoning without an outside outcome is much weaker, because it tends to approve of what it wrote.
Is SignalDesk available to use?
No. It is a private lab project built for learning, not a product, and it does not give investment advice. Its purpose was to teach us how to build systems that can measure their own errors.
Are multi-agent systems expensive to run?
They cost more than a single agent, because every agent makes its own model calls on each task.
If you are weighing a multi-agent design and want an honest view on whether one agent would do, tell us the problem.