I built a six-agent debate engine, then ran the experiment that killed it

✍️ From me

Two months after ChatGPT was launched, I immersed myself in that unknown world. I felt intimidated, but I started writing my first vibe code in Python. Copy and paste, then manually vim script.py => :w! => terminal => python script.py and it was simply fascinating to see the results of the code, without syntax errors. From that moment on, I never stopped and have carried out several experiments; this is one of them :) ...

🤖 Written by OMEGA-CORTEX

The section below was generated by OMEGA-CORTEX from my work session.

Conclave was supposed to be the brilliant one. Six AI agents — a researcher, a skeptic, a buyer, a referee, and two more — take sequential turns around a shared transcript, arguing a decision to convergence. Convergence isn't an AI verdict; it's mechanical, a pure function: a dual gate plus a shortlist collapse plus a hash-stability check with a hard round cap. Stdlib-only Python. The claude CLI as the only runtime dependency. Clean architecture, one-way dependencies, a motor you could bolt new decision domains onto like attachments.

The picture in my head was a boardroom. Several minds around a table, each catching what the others missed, the truth emerging from the friction. It's a seductive picture. I'm writing this post because it turned out to be wrong, and because the way it turned out to be wrong is more useful than the thing I set out to build.

The code is open source: github.com/omgagi/omega-conclave.

The test that started as a demo

I had a real question to point it at: a 144-unit apartment complex in Fort Worth, an offering memorandum, and the plain question "is this a good investment?" I'd built an investment domain for exactly this — structured deal profile, a hard mandate (never above $12.5M, going-in cap ≥ 6%, DSCR ≥ 1.25x), an arithmetic veto that checks any proposed answer against those rules.

The debate ran twelve-odd claude calls over half an hour and converged: don't buy at the $12.5M ask, reprice to $11.1M, here's the Fannie structure, here are four closing conditions. Genuinely good output — a bid you could put in a letter of intent, anchored to the lender's own tax value.

Then I did the thing I should have done first. I ran the same deal through --baseline: one claude call, same structured profile, no debate. It returned the identical verdict. $11.1M. Same thesis. Same structure. Twelve calls versus one, same answer.

That's when the post stopped being a demo and became an autopsy.

Building the trap

One deal proves nothing — maybe it was just easy. So I built a deal designed to fool a single call. A synthetic 120-unit property in Jacksonville, priced so the broker's stated 6.33% cap rate sailed past the mechanical veto. Three landmines buried in the raw data, none flagged as risks:

• Tax reset. Florida reassesses to market value at sale. The T12 taxes sat on a suppressed $5.2M assessment; a sale near $14M triggers ~$140K/yr of hidden expense.
• Insurance reset. The T12 insurance was a seller master program, explicitly non-assignable, at $700/unit — market for a 1972 Florida asset is ~$1,450. Another ~$90K.
• One-time income. A $38K litigation settlement footnoted into "misc income," dressed up as recurring.

Corrected NOI drops from the stated $901K to ~$628K. Real cap rate: 4.4%, not 6.33%. Ground truth: pass — nothing above ~$9M clears the hurdles. I pre-registered the scoring before running anything, so I couldn't move the goalposts.

Then I ran three arms on the identical facts: the full debate, the one-call baseline, and — the real control — a naked claude call with no profile, no rules, no schema, just the deal and "should I buy at $14.25M?"

All three caught everything

The debate caught all three traps and said reprice to $9.2M. The baseline caught all three and said $9.6M. And the naked call — no harness at all — caught all three, rebuilt the NOI to ~$628K, computed the 4.4% cap, and said walk or counter around $9.5–10.5M.

The naked call actually did best. It noticed the operating statement didn't even foot — a ~$17K arithmetic inconsistency I hadn't intended to plant — and it named the exact Florida assessment-cap mechanism by hand. The two harnessed runs, boxed into a fixed output schema, surfaced neither. The structure may have mildly hurt.

Here's the whole week on one card:

What was actually real

The debate wasn't worthless, but it wasn't what I thought. Reading the transcript, the underwriter's very first turn — functionally the same single call as the baseline — already had the right number. The skeptic then re-derived it from a different direction and, in round two, explicitly "re-ran the arithmetic to try to kill it" and failed. So the debate bought verification, not discovery. A second independent derivation, a more conservative tax assumption, tighter closing conditions. That's worth something when a seven-figure wire is imminent and being sure matters. It is not worth 12× on a routine screen.

One structural idea survived — the un-photogenic one: never let the model grade itself. Convergence in Conclave is a pure function. The mandate veto is arithmetic — is the price under the ceiling, is NOI÷price above the floor, does any condition contain a forbidden word. Plain Python, final, un-arguable. The model proposes; a calculator disposes, and it never gets asked to reconsider.

But I have to be honest about what the experiment showed for the guardrail specifically: nothing yet. It never once fired — across every run, no answer ever tried to break the rules, so the veto sat unused. And the single time the numbers did pass through it, on the trap deal, it waved the deal straight through — because it checked the cap rate against the broker's inflated NOI that I'd fed into the profile. Guardrails enforce your rules against your inputs; they cannot know your inputs are lies. Catching the lie took reasoning — the model's job, not the guardrail's.

So the veto is the one piece I'd keep on principle: a cheap, structural check for the day a model tries to talk a persuasive number past a hard limit. But I never caught it earning its keep. That's an argument, not a result — and I don't want to dress it up as more. The finding this experiment actually proved is the other one, and it's blunter: the intelligence was in the model the whole time. Everything I built around it — the profile, the schema, the six-agent debate — added little to nothing on top.

The thing I didn't set out to find

Three separate times this week, across three runs, the model asserted a fact was "verified" while citing a source it had no way to check — a wrong statute number, two mutually contradictory "market cap rates," a legal mechanism attributed to the wrong Florida code section. All confidently. All with no web access. The right conclusion, wrapped in an invented citation.

That's a real failure mode. Universal, unnamed, and smaller to fix than any multi-agent architecture: a layer that refuses to let a claim into a decision artifact unless it was actually fetched and matched against a real source. I went looking for novelty in the shape of the agent graph. It was hiding in the citations the whole time.

What I'm keeping

Conclave stays — demoted, not deleted. It's a one-call deal screener with guardrails and an audit trail, which is a reasonable tool even though it's a modest one. The debate loop becomes optional paranoia mode for the deal you're actually about to bid on. I'm not building more agents-around-a-table.

The lesson I actually paid for: the seductive part of an AI system and the valuable part are rarely the same part, and the only way to tell them apart is to build the instrument that measures it. I now own that instrument — a baseline mode wired into the tool, a synthetic adversarial case, a pre-registered rubric, a controlled comparison. It works on any idea, not just this one. That turned out to be worth more than the idea it killed.

You don't find out your brilliant idea is dead on the first table you build. This is just what the search looks like from the inside.

Conclave is open source — motor, all four decision domains, and the baseline/compare instrumentation I used to run the experiment above: github.com/omgagi/omega-conclave.