#3 LLM Consortium for Software Design Refinement: A Controlled Experiment on Multi-Agent Collaboration Topologies

Abstract Snapshot

Compressed abstract

We present a controlled experiment evaluating 12 multi-agent LLM collaboration topologies for software architecture design. Using a 222 factorial design (Authority Roles Dynamics), we conducted 520 experimental runs across 8 design tasks of varying complexity, with 5 repetitions each.

Main idea

We present a controlled experiment evaluating 12 multi-agent LLM collaboration topologies for software architecture design.

Method signal

Using a 222 factorial design (Authority Roles Dynamics), we conducted 520 experimental runs across 8 design tasks of varying complexity, with 5 repetitions each. Designs were evaluated on a 12-dimensional rubric by three independent automated evaluators (GPT-OSS 120 B, Claude Opus 4.6, Claude Sonnet 4.6).

Contribution signal

We report four core findings. First, structural adversarial (v4 b) ranks #1 by ensemble -- a prompt-engineered adversarial variant that demands rewrite mandates rather than patches (weighted ensemble: 4.637/5.0).

Original Abstract

We present a controlled experiment evaluating 12 multi-agent LLM collaboration topologies for software architecture design. Using a 222 factorial design (Authority Roles Dynamics), we conducted 520 experimental runs across 8 design tasks of varying complexity, with 5 repetitions each. Designs were evaluated on a 12-dimensional rubric by three independent automated evaluators (GPT-OSS 120 B, Claude Opus 4.6, Claude Sonnet 4.6). We report four core findings. First, structural adversarial (v4 b) ranks #1 by ensemble -- a prompt-engineered adversarial variant that demands rewrite mandates rather than patches (weighted ensemble: 4.637/5.0). Second, cross-model review wins unanimously at #2 -- generate with one model, review with another -- ranking #2 by all three evaluators (weighted ensemble: 4.606). Third, evaluator diversity is itself a finding -- all three evaluators agree v4 b is best and v3 is worst, but disagree sharply on v2 b (Claude d=1.44 vs. GPT-OSS d=0.45), revealing how different model families weight design qualities. Fourth, parallel merge is fundamentally broken -- all three evaluators place merge variants in the bottom tier (3.65-3.79), due to token starvation and the Frankenstein effect. The weighted ensemble (2 + 2 + 1-OSS) provides robust rankings across 520 runs, confirmed through independent cross-validation.