Kozuchi Agent — paper reproduction notebook

This notebook walks through every empirical claim in the paper Kozuchi Agent: A Language-Agnostic Open-Weight Agent for Software Repair and re-derives the cited number from the artifact bundled here. The companion file ../INDEX.md is the index between every paper number and the script + section + output cited below.

Cells use only the Python standard library + json. Re-running Cells -> Run All regenerates every value in this notebook from the files inside paper/artifacts/. Running the figure scripts is optional (it requires matplotlib / pandas).

0. Setup

import json, csv, subprocess, sys, os
from pathlib import Path

ARTIFACT_ROOT = Path('..').resolve()
DATA = ARTIFACT_ROOT / 'data'
STATS = ARTIFACT_ROOT / 'stats'
FIGURES = ARTIFACT_ROOT / 'figures'
PAPER = ARTIFACT_ROOT / 'paper'

print('artifact root :', ARTIFACT_ROOT)
print('data dir      :', DATA)
print('stats dir     :', STATS)
print('figures dir   :', FIGURES)
print('paper dir     :', PAPER)

artifact root : /home/mbahrami/repo/swe-sota-agent/paper/artifacts
data dir      : /home/mbahrami/repo/swe-sota-agent/paper/artifacts/data
stats dir     : /home/mbahrami/repo/swe-sota-agent/paper/artifacts/stats
figures dir   : /home/mbahrami/repo/swe-sota-agent/paper/artifacts/figures
paper dir     : /home/mbahrami/repo/swe-sota-agent/paper/artifacts/paper

# Regenerate every number under stats/. Idempotent.
env = os.environ.copy()
env['KOZUCHI_ARTIFACT_ROOT'] = str(ARTIFACT_ROOT)
result = subprocess.run(['bash', str(ARTIFACT_ROOT/'scripts/numbers/run_all.sh')], env=env, capture_output=True, text=True)
if result.returncode != 0:
    print('STDOUT:', result.stdout[-1500:])
    print('STDERR:', result.stderr[-1500:])
    raise RuntimeError(f'run_all.sh failed (code {result.returncode})')
print('Regenerated', sum(1 for _ in STATS.iterdir()), 'files under stats/')

Regenerated 37 files under stats/

def load_json(name):
    return json.loads((STATS / name).read_text())

def load_csv(name):
    with (STATS / name).open() as f:
        return list(csv.DictReader(f))

def show(name):
    print(f'-- {name} --')
    print((STATS/name).read_text().strip())
    print()

1. RQ1 - Headline accuracy and uncertainty

The paper reports 374/500 = 74.80% resolved on the official SWE-bench Verified cloud evaluator (Wilson 95% CI [70.82, 78.41]) and 376/500 = 75.20% on the internal Docker re-grade. The corresponding sources are data/experiments/.../src/csv/headline.csv and the cross-agent selector summary in data/trajectories/.../xcheck/results/...75p2.json.

wf = load_json('workflow_replacement.json')
res = wf['results']
tot = res['total_instances']
print(f"Official cloud:  {res['resolved_official_cloud']}/{tot} = {res['resolved_official_cloud']/tot:.4f}")
print(f"Internal Docker: {res['resolved_internal_docker']}/{tot} = {res['resolved_internal_docker']/tot:.4f}")
print(f"Patch apply rate: {res['patches_apply_cleanly_n']}/{res['patches_apply_total_n']}")

Official cloud:  374/500 = 0.7480
Internal Docker: 376/500 = 0.7520
Patch apply rate: 494/495

headline = list(csv.DictReader((DATA/'experiments/evaluation/verified/20260326_kozuchi-mini-swe-agent_qwen3.5-27b/src/csv/headline.csv').open()))
print('headline.csv (first row):')
print(json.dumps(headline[0], indent=2))

headline.csv (first row):
{
  "metric": "resolved_pass@1_TTS@8",
  "value": "0.748",
  "numerator": "374",
  "denominator": "500",
  "ci_lo": "0.7081521751849856",
  "ci_hi": "0.7840661518868807"
}

1.1 Per-repository breakdown (Table tab:byrepo)

by_repo = load_json('resolved_by_repo.json')
rows = by_repo['rows'] if isinstance(by_repo, dict) and 'rows' in by_repo else by_repo
for r in rows:
    print(f"  {r['key']:>30} {r['resolved']:>4}/{r['total']:<4}  rate={r['rate']*100:5.1f}%")

                 astropy/astropy   13/22    rate= 59.1%
                   django/django  177/231   rate= 76.6%
           matplotlib/matplotlib   23/34    rate= 67.7%
                 mwaskom/seaborn    1/2     rate= 50.0%
                   pallets/flask    1/1     rate=100.0%
                    psf/requests    8/8     rate=100.0%
                   pydata/xarray   18/22    rate= 81.8%
               pylint-dev/pylint    3/10    rate= 30.0%
               pytest-dev/pytest   16/19    rate= 84.2%
       scikit-learn/scikit-learn   27/32    rate= 84.4%
               sphinx-doc/sphinx   30/44    rate= 68.2%
                     sympy/sympy   57/75    rate= 76.0%

2. RQ2 - Peer-ranked position

We are 12th overall on a frozen leaderboard of 135 catalogued SWE-bench Verified submissions, and the highest-ranked open-weight system. McNemar paired tests with BH-FDR (q<=0.05) yield 16/17 wins among curated open-weight peers. Sources: data/experiments/.../src/csv/leaderboard.csv, mcnemar.csv, multiple_comparison_corrected.csv.

lb_path = DATA/'experiments/evaluation/verified/20260326_kozuchi-mini-swe-agent_qwen3.5-27b/src/csv/leaderboard.csv'
with lb_path.open() as f:
    rows = list(csv.DictReader(f))
print(f'leaderboard.csv has {len(rows)} rows')
ours = [r for r in rows if 'Kozuchi' in r.get('name','')]
print('Our row(s):')
for r in ours:
    keep = {k: r[k] for k in ('rank','name','rate','resolved','total','open_weight') if k in r}
    print(' ', keep)

leaderboard.csv has 135 rows
Our row(s):
  {'name': 'Kozuchi mini-swe-agent + Qwen3.5-27B', 'rate': '0.748', 'resolved': '374'}

3. RQ4 - Failure taxonomy and reliability

Of 126 unresolved instances, 91.3% are WRONG_FIX (clean-applying patches that miss hidden tests); zero are PATCH_DID_NOT_APPLY or EMPTY_PATCH; 494/495 produced patches apply cleanly through the SWE-bench harness.

fm_path = DATA/'experiments/evaluation/verified/20260326_kozuchi-mini-swe-agent_qwen3.5-27b/src/csv/failure_modes.csv'
fm = list(csv.DictReader(fm_path.open()))
for row in fm:
    print(' ', row)

  {'bucket': 'RESOLVED', 'n': '374', 'share_of_total': '0.748', 'share_of_unresolved': '0.0'}
  {'bucket': 'MISSING_ARTEFACT', 'n': '5', 'share_of_total': '0.01', 'share_of_unresolved': '0.03968253968253968'}
  {'bucket': 'EMPTY_PATCH', 'n': '0', 'share_of_total': '0.0', 'share_of_unresolved': '0.0'}
  {'bucket': 'PATCH_DID_NOT_APPLY', 'n': '0', 'share_of_total': '0.0', 'share_of_unresolved': '0.0'}
  {'bucket': 'WRONG_FIX', 'n': '115', 'share_of_total': '0.23', 'share_of_unresolved': '0.9126984126984127'}
  {'bucket': 'REGRESSION', 'n': '6', 'share_of_total': '0.012', 'share_of_unresolved': '0.047619047619047616'}

pa_path = DATA/'experiments/evaluation/verified/20260326_kozuchi-mini-swe-agent_qwen3.5-27b/src/csv/patch_apply_outcomes.csv'
for row in csv.DictReader(pa_path.open()):
    print(' ', row)

  {'patch_applied': 'false', 'n': '1', 'resolved': '1', 'rate': '1.0', 'ci_lo': '0.2065493117918027', 'ci_hi': '1.0'}
  {'patch_applied': 'true', 'n': '494', 'resolved': '373', 'rate': '0.7550607287449392', 'ci_lo': '0.7152648017620162', 'ci_hi': '0.7909204415607707'}

4. RQ5 - TTS@8 selection lift and cost-diversity envelope

Eight runs resolve 334-343 / 500 instances. Cross-agent selection raises this to 374 (cloud) / 376 (Docker), a +33 to +42 lift. Oracle union ceiling is 408; intersection 234.

per_run = load_json('per_run_pass1.json')
print('Eight Python runs:')
for r in per_run['runs']:
    print(f"  {r['run']}  resolved={r['resolved_instances']:>3}/{r['total_instances']}  unresolved={r['unresolved_instances']:>3}")
print(f"min={per_run['min_resolved']}, max={per_run['max_resolved']}, mean={per_run['mean_resolved']}, std={per_run['stdev_resolved_population']}")

Eight Python runs:
  r01_s1001  resolved=343/500  unresolved=139
  r02_s1002  resolved=339/500  unresolved=143
  r03_s1003  resolved=337/500  unresolved=145
  r04_s1004  resolved=341/500  unresolved=145
  r05_s1005  resolved=341/500  unresolved=144
  r06_s1006  resolved=334/500  unresolved=146
  r07_s1007  resolved=334/500  unresolved=148
  r08_s1008  resolved=340/500  unresolved=138
min=334, max=343, mean=338.625, std=3.12

xc = load_json('xcheck_summary.json')
print('Cross-agent selector (Docker re-grade):')
print(f"  resolved              = {xc['resolved_count']}/{xc['expected_count']}")
print(f"  selected              = {xc['selected_count']}")
print(f"  ties broken (length)  = {xc['tie_instances']}")
print(f"  changed_from_order    = {xc['changed_from_order']}")
print(f"  weights w_B / w_R     = {xc['f2p_weight']} / {xc['p2p_weight']}")
print('Order baseline:')
print(f"  resolved              = {xc['order_baseline']['resolved_count']}")

Cross-agent selector (Docker re-grade):
  resolved              = 376/500
  selected              = 495
  ties broken (length)  = 304
  changed_from_order    = 302
  weights w_B / w_R     = 0.3 / 0.7
Order baseline:
  resolved              = 362

ov = load_json('run_overlap.json')
print(f"resolved-in-any-run    = {ov['n_resolved_in_any_run']}")
print(f"resolved-in-every-run  = {ov['n_resolved_in_every_run']}")
import statistics
jacs = [p['jaccard'] for p in ov['pairwise_completed_overlap'] if p['jaccard'] is not None]
print(f"pairwise Jaccard       min={min(jacs):.3f} max={max(jacs):.3f} mean={statistics.mean(jacs):.3f}")

resolved-in-any-run    = 408
resolved-in-every-run  = 234
pairwise Jaccard       min=0.943 max=0.965 mean=0.957

ab = load_json('selector_ablation.json')
rows = [
    ('per-run mean (K=1)',           ab['per_run_mean_resolved']),
    ('per-run worst (K=1 worst)',    ab['per_run_worst_resolved']),
    ('per-run best  (K=1 best)',     ab['per_run_best_resolved']),
    ('size-order baseline',          ab['order_baseline_resolved']),
    ('cross-agent selector (Docker)', ab['cross_agent_selector_resolved_internal']),
    ('cross-agent selector (cloud)',  ab['cross_agent_selector_resolved_official']),
    ('oracle pass@8 (any run)',      ab['oracle_tts8_resolved']),
]
for name, val in rows:
    print(f"  {name:>32}  resolved={val:>5} / {ab['expected_count']}")
print('  -- variants (artifact-derived):')
for vname, v in ab['variants'].items():
    print(f"  {vname:>32}  resolved={v['resolved_count']:>5}")

                per-run mean (K=1)  resolved=338.625 / 500
         per-run worst (K=1 worst)  resolved=  334 / 500
          per-run best  (K=1 best)  resolved=  343 / 500
               size-order baseline  resolved=  362 / 500
     cross-agent selector (Docker)  resolved=  376 / 500
      cross-agent selector (cloud)  resolved=  374 / 500
           oracle pass@8 (any run)  resolved=  408 / 500
  -- variants (artifact-derived):
                   self_tests_only  resolved=  346
        cross_agent_bug_tests_only  resolved=  324
  cross_agent_regression_tests_only  resolved=  313
    cross_agent_all_tests_estimate  resolved=  331

5. RQ6 - Cross-language transfer to Multi-SWE-bench Java

Same 27 B agent, same 8-phase scaffold, same xcheck@8 family on Java: 41/128 = 32.03%, rank 4/42 overall, 1st strict open-weight submission. Per-phase message share within +/-5 pp of Python on every phase. The standalone reproduction bundle is ../scripts/rq6_cross_track/.

rq6_csv = list(csv.DictReader((ARTIFACT_ROOT/'scripts/rq6_cross_track/csv/headline.csv').open()))
print('rq6_cross_track/csv/headline.csv:')
for row in rq6_csv:
    print(' ', row)

rq6_cross_track/csv/headline.csv:
  {'track': 'Python', 'benchmark': 'SWE-bench Verified', 'n': '500', 'resolved': '374', 'rate': '0.7480', 'ci_lo': '0.7082', 'ci_hi': '0.7841', 'source': 'experiments/evaluation/verified/20260326_kozuchi-mini-swe-agent_qwen3.5-27b/src/csv/headline.csv'}
  {'track': 'Java', 'benchmark': 'Multi-SWE-bench Java', 'n': '128', 'resolved': '41', 'rate': '0.3203', 'ci_lo': '0.2457', 'ci_hi': '0.4054', 'source': 'experiments/java/kozuchi-mswe-java-20260429/src/csv/headline.csv'}

rq6_phase = list(csv.DictReader((ARTIFACT_ROOT/'scripts/rq6_cross_track/csv/phase_distribution.csv').open()))
print(f"{'phase':<20}{'python':>10}{'java':>10}{'delta_pp':>12}")
for row in rq6_phase:
    py = float(row['python_share']) * 100
    ja = float(row['java_share']) * 100
    print(f"  {row['phase']:<18}{py:>9.2f}%{ja:>9.2f}%{ja-py:>+12.2f}")

phase                   python      java    delta_pp
  ISSUE_REPRODUCT       12.50%    13.50%       +1.00
  TEST_SYNTHSIZE        11.50%    11.80%       +0.30
  CODE_LOCALIZE          9.40%    12.90%       +3.50
  TEST_LOCALIZE         16.50%    11.20%       -5.30
  CODE_FIX              21.90%    23.20%       +1.30
  VERIFY_PATCH          16.70%    12.60%       -4.10
  ISSUE_CLOSE            7.60%    10.40%       +2.80
  FINAL_REPORT           3.90%     4.40%       +0.50

6. Workflow replacement (Section 3.4) and operational outcomes (Section 11)

wf = load_json('workflow_replacement.json')
we = wf['workflow_estimate']
ci = wf['ci']
configs = wf['configs']
print(f"manual touch-points     pre={we['n_pre_steps']}  post={we['n_post_steps']}")
print(f"compression factor      x{we['compression_factor_x']}")
print(f"engineer-min saved      ~{we['minutes_saved_per_cycle_estimate']} per cycle")
print(f"action formats          {configs['n_action_formats']}")
print(f"backend model configs   {configs['n_model_configs']}")
print(f"CI reusable stages      {ci['n_reuse_vars']} reuse vars over {ci['n_stages']} CI stages")
print(f"cluster envs            {ci['n_cluster_env_names']} ({ci['cluster_env_names']})")

manual touch-points     pre=5  post=1
compression factor      x5
engineer-min saved      ~70 per cycle
action formats          4
backend model configs   17
CI reusable stages      6 reuse vars over 9 CI stages
cluster envs            5 (['abci', 'ashitaka', 'azalea', 'kagura', 'stratus'])

7. Inventories (Design section 5)

Phase graph = 8 phases / 15 edges; 4 action formats; tools = {line_trace, caller_trace, line_edit}; 14 phase-and-tool-gated skills.

p = load_json('phase_inventory.json')
print(f"phases: {p['n_phases']} ; edges: {p['n_edges']}")
print(f"initial_phase: {p['initial_phase']}")
print(f"phase names: {[ph['name'] for ph in p['phases']]}")

af = load_json('action_format_inventory.json')
print(f"\naction formats ({af['n_formats']}): {af['available_action_formats']}; default={af['default_action_format']}")

ti = load_json('tool_inventory.json')
print(f"\ntools active in config: {ti['active_in_config']}")
print(f"tools on disk:          {ti['on_disk']}")
print(f"tools disabled:         {ti.get('disabled_in_config', [])}")

si = load_json('skill_inventory.json')
print(f"\nskills: {si['n_skills']}")

phases: 8 ; edges: 15
initial_phase: ISSUE_REPRODUCT
phase names: ['ISSUE_REPRODUCT', 'TEST_SYNTHSIZE', 'CODE_LOCALIZE', 'TEST_LOCALIZE', 'CODE_FIX', 'VERIFY_PATCH', 'ISSUE_CLOSE', 'FINAL_REPORT']

action formats (4): ['markdown_bash', 'tool_call', 'toolcall', 'tool_bash']; default=tool_bash

tools active in config: ['line_trace', 'caller_trace', 'line_edit']
tools on disk:          ['caller_trace', 'kanban_update', 'line_edit', 'line_trace']
tools disabled:         ['kanban_update']

skills: 14

8. Optional - regenerate figures

These cells require matplotlib (and pandas/seaborn/openpyxl for the cover plot).

import importlib.util, subprocess
have_mpl = importlib.util.find_spec('matplotlib') is not None
if have_mpl:
    print('matplotlib detected; running figure scripts ...')
    subprocess.run([sys.executable, str(ARTIFACT_ROOT/'scripts/rq6_cross_track/build_rq6.py'),
                    '--paper-root', str(PAPER)], check=True)
    subprocess.run([sys.executable, str(ARTIFACT_ROOT/'scripts/figures/plot_python_vs_java.py')], check=True, env=env)
    print('done')
else:
    print('matplotlib not installed; skipping. Run ./reproduce.sh from a shell instead.')

matplotlib detected; running figure scripts ...

updated stats/cross_track_summary.tex
updated figures/cross_track_kozuchi.png
wrote out/csv/cross_track_summary.csv
wrote out/tables/cross_track_summary.tex
wrote out/figures/cross_track_kozuchi.png

wrote /home/mbahrami/repo/swe-sota-agent/paper/artifacts/figures/fig_python_vs_java.png
done

from IPython.display import Image, display, Markdown
for name in ('cross_track_kozuchi.png', 'fig_python_vs_java.png', 'broad_success_vs_params_with_java.png'):
    path = FIGURES / name
    if path.exists():
        display(Markdown(f'### {name}'))
        display(Image(filename=str(path)))
    else:
        display(Markdown(f'(figure missing: {name})'))

cross_track_kozuchi.png

fig_python_vs_java.png

broad_success_vs_params_with_java.png

9. Index

See ../INDEX.md for a per-row mapping of every paper claim to the script and section it comes from. See ../REFERENCES_INDEX.md for the annotated bibliography. The full paper is at ../paper/main.pdf.