# MGSA pilot evaluation: baselines, ablations and seed stability

Updated 25 October 2026. These are the existing saved pilot runs, not a new experiment.

## Scope and evaluation protocol

- Task: internal measure-gap / phrase-ending boundary detection in symbolic scores, not complete form classification or OMR.
- Corpus: DCML Annotated Mozart Sonatas; 18 sonatas / 54 movements. Fold 0 uses 10 / 4 / 4 sonatas (30 / 12 / 12 movements) for train / validation / test, grouped by sonata.
- Learned models: five training epochs for each seed 42, 123, 2026, 3407 and 8888. Novelty is a single deterministic baseline without training.
- Targets: annotated phrase structural-ending onsets projected to the nearest inter-measure gap; mid-measure ties project right. No expert boundary targets enter score-derived prior inputs.
- Exact F1: one-to-one matching at the same gap. ±1-measure F1: one-to-one matching within one gap. Test event micro F1 pools counts across movements before calculating the score. F1 is not the fraction of all gaps correctly classified.
- Checkpoint and threshold are selected on validation, not test. Reported ± values are sample SD across seeds (ddof=1), not confidence intervals across musical works.

## Baseline comparison

| Method | Exact F1 | ±1-measure F1 |
|---|---:|---:|
| Novelty (single deterministic run) | 0.2639 (single run) | 0.3624 (single run) |
| Mean-pooling Transformer | 0.3133 ± 0.0164 | 0.4324 ± 0.0109 |
| Matched pooling / no priors | 0.3297 ± 0.0135 | 0.4588 ± 0.0136 |
| Full MGSA | 0.3332 ± 0.0156 | 0.4569 ± 0.0317 |

![Model comparison](../figures/mgsa-baseline-20261025.svg)

The mean-pooling baseline changes the pooling architecture. The matched no-prior control retains learned pooling, the encoder, boundary decoder and auxiliary cadence supervision, while disabling the six attention-logit priors. This is the primary comparison for the added attention priors.

Full MGSA minus matched control: exact F1 **+0.35 percentage points**, tolerant F1 **-0.19 percentage points**.

## Retrained prior ablations

Each variant removes one attention prior and is retrained on the same fold, seeds and five-epoch budget. Pooling and auxiliary cadence supervision are held fixed; removing the cadence attention prior does not remove the auxiliary cadence task.

| Configuration | Exact F1 | ±1-measure F1 |
|---|---:|---:|
| Full MGSA | 0.3332 ± 0.0156 | 0.4569 ± 0.0317 |
| − Metric | 0.3293 ± 0.0053 | 0.4462 ± 0.0254 |
| − Motif | 0.3319 ± 0.0096 | 0.4538 ± 0.0188 |
| − Harmony | 0.3346 ± 0.0085 | 0.4528 ± 0.0199 |
| − Cadence | 0.3353 ± 0.0199 | 0.4477 ± 0.0155 |
| − Repetition | 0.3365 ± 0.0148 | 0.4501 ± 0.0140 |
| − Boundary | 0.3346 ± 0.0111 | 0.4514 ± 0.0214 |

![Prior ablations](../figures/mgsa-ablation-20261025.svg)

All six removals have lower mean tolerant F1 than Full MGSA, while exact F1 shows mixed changes. This is descriptive evidence, not proof that every prior has a reliable effect. The error bars are seed SD, not a test of significance.

## Multi-seed stability

![Per-seed results](../figures/mgsa-seed-stability-20261025.svg)

### Exact F1

| Seed | Mean pooling | Matched pooling | Full MGSA | MGSA − matched |
|---|---:|---:|---:|---:|
| 42 | 0.2913 | 0.3327 | 0.3235 | -0.0092 |
| 123 | 0.3035 | 0.3124 | 0.3222 | +0.0098 |
| 2026 | 0.3237 | 0.3300 | 0.3355 | +0.0055 |
| 3407 | 0.3154 | 0.3239 | 0.3255 | +0.0016 |
| 8888 | 0.3328 | 0.3495 | 0.3595 | +0.0100 |

### ±1-measure F1

| Seed | Mean pooling | Matched pooling | Full MGSA | MGSA − matched |
|---|---:|---:|---:|---:|
| 42 | 0.4134 | 0.4448 | 0.4338 | -0.0110 |
| 123 | 0.4351 | 0.4533 | 0.4402 | -0.0131 |
| 2026 | 0.4388 | 0.4620 | 0.4485 | -0.0135 |
| 3407 | 0.4346 | 0.4534 | 0.4497 | -0.0038 |
| 8888 | 0.4402 | 0.4805 | 0.5124 | +0.0319 |

MGSA exceeds the matched control on exact F1 in **4/5 seeds**, but on tolerant F1 in only **1/5 seeds**. Its tolerant seed SD (0.0317) is larger than the matched control's (0.0136). The data do not establish improved stability.

## Current conclusion and limitations

**No consistent advantage over the matched control has been established.** This pilot covers one composer, one held-out fold and a short training budget. It does not establish statistical significance, cross-composer generalization, complete form recognition or improved OMR. Section-relation and global-form metrics are unavailable. Longer training, more folds and a frozen evaluation protocol remain future work.

## Inspect and reproduce the reporting

- [51 selected public run records](../evidence/pilot_runs.json)
- [Saved summary](../evidence/pilot_summary.json)
- [Aggregate CSV](../evidence/evaluation_summary.csv)
- [Per-seed CSV](../evidence/per_seed_metrics.csv)
- [Paired differences](../evidence/paired_seed_differences.json)
- `python3 examples/summarize_public_metrics.py` recomputes aggregates from the public records. It does not run training.

The score corpus belongs to [DCMLab](https://github.com/DCMLab/mozart_piano_sonatas) under its own terms. No additional scores, checkpoints, private model code or private training configuration are released by this update.