# Final Paper-Ready Result: ESMFold2 Generalizability

## Result Summary

We audited Biohub ESMFold2 using SPECTRA-style spectral performance analyses over accumulated target-level evaluations spanning CAMEO and multiple RCSB-derived panels. The initial CAMEO audit found a strong degradation boundary with low evolutionary support, but MSA depth was not globally comparable across the later RCSB panels. We therefore searched for prospective, sequence- or metadata-defined non-MSA axes that explain ESMFold2 performance degradation across saved evaluations.

The broadest non-MSA axis that held across the pooled evidence was **high disulfide/cysteine capacity**. The strongest localized failure mode was **class-II-like viral envelope E/E2 / envelope protein E**. These failures were generally accompanied by low ESMFold2 confidence, so the final result should be framed as a **recognized low-confidence generalization boundary**, not as a hidden overconfident failure mode.

## Evaluated Evidence

The final global evidence table contains:

- 706 unique scored evaluation rows after exact result de-duplication.
- 652 exact-sequence primary rows used for the main global analysis.
- Source composition of the 652 primary rows:
  - CAMEO/CAMEO follow-up: 138
  - external RCSB: 128
  - fresh general RCSB: 267
  - viral RCSB: 119

Inference settings were not perfectly uniform across all rows:

- 557 of 652 primary exact-sequence rows used the scaled ESMFold2 setting: 10 loops, 68 sampling steps, 5 diffusion samples, seed 0, best-by-pTM selection.
- 95 CAMEO rows came from the earlier lower-budget/default audit.

The scaled-only subset still showed a high-disulfide degradation:

- scaled-only n = 557
- high-disulfide top 20% n = 112
- CA-lDDT gap = -0.134

## Primary Non-MSA Axis: High Disulfide Capacity

Feature definition:

```text
disulfide_capacity_per100 = floor(number_of_cysteines / 2) / sequence_length * 100
```

This is a prospective sequence-only proxy. It does not use the reference structure and does not measure actual disulfide bonds. It estimates how many cysteine pairs the sequence could form per 100 amino acids.

Final threshold:

```text
high_disulfide = disulfide_capacity_per100 >= 1.3193
```

This threshold is the top 20% of the exact-sequence primary evidence table.

Global effect:

| group | n | mean CA-lDDT | failure rate CA-lDDT < 0.6 |
|---|---:|---:|---:|
| high disulfide | 134 | 0.713 | 0.306 |
| lower disulfide | 518 | 0.862 | 0.083 |

Top-vs-lower effects:

- Raw CA-lDDT gap: -0.149
- Bootstrap 95% CI: [-0.190, -0.106]
- Adjusted coefficient controlling for log length, fixed baseline CA-lDDT, and source family: -0.081

Source-slice effects:

| source slice | lower n | high n | lower CA-lDDT | high CA-lDDT | high-lower gap |
|---|---:|---:|---:|---:|---:|
| CAMEO/CAMEO follow-up | 126 | 12 | 0.885 | 0.753 | -0.132 |
| external RCSB | 111 | 17 | 0.866 | 0.785 | -0.081 |
| fresh general RCSB | 225 | 42 | 0.895 | 0.787 | -0.108 |
| viral RCSB | 56 | 63 | 0.668 | 0.637 | -0.031 |

The viral RCSB slice is already difficult overall, so the incremental high-disulfide contrast is smaller there.

## Reviewer-Proofing Analyses

### Cluster-Aware Validation

Exact-sequence deduplication was supplemented with approximate k-mer sequence clustering. This is not a substitute for a final MMseqs2/CD-HIT run, but it directly tests whether the effect is driven solely by duplicated or near-duplicated families.

| k-mer Jaccard threshold | clusters | largest cluster | one-rep-per-cluster effect | 95% CI |
|---:|---:|---:|---:|---|
| 0.50 | 518 | 10 | -0.117 | [-0.166, -0.069] |
| 0.35 | 489 | 19 | -0.118 | [-0.169, -0.070] |
| 0.25 | 477 | 21 | -0.121 | [-0.172, -0.071] |

The disulfide effect remains negative after cluster-aware downweighting.

### Matched Controls

High-disulfide rows were greedily matched to lower-disulfide controls on:

- source family
- viral/nonviral status
- log sequence length
- TM/hydrophobic proxy

Matched result:

- matched pairs: 127
- mean high-minus-control CA-lDDT: -0.061
- 95% CI: [-0.107, -0.016]
- median high-minus-control CA-lDDT: -0.034
- high-disulfide target worse in 63.8% of pairs

Matched source slices:

| source slice | pairs | mean high-control CA-lDDT | 95% CI | high worse fraction |
|---|---:|---:|---|---:|
| all matched | 127 | -0.061 | [-0.107, -0.016] | 0.638 |
| CAMEO/CAMEO follow-up | 12 | -0.066 | [-0.222, 0.131] | 0.750 |
| external RCSB | 17 | -0.103 | [-0.245, 0.040] | 0.706 |
| fresh general RCSB | 42 | -0.098 | [-0.159, -0.041] | 0.690 |
| viral RCSB | 56 | -0.020 | [-0.090, 0.054] | 0.554 |

The matched analysis supports a real broad disulfide-capacity effect, but the effect is smaller than the raw pooled contrast.

## Strong Localized Failure Mode: Class-II Viral E/E2

The strongest localized degradation axis is class-II-like viral envelope E/E2 / envelope protein E. This axis is only evaluable in rows with useful RCSB-style metadata, so it is not a global all-row sequence-only axis.

Metadata-evaluable result:

| group | n | mean CA-lDDT | failure rate CA-lDDT < 0.6 | mean pTM |
|---|---:|---:|---:|---:|
| class-II E/E2/envelope-E | 11 | 0.396 | 0.818 | 0.287 |
| other metadata-evaluable rows | 278 | 0.803 | 0.165 | 0.710 |

Effect:

- CA-lDDT gap: -0.407
- Bootstrap 95% CI: [-0.489, -0.322]
- Adjusted coefficient controlling for log length, fixed baseline CA-lDDT, and source family: -0.253
- Within viral RCSB only: CA-lDDT gap -0.282; adjusted -0.220

Interpretation: class-II viral E/E2/envelope-E is the sharpest failure subtype discovered, but it should be reported as a localized failure mode rather than the broad global axis.

## Confidence and Calibration

ESMFold2 confidence largely identifies these failures.

| axis | n | top n | top CA-lDDT | lower CA-lDDT | top pTM | lower pTM | pTM AUC for CA-lDDT >= 0.6 | top overconfident failure rate |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| high disulfide | 652 | 134 | 0.713 | 0.862 | 0.638 | 0.770 | 0.967 | 0.000 |
| class-II E/E2/envelope-E | 289 | 11 | 0.396 | 0.803 | 0.287 | 0.710 | 0.953 | 0.000 |

This means the result is not primarily that ESMFold2 is confidently wrong. The stronger paper framing is that SPECTRA identifies a biologically interpretable, prospective low-confidence boundary where ESMFold2’s structure quality degrades.

## Example Structures

Three selected failures were rerun with coordinate saving using the same scaled setting: 10 loops, 68 sampling steps, 5 diffusion samples, seed 0, best-by-pTM selection. CA-only superposed prediction/reference overlays were saved.

| target | description | CA-lDDT | pTM | CA-RMSD |
|---|---|---:|---:|---:|
| 9B6W_1_A | Yellow fever virus envelope protein | 0.228 | 0.203 | 34.08 |
| 8R2L_1_AAA | Tick-borne encephalitis virus envelope protein E | 0.238 | 0.185 | 31.37 |
| 8R8L_1_A | Tick-borne encephalitis virus peptide 2k / envelope-region chain | 0.252 | 0.179 | 28.87 |

Overlay files are in:

```text
example_structures/predicted_best_by_ptm/
```

## Prior Axes and Why They Are Not Final

MSA depth:

- Strong and claim-valid in CAMEO.
- Low MSA support vs high MSA support gave a CA-lDDT drop of approximately -0.196.
- Not used as the global final axis because most RCSB panels lack comparable MSA-depth fields.

Flexible/non-helical composition:

- Strong in CAMEO and survived CAMEO inference-time scaling.
- External RCSB expansion showed only weak support; paired median delta was approximately zero.
- Retained as CAMEO-local, not broad portable.

Reference geometry/topology:

- Strong benchmark-only axis in CAMEO.
- Not deployable because it uses reference-structure information.

Charge and acidic composition:

- High charge looked promising in saved CAMEO/RCSB outputs.
- Fresh RCSB charge confirmation failed: top charge effect approximately +0.007 CA-lDDT.
- Acidic-window effects were weak and not final.

Broad viral surface/disulfide/low-helical:

- Directionally useful in some panels.
- Broad frozen definitions failed; sharper subtype analysis identified class-II E/E2/envelope-E as the real localized axis.

## Final Claim Boundary

Paper-ready claim:

> In accumulated ESMFold2 evaluations across CAMEO and RCSB-derived panels, high sequence disulfide/cysteine capacity is a broad prospective risk proxy for reduced ESMFold2 structural accuracy, and class-II-like viral E/E2/envelope-E proteins form a sharper localized failure mode. These failures are generally accompanied by low ESMFold2 confidence, making the result a calibrated low-confidence generalization boundary rather than an overconfident hallucination regime.

Current limitations to state explicitly:

- The global table mixes scaled ESMFold2 inference with a smaller number of older CAMEO/default rows.
- Cluster-aware validation uses approximate k-mer clustering. A final publication version can replace this with MMseqs2/CD-HIT if desired.
- Class-II E/E2 is metadata-defined and localized; it is not a global all-protein sequence axis.
- The CA-only overlays are illustrative examples, not full structural biology renderings.