SPECTRA

Generalizability audits for foundation models

SPECTRA turns model generalization claims into validated spectral performance curves.

This hosted knowledge server exposes the SPECTRA protocol, prior audit findings, and artifact references for agents and humans. Agents connect at https://spectra.yashaektefaie.com/mcp.

3models
3findings
63artifact refs
high similarity low similarity performance freeze prospective axis audit degradation confirm boundary

Protocol

What SPECTRA Requires

01Define the scientific unit

Choose the model, data, task, metric, and prospective novelty axis before target scoring.

02Freeze and validate the split

Measure that train-test or pretraining-test similarity decreases across levels.

03Score baselines and model

Run fixed baselines where labels exist, then evaluate the target model on frozen levels.

04Confirm live hypotheses

Outcome-mined axes are exploratory until frozen and confirmed on fresh or adequate evidence.

05Ledger weak and negative axes

Non-explanatory curves are findings that route back into prospective-axis discovery.

06State the claim boundary

A valid SPC closes only the deployment or mechanism boundary it actually tested.

07Record provenance

Publish model, weight, dataset, metadata, download, cache, and known-gap records with every stored finding.

Provenance

Where the evidence came from

Agents should call get_spectra_provenance or list_spectra_sources before using a stored finding. New findings are incomplete unless they include source repositories, checkpoint or score provenance, dataset access routes, metadata resources, cache roots, and explicit known gaps.

{
  "tool": "get_spectra_provenance",
  "arguments": {
    "model": "ESMFold2"
  }
}

Downloads

Raw tables leave MCP as files

Large CSVs and reproducibility-critical artifacts are published as normal HTTPS downloads with row counts and SHA-256 checksums. Agents should call list_spectra_downloads, then use curl, wget, or a dataframe loader on the returned URL.

{
  "tool": "list_spectra_downloads",
  "arguments": {
    "model": "STATE",
    "query": "target_model_results"
  }
}
CONCHTUM tissue/stain-fraction proximity tail ESMFold2high disulfide/cysteine capacity STATEpathway/module train-support density

Current Evidence

Stored SPECTRA Findings

These summaries are human-readable views of the same structured records exposed through the MCP tools.

CONCH

conch_tum_tissue_stain_tail_20260603

valid localized public crc tum tail boundary / weak independent site generalization

CONCH zero-shot CRC ROI classification shows a claim-valid but localized degradation boundary: true TUM recall falls in the low-similarity tail of a prospective tissue/stain-fraction axis. The result is bounded to public CRC ROI tiles and the CONCH CRC prompt ensemble. It is not an all-class, whole-slide, independent-site, non-CRC, or medical deployment claim.

Primary Axis TUM tissue/stain-fraction proximity tail

Euclidean distance in tissue, white/background, dark, and high-saturation pixel fractions from the class-balanced NCT-CRC-HE-100K reference median. Higher numbered levels mean lower similarity; split membership is frozen before CONCH scoring.

Discovery tiles7,180
CRC-VAL balanced accuracy0.785
Confirmation TUM tiles1,800
Expanded TUM tiles4,000
Low-tail recall gap0.223
Expanded Spearman-0.77

Claim Boundary

Public CRC 224x224 H&E ROI tiles scored with the CONCH CRC prompt ensemble. Confirmation and expansion are fresh relative to CRC-VAL discovery but still from the public Kather/NCT source family, so independent-site or clinical deployment generalization remains weak.

Source Provenance

complete
Model
MahmoodLab/conch
Weights / Scores
https://huggingface.co/MahmoodLab/conch
Execution
fresh_zero_shot_inference
Data
  • CRC-VAL-HE-7K
  • NCT-CRC-HE-100K reference stream
  • NCT-CRC-HE-100K-NONORM TUM confirmation and expansion panels
Metadata
  • Zenodo API record 1214456 and Hugging Face dataset README/API
Public downloads

Limitations

  • Confirmation and expansion are still from the public Kather/NCT source family rather than an independent site cohort.
  • The primary result is TUM-class localized, not an all-class or deployment-wide generalizability statement.
  • The expanded curve is threshold-shaped rather than strictly monotone; levels 8-10 are degraded but levels 9 and 10 rebound slightly relative to level 8.

Downgraded Axes

  • broader NCT color/texture proximity: split-valid but not claim-closing
  • all-class CRC ROI generalizability: not supported as broad closure
Key MCP artifact IDs
  • conch:final_report.md
  • conch:artifacts:tum_expansion:expanded_finding_report.md
  • conch:artifacts:tum_expansion:expanded_metrics.json
  • conch:artifacts:tum_expansion:analysis.json
  • conch:artifacts:tum_expansion:scored_tum_expansion_with_baselines.csv
  • conch:artifacts:tum_expansion:target_model_results_tum_expansion.csv
  • conch:artifacts:tum_confirmation:analysis.json
  • conch:artifacts:tum_confirmation:decile_densification.json
  • and 7 more key artifacts in the MCP store

ESMFold2

esmfold2_high_disulfide_boundary_20260602

valid current pool confirmed

Across accumulated CAMEO and RCSB-derived ESMFold2 evaluations, high disulfide/cysteine capacity is a broad prospective risk proxy for reduced structural accuracy. Class-II-like viral E/E2/envelope-E proteins form a sharper localized failure mode. The failures are generally accompanied by low ESMFold2 confidence, so the result is a calibrated low-confidence generalization boundary rather than a hidden overconfident failure regime.

Primary Axis high disulfide/cysteine capacity

disulfide_capacity_per100 = floor(number_of_cysteines / 2) / sequence_length * 100

Primary rows652
High-disulfide n134
Lower-disulfide n518
Raw CA-lDDT gap-0.149
Matched-control gap-0.061
Class-II CA-lDDT gap-0.407

Claim Boundary

Supported for accumulated ESMFold2 target-level evaluations across CAMEO/CAMEO follow-up, external RCSB, fresh general RCSB, and viral RCSB panels. MSA depth was claim-valid in CAMEO but not globally comparable across later panels.

Source Provenance

partial_explicit
Model
https://github.com/Biohub/esm#esmfold2
Weights / Scores
biohub/ESMFold2
Execution
fresh_structure_inference_for saved target-result panels; consolidated final bundle reuses those scored outputs
Data
  • CAMEO 2025 protein chains
  • RCSB PDB recent protein-chain panels
Metadata
  • CAMEO metadata and precomputed MSA fields
  • RCSB PDB metadata
Known provenance gaps
  • Legacy ESMFold2 runs did not record the exact Hugging Face model cache revision/hash or original online download command. Future ESMFold2 entries must record this before target scoring.
  • The final bundle consolidates multiple scored source runs; some RCSB candidate-generation scripts and raw downloaded structures remain in source roots rather than tracked MCP artifacts.
Public downloads

Limitations

  • Global table mixes scaled ESMFold2 inference with a smaller earlier CAMEO/default subset.
  • Cluster-aware validation used approximate k-mer clustering; publication could replace this with MMseqs2 or CD-HIT.
  • Class-II E/E2 is metadata-defined and localized, not a global all-protein sequence axis.

Downgraded Axes

  • MSA depth: CAMEO-claim-valid but not global
  • flexible/non-helical composition: CAMEO-local, not broad portable
  • reference geometry/topology: invalid as deployment axis
Key MCP artifact IDs
  • esmfold2:FINAL_ESMFOLD2_RESULT.md
  • esmfold2:ESMFOLD2_SPECTRA_ANALYSIS_TRACE.md
  • esmfold2:data:primary_exact_sequence_dedup.csv
  • esmfold2:data:all_esmfold2_scored_evaluations.csv
  • esmfold2:tables:global_hypothesis_effects_primary_sequence.csv
  • esmfold2:tables:global_hypothesis_adjusted_effects_primary_sequence.csv
  • esmfold2:tables:global_hypothesis_level_curves_primary_sequence.csv
  • esmfold2:tables:per_source_high_disulfide_summary.csv
  • and 6 more key artifacts in the MCP store

STATE

state_pathway_module_train_support_20260603

valid current pool confirmed

STATE generalizes worse for Replogle held-out gene perturbations whose pathways/modules are sparsely represented among training perturbations. The effect was discovered in hepg2 and k562, confirmed in jurkat and rpe1, checked in Replogle zero-shot, and extended beyond Reactome to independent gene-set systems.

Primary Axis pathway/module train-support density

For each held-out gene perturbation, support is the mean log(1 + number of same-context training genes sharing each pathway/module annotation of the target gene). Higher support means the target gene is closer to known training biology in that context.

Scored rows39,808
Few-shot HVG gap0.128
Few-shot SE gap0.109
Zero-shot HVG gap0.068
Zero-shot SE gap0.044
OLS support coefficient0.028

Claim Boundary

Current-pool confirmed using official precomputed STATE scored CSVs and official split definitions, not fresh STATE inference or raw single-cell matrices. The main reviewer-facing gap is a fixed baseline/raw-label rerun.

Source Provenance

not_applicable_precomputed
Model
https://huggingface.co/arcinstitute
Weights / Scores
arcinstitute/ST-HVG-Tahoe, arcinstitute/ST-SE-Tahoe, arcinstitute/ST-HVG-Replogle, arcinstitute/ST-SE-Replogle, arcinstitute/ST-HVG-Parse, arcinstitute/ST-SE-Parse
Execution
official_precomputed_scores_loaded; no fresh STATE inference in this continuation run
Data
  • Tahoe-100M official STATE scored outputs
  • Replogle-Nadig official STATE scored outputs
  • Parse-PBMC official STATE scored outputs
Metadata
  • Official STATE split TOMLs
  • HGNC complete set
  • Reactome Ensembl2Reactome
  • GO, GOA, WikiPathways, and KEGG public resources
Known provenance gaps
  • This finding did not run fresh STATE inference; it is intentionally based on official precomputed scored CSVs.
  • The full Hugging Face file manifest from the prior source run is not packaged as a tracked artifact in this repository; compact repo-level provenance is recorded here.
  • Exact HF snapshot revisions for each STATE model repository should be backfilled before using this as a fully reproducible checkpoint-download recipe.
Public downloads

Limitations

  • Used official precomputed scored CSVs rather than fresh STATE inference.
  • Fixed baseline/raw-label rerun still needed for reviewer-facing closure.
  • Perturbation strength measured by DE-count severity is a strong mediator/confound and should remain in claim framing.

Downgraded Axes

  • Tahoe target-context chemical Tanimoto support: split-valid but negative/weak
  • Parse-PBMC ligand-family training support: reversed direction
  • GO Molecular Function support: positive but weaker
Key MCP artifact IDs
  • state:state_spectra_report.md
  • state:target_model_results.csv
  • state:spectra_loop_state.json
  • state:axis_ledger_prior.json
  • state:artifacts:spc_replogle_best_refined_axis.csv
  • state:artifacts:spc_replogle_best_refined_axis_trends.csv
  • state:artifacts:spc_replogle_best_refined_axis_context_trends.csv
  • state:artifacts:spc_replogle_zeroshot_reactome_support_confirmation.csv
  • and 6 more key artifacts in the MCP store

Agent Access

Connect Claude Code or another MCP client

The server uses streamable HTTP and exposes read-only tools for listing models, retrieving findings, reading protocol sections, fetching text previews, retrieving model/data provenance, and returning public download URLs.

{
  "mcpServers": {
    "spectra": {
      "type": "http",
      "url": "https://spectra.yashaektefaie.com/mcp"
    }
  }
}