EndoTwin-W Receptivity Benchmark v1.0

An open, standardized, reproducible benchmark for computational models of human endometrial cell-state and pathway behavior. It ships fixed public test datasets, three null baselines, and a scoring harness. It is deliberately honest: the reference model (EndoTwin-W) does not win every metric — the benchmark is built to show where mechanistic modeling adds value and where simple baselines are already competitive.

Status: research use only. Community submissions welcome.

1. Task

Given systemic hormone context (and, where available, condition labels), predict endometrial cell-state activity across 17 states and the direction and relative magnitude of pathway disruption for a sample, scored against independent public transcriptomic data.

Two scored sub-tasks

• T1 — cell-state recovery: per-state Spearman correlation between predicted and reference cell-state activity across the menstrual cycle.
• T2 — condition disruption: Spearman correlation and direction concordance between predicted and observed pathway/gene disruption for disease conditions.

2. Test datasets (public, fixed)

All datasets are publicly available from NCBI GEO. Reference labels and the held-out split are fixed and recorded with a SHA-256 manifest.

3. Metrics

• Per-state Spearman r (T1) — mean and per-state.
• Balanced classification accuracy for cycle-phase assignment (fair under class imbalance).
• Direction concordance (T2) — fraction of gene-condition pairs with correct sign.
• Beats-all-nulls count — states where a model significantly exceeds every null baseline.

4. Baselines (null models)

Three non-mechanistic baselines are provided so any new model can be judged against simple alternatives:

• Majority — always predict the dominant cycle phase.
• Ordinal — predict by cycle-day ordering only.
• Gaussian — smooth cyclic prior with no mechanism.

5. Leaderboard v1.0

An honest, mixed result: null baselines lead on mean per-state correlation, while the mechanistic model leads on balanced cycle-phase accuracy and uniquely beats all nulls on 7 of 17 states.

17 cell-states scored. Frozen results: benchmarks/leaderboard_v1.0.json.

6. How to submit

Produce predictions in the submission format — one row per (state, sample) with your predicted value alongside the reference:

state,sample_id,predicted,reference
epithelial_mid_secretory,S1,0.82,0.79
stromal_decidualized,S1,0.71,0.68
...

1. Clone the repository and read benchmarks/BENCHMARK.md for the full protocol and scoring harness.
2. Fill in benchmarks/submission_template.csv with your model’s predictions on the fixed test split.
3. Run the scoring harness to compute T1/T2 metrics against the frozen references.
4. Open a pull request on GitHub adding your scored entry to the leaderboard, or email the maintainers.

Repository: github.com/goyal74/EndoTwin_W · benchmark spec, baselines, and scoring harness under benchmarks/.

7. Reproducibility

The task, test split, reference labels, and baselines are version-frozen. Every leaderboard number is regenerable from public GEO data plus the harness in the repository — no private data is required to reproduce or to submit. This is a v1.0 open benchmark; the split and metrics are fixed for v1.0 and will be versioned if revised.