Files
Shanghua Gao 1b02920da6 Restructure the case-studies README as input, process, output (#541)
* Rewrite the case-studies README around the science, not the verification

The README read like a test report. Its first substantive sentence was about
matching published values, it carried three Published-vs-Observed tables, and
the longest section was forensics on a ChEMBL outage. Someone arriving wants to
run a scientific investigation, not audit a regression run.

Restructure around what each case actually does: the question, where the
evidence turns, and what the workflow concludes when two valid results point
opposite ways. Each case now shows the pivotal moment as real output — the
DepMap dependency that reframes BLM, the mechanism check that disqualifies
nolasiban, the adjusted odds ratio and what it does not support.

The verification material stays, but as a short "Reading the output" section:
how the published: annotations work, which values are expected to drift and
which are not, and the ChEMBL outage in three lines instead of thirty.

233 lines to 151. Every number shown is copied from an actual run.

* Restructure the case-studies README as input, process, output

The previous rewrite told each case as a narrative, which read better than the
test report it replaced but still left the mechanics implicit: what you hand
the script, what it does with it, what comes back.

Give every case the same three headings. Input states the identifier or the
data files and the question. Process is a table of steps with the tools each
one calls and what it establishes, with the pivotal step marked. Output is the
verdict, with the actual printed result above it.

The header carries a diagram of that shape and a summary table, so the pattern
is visible before reading any single case.

Step and tool counts in the tables are extracted from the scripts rather than
written by hand: 7 steps/15 tools, 6 steps/15 tools, 3 steps/1 tool in three
modes. Every value quoted comes from a recorded run.

* Stop pinning Case 2 to an unstable affinity value

ChEMBL came back up, so the two steps that had never run against the live
service finally did. Case 1's analog comparison reproduced exactly
(DILI 0.987-0.992). Case 2's affinity lookup did not: it reported 0.5 nM
against a published "Ki as low as 3.2 nM".

The cause is not the data. ChEMBL returns activities unsorted, so the minimum
over one page depends on which page you fetch: 0.50 nM at offset 0, 1.0 at
offset 100, 0.09 at offset 200. Annotating that number against one published
value makes the case appear to fail at random.

Report what actually reproduces instead — that the receptor has nanomolar
ligands — and label the sampled minimum as page-dependent rather than as a
value being matched. The README's ChEMBL note is updated too: it claimed both
steps were unconfirmed, which is no longer true.

* Find the real lowest Ki for OXTR instead of sampling one page

Case 2 reported 0.5 nM where the write-up says "Ki as low as 3.2 nM". Neither
is the lowest measured affinity. ChEMBL returns activities unsorted and
paginated, so the minimum over any single page is an artefact of which page was
fetched: 0.50 nM at offset 0, 1.0 at offset 100, 0.09 at offset 200.

ChEMBL holds 812 Ki values in nM for CHEMBL2049. The actual minimum is
0.025 nM (CHEMBL2037514, no validity flag, not a duplicate), and 182 of the 812
sit below the published 3.2 nM. The published figure is understated by roughly
130x because it too came from a partial sample.

Page through the whole set rather than one page, restrict to Ki in nM, drop
rows carrying a data_validity_comment, and report the count alongside the
minimum so the figure is interpretable. The case's conclusion is unchanged, and
better supported: OXTR has well-characterised high-affinity ligands.
2026-08-19 18:24:34 -04:00
..

Three research questions, answered end to end

Each script takes a research question, chains tools across databases, structures, screens and predictive models to answer it, and prints the evidence as it arrives. This is the same sequence the AI scientist ran in the write-up.

Every case has the same shape:

input                    process                              output
─────────────────────    ──────────────────────────────────   ──────────────────
a question + an          N steps, each one or more tool       a verdict, with
identifier or a file     calls, each step's result feeding    every number
                         the next                             traceable to a call
Input Process Output
1 gene BLM 7 steps, 15 tools is it a target, and is ML216 developable?
2 gene OXTR 6 steps, 15 tools is the chemistry right for the autism hypothesis?
3 2 trial tables (CSV) 3 steps, 1 tool in 3 modes does BCG associate with worse adverse events?

Setup

python -m venv .venv && source .venv/bin/activate    # or: uv venv .venv
pip install 'tooluniverse[ml]'                       # ml extra = ADMET-AI models

git clone https://github.com/mims-harvard/ToolUniverse.git
cd ToolUniverse/examples/case_studies
pip install -r requirements.txt

No API keys. Every database used — Open Targets, ClinVar, GeneBe, AlphaFold, PDBe, Europe PMC, PubChem, ChEMBL — is open, and ADMET-AI runs locally.

python case1_blm_target_assessment.py     # ~2 min, mostly ADMET-AI model load
python case2_oxtr_druggability.py         # ~2 min

python download_bcg_data.py               # ~4 MB of trial tables from Zenodo, CC0
python case3_bcg_ae_severity.py           # seconds

Case 1 — is BLM a target?

Input

gene:     BLM          (a DNA-repair helicase; losing it causes cancer)
question: can blocking it nevertheless treat cancer,
          and is its inhibitor ML216 developable?

Process

Step Tools What it establishes
1 Resolve the gene OpenTargets_multi_entity_search_by_query_string OpenTargets_get_target_gene_ontology_by_ensemblID it is a genome-maintenance helicase
2 Cancer-risk genetics OpenTargets_get_diseases_phenotypes_by_target_ensembl 464 disease associations, led by Bloom syndrome
3 Classify a variant ClinVar_search_variants GeneBe_classify_variant c.520C>T is pathogenic — evidence against inhibiting
4 Dependency screen OpenTargets_get_target_depmap_essentiality the turn: not pan-essential, but concentrated in specific tumours
5 Structure alphafold_get_summary PDBe_get_uniprot_structure_coverage use experimental structures, not the mixed-confidence prediction
6 Precedent EuropePMC_search_articles paralog WRN is already a validated synthetic-lethal target
7 Triage the chemistry OpenTargets_get_chemical_probes_by_target_ensemblID PubChem_* ADMETAI_* ChEMBL_search_similar_molecules ML216 is drug-like but carries a liability

Step 4 is why the case exists — it contradicts step 3:

  mean gene effect: -0.18
  lines dependent at < -0.5: 94/1258 (7.5%)
  Most-dependent cancer types (mean gene effect, n>=5 lines):
    Mature T and NK Neoplasms          -0.47  (n=8)
    Cutaneous Squamous Cell Carcinoma  -0.44  (n=5)

Output

Not a target — a biomarker-defined hypothesis. The germline genetics say protect BLM; the screen says it is a selective vulnerability in defined contexts. Both are right, so the workflow returns the contexts worth testing rather than a yes/no. ML216 is a probe, not a lead: its predicted liver-injury signal stays near maximum across every close analog, implicating the scaffold rather than the molecule.


Case 2 — does OXTR chemistry fit the hypothesis?

Input

gene:     OXTR         (oxytocin receptor, a GPCR)
question: is it druggable, and does its pharmacology offer a
          starting point for the emerging autism association?

Process

Step Tools What it establishes
1 Resolve + structure OpenTargets_multi_entity_search_by_query_string OpenTargets_get_target_gene_ontology_by_ensemblID alphafold_get_summary PDBe_get_uniprot_structure_coverage active and inactive experimental structures both exist
2 Disease landscape OpenTargets_get_diseases_phenotypes_by_target_ensembl reproductive indications, plus an emerging autism link
3 Tractability + drugs OpenTargets_get_target_tractability_by_ensemblID OpenTargets_get_associated_drugs_by_target_ensemblID druggable GPCR, 9 known agents, approved ones are peptides
4 Mechanism OpenTargets_get_drug_mechanisms_of_action_by_chemblId already drugged in both directions
5 Measured affinity ChEMBL_search_targets ChEMBL_get_target_activities nanomolar ligands exist
6 CNS + mechanism EuropePMC_search_articles PubChem_* ADMETAI_predict_BBB_penetrance OpenTargets_get_drug_mechanisms_of_action_by_chemblId the turn: the brain-penetrant candidate acts the wrong way

Nolasiban passes every property filter — then fails the one that matters:

  nolasiban action type: ['ANTAGONIST']
  Read: brain-penetrant non-peptide OXTR chemistry is attainable,
  but this chemotype BLOCKS the receptor, and the pro-social
  hypothesis needs it ACTIVATED. Wrong direction.

Output

Tractable target, wrong chemotype. The objective is stated precisely instead of a repurposing suggestion: a brain-penetrant agonist or positive allosteric modulator, selective over the vasopressin receptors. Screening on affinity and brain penetrance alone would have produced a confident, wrong recommendation.


Case 3 — did BCG worsen adverse events?

Unlike the others, this one computes on raw data rather than querying databases.

Input

data:     TASK008_BCG-CORONA_DM.csv   demographics, 1000 subjects
          TASK008_BCG-CORONA_AE.csv   adverse events, 2694 records
          (public BCG-CORONA trial, Zenodo 12737228, CC0 — download_bcg_data.py fetches it)
question: is BCG associated with higher adverse-event severity,
          after adjusting for how often participants saw patients?

Process

One tool, clinical_trial_ae_severity_test, in three modes:

Step Mode What it does
1 prepare inner-join AE onto DM by USUBJID, reduce each subject to their max AESEV → 791 evaluable subjects
2 chi-square unadjusted treatment × severity association
3 ordinal proportional-odds regression adjusting for patients_seen, expect_interact, work_hours

Output

  evaluable subjects: 791
  chi-square: 10.12   dof: 3   p-value: 0.018
  OR for higher severity with BCG: 1.53
  95% CI: (1.16, 2.01)
  p-value: 0.0024

A significant association that survives adjustment. Two things the script is careful about, and you should be too:

  • Direction. The tool reports the odds ratio against its own reference level (BCG=0, Placebo=1 alphabetically), so the raw 0.65 is Placebo-vs-BCG and the 1.53 above is its reciprocal. Reporting the raw number as the BCG effect would invert the finding.
  • What it supports. The cohort is restricted to participants with at least one adverse event, and severity is a derived per-subject maximum. That makes this a reproducible association from a secondary analysis — not evidence that vaccination changes severity.

Reading the output

Every value prints with the originally reported number beside it:

  DILI (liver injury): 0.988   (published: 0.99)

So a run tells you at a glance whether the science still holds against live databases, and exactly which number moved if it doesn't. Values marked published: come from the write-up or from the supplementary note it condenses (arXiv:2509.23426) — the note carries the intermediate ones such as ACMG scores, PDB accessions and per-context DepMap effects.

Expect some movement, and know which kind matters:

  • Live databases get re-scored. OXTR disease associations were 393 at the time of writing and read 465 today; Open Targets re-scores every release. That is a release difference, not a failed reproduction, and no conclusion rests on the count.
  • Case 3 should never move. Frozen dataset, deterministic model, identical on pandas 2 and 3. If its numbers change, something in your environment did.
  • One published number is understated. The write-up reports OXTR Ki "as low as 3.2 nM". ChEMBL returns activities unsorted and paginated, so a single page gives an arbitrary minimum — 0.50, 1.0 and 0.09 nM at offsets 0, 100 and 200. Case 2 now pages through all 811 Ki measurements and finds the real figure, 0.025 nM, about 130× lower. The conclusion the case draws from it, that OXTR has well-characterised high-affinity ligands, is unaffected — if anything it is stronger.
  • ChEMBL's data API goes down periodically (HTTP 500 upstream, known issue). When it does, the two steps that need it print [unavailable] and the case carries on rather than failing. Everything else is unaffected.