Methodological red-team · framework v0.8

The protocol was transparent. Now it is asking the right questions.

This pass attacks the review before outsiders do. It separates causal comparison from rare-event surveillance, promotes all-cause serious events over subjective attribution, and prevents “not reported” from becoming zero.

Corrected2 products

Comparative effect and monitored-exposure safety are no longer one analysis.

Reordered12 outcomes

All-cause serious events lead; attribution-based events are secondary.

Expanded2 searches

A focused depression search plus broad safety-capture search await PRESS review.

Still blocked5 human gates

Information specialist, statistician, PPI, reviewers, and registration.

01 · The core correction

One review becomes two linked evidence products.

Randomized trials answer whether assignment changes risk. A wider set of prospectively monitored administrations can help detect rare events, but cannot inherit the same causal interpretation.

Comparative

Randomized safety effect

  • Direct MDD randomized trials
  • Compatible comparator and time window
  • Arm-level risk and risk difference
  • RoB 2 and outcome-level GRADE

Can say: whether assignment to psilocybin changed a prespecified harm relative to the declared control.

Descriptive

Monitored-exposure safety inventory

  • RCT intervention arms plus prospective monitored cohorts
  • Explicit dosed denominator
  • Stratified by population, design, sponsor, and ascertainment
  • No causal comparison

Can say: what was observed after the studied monitored administrations and how uncertain or incomplete that observation is.

02 · Red-team repairs

The strongest fixes are about meaning, not more decimals.

Outcome attribution

All-cause first

Before
Drug-related SAE led the result.
Now
All-cause SAE is primary; attribution is secondary.

Investigator attribution can be affected by unblinding and reporting practice.

Missingness

Zero must be proven

Before
Blank and zero could look similar.
Now
Confirmed zero, numeric unavailable, mentioned only, and not reported are distinct.

A study that never reports an outcome cannot contribute a zero.

Harms taxonomy

Serious is not severe

Severe
Intensity of an event.
Serious
Regulatory consequence or medical importance.

Both are stored independently, preserving the source definition.

Review independence

Conceal decisions, not reality

Overclaim
Reviewers are “blinded.”
Defensible
Author decisions remain concealed until first-pass locks.

Reviewers may already know prominent public trials and results.

Search recall

No harms filter

Harms are inconsistently indexed. The protocol now adds a broad psilocybin-only safety search rather than requiring an adverse-event word in the title or abstract.

Human meaning

Participant/public content review

People affected by the outcomes review prolonged distress, functional impairment, rescue procedures, and public misinterpretation before registration.

Analysis governance

Statistician lock

The sparse-data model and pooling rules must be signed before independent extraction results are unmasked.

AI governance

Assistance disclosed

AI helped draft code and methods. It cannot count as an independent reviewer, information specialist, statistician, or accountable investigator.

03 · Outcome hierarchy

The new taxonomy prevents incompatible harms from collapsing into one label.

PriorityOutcomeWhy it is separateRequired evidence
PrimaryAll-cause serious adverse eventDoes not depend on subjective drug attribution.Explicit seriousness definition, participant count, denominator, window.
PrimaryDeath; transfer/hospitalization; ICU/organ supportObjective consequences should not be hidden inside a composite.Verified event and exact window.
PrimarySevere psychiatric/behavioral event requiring interventionCaptures clinically important panic, mania, psychosis, dissociation, or agitation.Symptom definition plus rescue, observation, restraint, transfer, or admission.
PrimarySelf-harm or clinically important worseningIdeation, behavior, and intervention are not interchangeable.Instrument, threshold, event type, and intervention.
SecondaryDrug-related serious eventUseful, but attribution is a judgment and may be biased.Who attributed, using what rule, with what source data.
SecondaryRescue medication and extended observationProcess outcomes can reveal important distress without meeting SAE definitions.Medication, indication, duration, and planned-vs-actual observation.

Download all 12 outcome definitions →

04 · Remaining gates

The package is harder to fool, but it still needs people.

Information-specialist review

PRESS-review the focused and broad searches, including database translations.

Statistical sign-off

Freeze sparse-data, zero-event, pooling, and missing-data rules before results are exposed.

Participant/public content review

Review relevance, omitted harms, interpretation boundaries, and public-facing language.

Independent review team

Assign two reviewers and an adjudicator with completed independence and conflict forms.

Prospective registration and search execution

Time-stamp the human-approved protocol, then run, archive, deduplicate, and hash the registered corpus.

The 13-record horizon corpus is no longer treated as complete for the expanded protocol.

It was built around randomized depression candidates. Product B requires a new broad safety-capture search. The project will not patch that gap by hand-picking a few extra favorable or alarming studies.

05 · Research files

Every material change is inspectable.

Method sources

Built against established harms-review standards.

Cochrane Chapter 19: Adverse effects; PRISMA-Harms; CONSORT Harms 2022; PRESS 2015; and the official RoB 2 resources.