Evaluating eligibility criteria of oncology trials using real-world data and AI

Clinical trial curation

In this examine, we centered on aNSCLC, as a result of aNSCLC is a prevalent most cancers kind and has the most important quantity of sufferers within the Flatiron Health database. We systematically recognized all of the aNSCLC trials which might be accessible for our evaluation. A complete of 3,684 interventional scientific trials of NSCLC have been retrieved from the ClinicalTrials.gov web site of the National Library of Medicine (queried on 8 November 2019). A scientific choice of trials was carried out using the next filters: (1) trials have been interventional and solely had two arms; (2) therapies consisted of medicine or biologicals solely; (3) the medicine chosen in every arm are advisable for aNSCLC as listed on the NIH web site (https://www.cancer.gov/about-cancer/treatment/drugs/lung); (4) at the least 250 sufferers in every arm have been discovered within the Flatiron Health dataset who match the outline of the sufferers within the trials; (5) the trial was carried out in section III; and (6) protocols have been accessible. The remaining checklist of chosen aNSCLC trials included FLAURA²⁹, LUX8³⁰, Checkmate017³¹, Checkmate057³², Checkmate078³³, Keynote010³⁴, Keynote189³⁵, Keynote407³⁶, BEYOND³⁷ and OAK³⁸. Detailed info on these trials could be present in Extended Data Table 1. To make sure the completeness of the trial criteria, we rigorously extracted all of the eligibility guidelines immediately from the unique trial protocols quite than from ClinicalTrials.gov. The eligibility criteria have been extracted from the unique scientific trial protocol paperwork and the programmatic encoding of the criteria was verified by a crew of skilled oncology data scientists and scientific trial specialists. Additional details about the encoding of the criteria is supplied within the Supplementary Methods and Supplementary Discussion. Trial Pathfinder is a versatile framework that may be utilized to different scientific trials.

Flatiron Health dataset

The data that help the findings of this examine have been obtained by Flatiron Health, a nationwide EHR-derived de-identified database containing 219,312 sufferers with most cancers with a mean of 2.6 years of follow-up. The Flatiron data leveraged on this examine (the February 2020 data minimize) comes from a mixture of EHR-derived data and exterior industrial and US Social Security Death Index data. The Flatiron Health database is taken into account one of the trade’s main analysis databases in oncology owing to the rigorous data curation and abstraction processes in addition to publications by which their efforts to validate outcomes are demonstrated. In earlier validation research by which the Flatiron mortality data are in comparison with data from the gold-standard National Death Index, the sensitivity of mortality seize in a inhabitants of sufferers with aNSCLC was proven to be 91%, and that the impact of the remaining lacking deaths on survival analyses was minimal^39,40. In addition to curation accuracy, the Flatiron data are harmonized and aggregated throughout roughly 280 most cancers clinics throughout the nation, which permits its data to be extra consultant than the EHRs of a single healthcare centre. The majority of sufferers within the database originate from group oncology settings; relative group/educational proportions might differ relying on the examine cohort. Data supplied to investigators was de-identified and topic to obligations to stop re-identification and to guard the confidentiality of the sufferers. These de-identified data could also be made accessible upon request, and are topic to a licence settlement with Flatiron Health; researchers can contact DataEntry@flatiron.com to find out licensing phrases. Institutional Review Board approval with a waiver of knowledgeable consent was obtained earlier than the examine was carried out.

Flatiron Health takes a complete strategy to data curation, which entails the gathering of each structured and unstructured data from the EHRs. Structured data factors, reminiscent of laboratory take a look at outcomes, are harmonized throughout totally different EHRs and mapped into frequent terminologies. Unstructured data processing, reminiscent of data that come from clinician notes or biomarker stories, leverages technology-enabled abstraction. Through this course of, certified abstractors extract key data factors from unstructured paperwork and are aided by software program that facilitates this course of by means of group, looking and surfacing of key paperwork all through the abstraction course of. Flatiron’s community of abstractors contains licensed tumour registrars, oncology nurses and oncology scientific researchers.

Patients within the Flatiron Health community have been thought-about to be half of the aNSCLC real-world cohort in the event that they have been identified with lung most cancers (the ninth revision of the worldwide classification of illnesses (ICD-9) code 162.x; or the tenth revision of the worldwide classification of illnesses (ICD-10) code C34x or C39.9); had at the least two documented scientific visits on or after 1 January 2011; had pathology according to NSCLC; and have been identified with stage IIIB, IIIC, IVA or IVB NSCLC on or after 1 January 2011, or identified with early-stage NSCLC and subsequently developed recurrent or progressive illness on or after 1 January 2011. Patients have been excluded if there was an absence of related unstructured paperwork within the Flatiron Health database for evaluate by the abstraction crew.

A listing of the criteria that it was potential to emulate using the Flatiron Health database could be present in Supplementary Table 1. There are some criteria for which Flatiron Health doesn’t presently summary info from EHRs—for instance, reproductive well being, some prior co-morbidities, some earlier therapies, imaging procedures and outcomes—and these weren’t included within the current examine. For these criteria which might be accessible within the database, we additionally evaluated the share of lacking ECOG and laboratory worth info for every affected person initially of the primary or second line of remedy (Supplementary Table 38). To carefully mirror the precise trial screenings, we thought-about scientific measurements taken inside a window from 30 days earlier than to 7 days after the beginning of the road of remedy⁴⁰.

Data on hostile occasions

We additional help our findings by analysing toxicity data for a real-world cohort of 1,000 sufferers with aNSCLC from the Flatiron database. These sufferers have been randomly chosen from the broader aNSCLC cohort primarily based on receipt of anti-PD-1/PD-L1 remedy, and underwent extra data abstraction to find out the explanations for therapy discontinuation, together with toxicity. In addition, we recognized 22 Roche oncology trials with accessible scientific examine stories, and extracted statistics from the examine stories on the quantity of sufferers who withdrew from therapy owing to hostile occasions.

The Trial Pathfinder workflow

In step one of Trial Pathfinder—trial emulation—we recognized people within the real-world dataset who met the accessible eligibility criteria as initially revealed within the scientific trial protocol. The eligibility criteria have been encoded as logic statements and have been routinely utilized by our workflow. More info on how the semi-structured free-text criteria within the scientific trial protocols have been encoded into programmatic statements is supplied within the Supplementary Methods. Patients with lacking data factors (for instance, ECOG or laboratory values) within the corresponding criteria weren’t filtered by these criteria. We then assigned the chosen sufferers to the therapy teams that have been according to their therapy data within the database (for instance, atezolizumab versus docetaxel). To emulate the randomization and blind project within the trials, we used inverse likelihood of therapy weighting (IPTW) to regulate for baseline confounding components. Time zero was set to be the beginning of the corresponding line of remedy. Finally, we carried out survival evaluation for the emulated trials using the hazard ratio of the general survival as the result. Each particular person was adopted till the prevalence of loss of life or censored on the newest reported exercise. Outcomes that happen after 27 months within the Flatiron database are thought-about censored in our evaluation to match the unique trial settings. The outcomes are strong to the particular window lengths mentioned right here (Supplementary Table 39). The Trial Pathfinder open supply code was written in Python model 3.6.

Trial Pathfinder trial emulation and survival evaluation

To emulate the blind project and receive unbiased estimates of therapy results, we used IPTW to regulate for the baseline covariates. During the survival evaluation, affected person i is given the burden outlined in equation (1), by which Z_i is the indicator variable representing whether or not affected person i is handled or not, with Z_i = 1 indicating a handled case. The propensity rating e_i is outlined in equation (2), by which X_i denotes the baseline covariates. We used a logistic regression mannequin to estimate e_i. In our experiments of aNSCLC, the covariates X have been: age, gender, composite race or ethnicity, histology, smoking standing, staging, ECOG and biomarker standing, together with ALK, EGFR, PDL1, ROS1, KRAS and BRAF. Adjustment by propensity rating is efficient in balancing all of the covariates between the artificial therapy and management teams (Extended Data Fig. 3).

We additional carried out survival evaluation on the emulated trials. For every affected person, the index date or time zero, resembling the randomization level in a scientific trial, was chosen to be the beginning date of the road of remedy of that trial (both first or second). This selection of time zero ensures that there isn’t a immortal time bias⁴¹. Patients have been adopted till the prevalence of loss of life, censoring these sufferers and not using a loss of life occasion. The Cox proportional-hazards mannequin was used to compute hazard ratios and confidence intervals of total survival. Survival curves have been estimated with the Kaplan–Meier technique.

Eligibility criteria analysis with Shapley values

To consider the affect of a person criterion we used the Shapley worth, which is the common anticipated marginal contribution of including one criterion to the hazard ratio in any case potential mixtures of criteria have been thought-about. The Shapley worth has not too long ago been proposed in machine studying as a principled strategy to quantify the contribution of particular person options and data²⁸. The definition of the Shapley worth of the ith criterion is given in equation (3), by which n is the entire quantity of criteria and HR(S) signifies the hazard ratio computed when the criteria subset S is used to pick sufferers. The sum in equation (3) is taken over all potential subsets S of the n authentic criteria (denoted as N for brief) that didn’t comprise i.

The Shapley worth of the ith criterion is a weighted common of the impact of including this criterion to totally different subsets of inclusion/exclusion criteria. The weights normalize for the quantity of potential units which have the identical cardinality and are required to fulfill the Shapley attribution properties.

Exhaustively computing the hazard ratios of total survival for all potential subsets of criteria (order of n!) was computationally prohibitive. Here we estimated the Shapley worth by Monte Carlo sampling subsets of criteria S. The Monte Carlo sampling offers an unbiased estimate of the Shapley worth. Following the beforehand proposed algorithm⁴², we cease sampling when the Shapley estimate has converged (that’s, when the usual error of the Monte Carlo imply is lower than 0.001). In observe, convergence occurred after 100 iterations for every criterion. A number of thousand Monte Carlo samples mixed is enough for a trial with tens of criteria to judge. This makes Trial Pathfinder computationally environment friendly (Extended Data Fig. 4) and solely wants round half an hour to run with a single CPU for one trial. For every trial, we averaged its outcomes evaluating on a distinct criteria set from the trials in the identical line of remedy (both first or second). A Shapley worth bigger than zero signifies that the contribution of that criterion is to extend the hazard ratio on common. Conversely, a adverse Shapley worth signifies that the contribution of that criterion is to lower the hazard ratio on common. Finally, Shapley values which might be near zero correspond to a criterion that doesn’t have an effect on the hazard ratio.

Trial Pathfinder stories the subset of criteria utilized by the unique trial which have a Shapley worth smaller than Zero as data-driven criteria. Once the data-driven subset of criteria was chosen, Trial Pathfinder computed the quantity of eligible sufferers and the hazard ratio of the general survival between the artificial therapy and management arms.

Additional validation analyses

We stratified our 61,094 sufferers with aNSCLC from the Flatiron database by their geography of residence as within the US census—Northeast (n = 11,777), Midwest (n = 8,895), South (n = 23,895) and West (n = 9,061). We then evaluated the inclusion/exclusion criteria chosen by Trial Pathfinder for every of the 10 aNSCLC trials for sufferers from every geographical area individually (Supplementary Tables 22–25). We additionally stratified our aNSCLC cohort by their insurance coverage plan as an extra robustness evaluation—industrial well being plans (n = 22,423), Medicare (n = 10,841) and the remaining sufferers (n = 22,361). We evaluated our beforehand chosen inclusion/exclusion criteria for every of the 10 aNSCLC trials for sufferers underneath the three sorts of insurance policy individually (Supplementary Tables 26–28). We used the nationwide (US-based) de-identified Flatiron Health-Foundation Medicine aNSCLC clinicogenomic database (FH-FMI CGDB) for additional validation⁴³. Genomic alterations have been recognized by means of complete genomic profiling of greater than 300 cancer-related genes on the next-generation sequencing-based FoundationOne panel of the FMI⁴⁴. Retrospective longitudinal scientific data have been derived from EHR data from clinics within the Flatiron community, consisting of patient-level structured and unstructured data, curated by technology-enabled abstraction, and have been linked to genomic data derived from complete genomic profiling assessments of the FMI within the FH-FMI CGDB by de-identified and deterministic matching⁴³. To leverage the wealthy genomics info of FH-FMI CGDB, we added 17 extra genes to the adjustment of the covariates which have alterations in at the least 1,000 sufferers (Supplementary Table 31). For every of the 10 aNSCLC trials, we utilized the inclusion/exclusion criteria that Trial Pathfinder chosen on the Flatiron data and used it to emulate a trial using the FH-FMI CGDB cohort (Supplementary Table 30). Progression is used as the tip level and progression-free survival hazard ratios are computed.

Statistical evaluation

We bootstrapped the cohorts to estimate the usual deviations for the Shapley values. The confidence intervals for the hazard ratios have been estimated from the variance matrix of the coefficients in becoming the Cox proportional-hazards mannequin. For the security influence evaluation on 22 Roche oncology trials, we use two-sided P values from Fisher’s precise assessments to measure the distinction within the withdrawal ratio given two units of trials (Supplementary Table 35). When analysing toxicity data, we use two-sided P values from two-tailed Student’s t-tests to judge whether or not there’s a vital distinction within the baseline laboratory values between two toxicity teams (Extended Data Fig. 6).

Reporting abstract

Further info on analysis design is accessible within the Nature Research Reporting Summary linked to this paper.