At FDA/AACR Liquid Biopsy Meeting, Researchers Detail Concordance Challenges, Standards

EW YORK (GenomeWeb) – Stakeholders in the liquid biopsy field gathered yesterday to discuss the state of the science for blood-based cancer genomics, to quantify the challenges facing test developers and regulators, and to discuss ongoing projects and new data.

The meeting, which was webcast, was the second of its kind co-sponsored by the US Food and Drug Administration and the American Association for Cancer Research — designed to bring together diagnostic firms, clinical researchers, FDA officials, and drugmakers in a discussion of technology that is advancing quickly but also presents unique regulatory challenges in the establishment and adjudication of test validity.

The prior workshop, which took place in July 2016, centered on the recent approval of Roche's blood-based cobas EGFR Mutation Test as a companion diagnostic for Genentech's Tarceva (erlotinib).

Though it was the first liquid biopsy test approved by the FDA, clinicians and competitors have since begun to raise questions about whether the cobas assay is the best test to use in clinical practice, or if other laboratory-developed tests might be superior. Geoffrey Oxnard and colleagues at Dana Farber have embraced droplet digital PCR, for example, while others at the annual meeting of the American Society for Clinical Oncology meeting this year reported adopting next-gen sequencing tests, like the one developed by Inivata.

Numerous speakers at the follow-up workshop this week addressed issues for companies seeking to develop alternative tests, whether PCR- or sequencing-based, to the Cobas assay.

Carl Barrett, AstraZeneca's VP of Oncology Translational Sciences, for example, reported that since working with Roche to move the cobas assay through CDx approval, it has collected new data that not only demonstrates that NGS testing and digital PCR approaches can closely match cobas, but that some appear superior.

In a shift that reflects the rapid advance of these technologies, the meeting this week also focused on the use of multi-gene NGS panels as opposed to PCR, as well as on moving from diagnosis in advanced cancer to using tests in disease monitoring, for measuring surrogate endpoint markers in drug development, and in early cancer detection.

Liquid biopsy bakeoff

A highlight of the day-long workshop was the second half of Barrett's presentation, which featured data from a head-to-head evaluation the company performed comparing four commercial NGS liquid biopsy tests, which Barrett said the firm had agreed not to name.

Barrett said that the analysis, which AstraZeneca called a "PlamsaSeq Bakeofff," was prompted both by the drugmaker's own observations, and by outside reports like the one published by University of Washington researcher Tony Blau and colleagues in JAMA Oncology last year.

The UW team's results — a "rather disturbing observation," according to Barrett — demonstrated significant discordance between the mutation calls from Foundation Medicine's FoundationOne tissue sequencing test and Guardant Health's Guardant360 liquid biopsy assay, when applied to samples from the same patients.

Though the study was small, involving just nine subjects, the authors concluded that only 22 percent of detected mutations were reported concordantly by the two companies.

"Companies doing this say that its tumor heterogeneity that is responsible [for discordance between liquid biopsy and tissue]," Barrett said. "But there are also technical reasons … which is what we tried to sort out."

In its bakeoff, AstraZeneca sent samples from 24 cancer patients to the four commercial vendors, comparing their results to each other and to normal tissue and cancer tissue sequencing data.

Each firm extracted DNA according to their own protocols and returned data to AstraZeneca, including not only mutation calls, but also QC and raw data files — a major accomplishment according to Barrett, considering many commercial testing firms' reluctance to reveal proprietary information.

The company defined true positives as cases for which any one particular plasma test matched the tissue sequencing results, or in which two plasma tests were concordant, regardless of the tissue data.

False positive calls were those made by only a single blood-based vendor but not seen in the tumor tissue data. And false negatives were missed by at least one liquid biopsy test, but reported in tissue and at least one blood-based assay.

Overall, Barrett argued, the concordance results indicate that tumor heterogeneity falls short of explaining all discordance between liquid biopsy and tissue sequencing.

One takeaway from the results, Barrett said, is that while false positives occurred mainly below 1 percent allele fractions, true positives ranged from very high AF to below 0.2 percent across the assays. This unfortunately precludes the possibility of setting a cutoff point above the majority of false positives as a way to increase specificity.

Another interesting finding was that many of the false positives that occurred at low allele fractions appeared to be novel, meaning they were absent in public databases and were often characterized by suspicious sequence changes.

Sensitivity of the four vendors ranged from 40 percent to 90 percent, according to the company's definitions of true and false results. Their positive predictive value ranged from 50 percent to 89 percent.

Interestingly, when the researchers separated the calls by allele fraction, the sensitivity dropped for three vendors below 1 percent AF compared to above, while the fourth, vendor C, actually improved. Across the board, however, PPV was lower below 1 percent AF than above.

Having the raw data for each assay allowed the AstraZeneca team to further investigate specific discordance cases. Barrett said many of these examples illustrate why the results contradict vendors' assertions that discordance largely follows from tumor heterogeneity.

One case of a false negative PIK3CA mutation was called by Vendor C but not Vendor B.  Examining the data, Vendor C's assay had picked up five reads of the variant allele out of about 2,000 total, yielding an allele fraction of about .2 percent. For Vendor B, there were only four reads, and that company's bioinformatics cutoffs deemed that a negative call.

A false negative call by Vendor C was due to the company's designation of a BRCA2 mutation present at 17 percent allele frequency as a germline rather than a somatic variant.

In another case, Vendor D missed a TP53 alteration that Vendor C reported.  The data showed that the mutation was present at .4 percent AF for vendor C — in six reads out of about 1,400. Vendor D sequenced at much higher depth — more than 42,000x — and the noise created by that camouflaged the alteration.

Barrett added that when AstraZeneca looked at novel mutation calls in the false positive cases, the fraction of T-to-A and A-to-T base changes was disproportionately high — "a red flag for sequencing errors," he explained.

Variability wasn't just an issue between the test vendors. When the drugmaker sent triplicate samples to a single vendor to look at reproducibility, only two of seven cases received the same report back for each of the three replicate samples.

This doesn't mean that tumor heterogeneity doesn't contribute to cases where blood-based tests identified a mutation that tissue sequencing did not, Barrett added. The company's evaluation included a case where there was an EGFR alteration seen in two liquid biopsy vendors' results but not in tumor tissue, suggesting that heterogeneity played a role there.

In a second case, a variant was present at 0.5 percent AF in tumor tissue. "We would never call that," Barrett said. But plasma results from two vendors confirmed the alteration.

Overall, Barrett said that AstraZeneca's takeaway has been that NGS enables selection for well-qualified markers like EGFR, KRAS, and AKT alterations. But existing commercial panels are clearly variable in performance and are most reliable over 1 percent AF. Below that level, recipients of these reports should use caution, and looking at the raw data can help. If a call seems suspicious, it helps to consider mutation bias and whether the alteration in question appears to be a novel variant, said Barrett.

Taking aim at standards

Following Barrett's presentation, Mickey Williams, director of the Molecular Characterization Laboratory at the Frederick National Lab, debuted an effort he is now leading as part of the Foundation of the National Institutes of Health-sponsored ctDNA Reference Material Working Group. The working group has been greenlighted as of about a month ago, Williams said, to develop and validate reference materials for ctDNA test development and quality control.

As liquid biopsy genetic tests have quickly found their way into clinical practice, the field has begun to pay increasing attention to the lack of tools to ensure assays are validated according to common standards. Last fall, for example, a consortium called the Blood Profiling Atlas in Cancer (BloodPAC) launched with the aim of aggregating and harmonizing data on current liquid biopsy tests and on standardizing protocols for sample collection, preparation, and analysis for future test development.

Commercial firms have also begun to highlight new reference products, recognizing the growing appetite for these tools.

"Good clinical specimens are hard to get a hand on, so reproducibility studies and platform comparisons are very difficult," Williams explained.

"As [Barrett] was showing, it, takes a lot of effort … and it's obvious that we need some sort of reference material. … So that's the endgame [of our project], to deliver a set of reference samples, [because] our concern is that if we don't get this into [users'] hands soon, we could do as much harm as good when studies begin to conflict with each other."

Contrived samples can't take the place of real clinical specimens, Williams added. However, the FDA has recognized that liquid biopsy presents a particularly challenging arena for validation, and as long as synthetic samples are well proven to be commutable, or comparable to clinical samples, they can have a place in the validation process.

The FNIH project will be focused on creating standardizes materials to support the steps from extraction through analysis, excluding pre-analytical aspects like blood collection tubes, etc.

The team is working with a list of about 20 variants, and has brought three commercial companies —  Horizon Discovery, SeraCare, and Thermo Fisher Scientific — to the table as contributors.

The project will use three labs to validate the reference materials: the molecular characterization lab at Frederick National Laboratory, a lab at the National Institute of Standards and Technology, and another at Harvard. These labs will use at least two different NGS assay, hopefully one amplicon-based and one probe-based, and also droplet digital PCR.

In phase II of the project, Williams said the group will conduct a clinical pilot using sample replicates at several different dilutions, and are hoping for at least 10 labs to participate, representing a mix of commercial companies and academic centers.

"It will be blinded specimens," Williams said, "and we are planning to do what [AstraZeneca] did — to pull data in and look at how well everyone is doing."