Benchmarking

Overview

Resources

Group Logo

The Benchmarking Task Team is working to develop benchmarking toolkits for variant calls. The initial focus of the team has been developing standardized definitions of performance metrics, as well as tools to calculate and stratify these performance metrics for human germline small variants, including SNPs, small indels, and complex variants. Our work includes developing a sophisticated tool to compare variant call files, including handling of different representations of complex variants (presentation). The team has also been developing standardized definitions for performance metrics and standardized ways to stratify performance into different categories (link), including some standard bed files of genome regions for stratification (e.g., GC content, low complexity and repetitive regions, segmental duplications, and functional regions (link). This work is closely aligned with the Genome in a Bottle Consortium, which has been developing high-confidence variant call sets for benchmarking variant calls in human genomes.
Authors of 3 variant comparison tools that handle complex variant comparisons are involved in the Team:

Future work may include developing benchmarking toolkits for structural variants, somatic mutations, RNA-sequencing, and variant annotations.

Leadership

  • Justin  Zook Justin Zook
    • National Institute of Standards and Technology
    • Gaithersburg, United States
    • Lead, Group

Coordinator

Team

  • Justin Zook , National Institute of Standards and Technology , Gaithersburg, United States
  • Lee Lichtenstein , Broad Institute , Cambridge, United States
  • John Browning
  • Brad Chapman , Harvard School of Public Health , Boston, United States
  • Bob Davison
  • Alpha Diallo , DNAnexus , Mountain View, United States
  • Mark Fleharty
  • Mark Fleharty
  • Kaushik Ghose , SB Genomics , Cambridge, United States
  • David Heckerman , Microsoft Genomics
  • Kevin Jacobs
  • Andrew Jesitis , Golden Helix , Bozeman, United States
  • Andrea Komoroczi
  • Peter Krusche , Illumina, Inc. , Camrbdige, United Kingdom
  • Jiayong Li
  • Yong Mao , Philips , Amsterdam, Netherlands
  • Marghoob Mohiyuddin , Bina Technologies Inc. , Redwood City, United States
  • John Mu , Bina Technologies Inc. , Redwood City, United States
  • Brendan OFallon , 23andMe , Mountain View, United States
  • Larsson Omberg , SAGE Base , Seattle, United States
  • Justin Paschall , EMBL-European Bioinformatics Institute , Hinxton, United Kingdom
  • Martin Pollard , Wellcome Trust Sanger Institute , Hinxton, United Kingdom
  • Goran Rakocevic , Seven Bridges Genomics, Inc. , London, United Kingdom
  • Gabe Rudy , Golden Helix , Bozeman, United States
  • Naina Thangaraj , DNAnexus , Mountain View, United States
  • Len Trigg
  • Rebecca Truty , Complete Genomics , Mountain View, United States