softwarE DEVELOPMENT

We're building scalable software and platforms to enable efficient analysis of very large genetic data. All of our tools are open access and free for use by the scientific community.

hail

The widespread application of massively parallel sequencing for complex trait analysis offers unprecedented power to link genetics with disease risk. However, these projects pose substantial challenges of scale and complexity, making even trivial analytic tasks increasingly cumbersome.  To address these challenges we are actively developing Hail, an open-source framework for scalable genetic data analysis.

The foundation of Hail is infrastructure for representing and computing on genetic data.  This infrastructure builds on open-source distributed computing frameworks including Hadoop and Spark.  Hail achieves near-perfect scalability for many tasks and scales seamlessly to whole genome datasets of thousands of individuals.  On top of this infrastructure, we have implemented a suite of standard tools and analysis modules including: data import/export, quality control (QC), analysis of population structure, and methods for performing both common and rare variant association. Simultaneously, we and other groups are using Hail to manage the engineering details of distributed computation in order to develop and deploy new methods at scale.

In addition, Hail exposes a high-level domain-specific language (DSL) for manipulating genetic data and assembling pipelines.  As an example, porting a rare-variant analysis from Python to Hail reduced the number of lines of code by ~10x and improved performance by ~100x.  We aim to grow Hail into a scalable, reliable and expressive framework on which the genetics community develops, validates, and shares new analytic approaches on massive datasets to uncover the biology of disease.