To create the objects that are ultimately returned to users there are a number of steps taken beyond what is contained in the curatedMetagenomicData package – this document is intended to add some clarity and transparency to the process. Roughly, the process is as follows: a study is selected for inclusion, metadata for each sample in the study is manually curated, raw data for each sample is downloaded (usually from NCBI SRA), raw data is processed into the six data types (gene families, marker abundance, marker presence, pathway abundance, pathway coverage, and relative abundance) using MetaPhlAn3 and HUMAnN3, processed data for all samples in a study are combined into a matrix for each data type, the six matrices are saved as R objects and uploaded to ExperimentHub, and, finally, the matrices are used to construct (Tree)SummarizedExperiment objects on the fly that are returned to users.
The curation process takes place on GitHub in the curatedMetagenomicDataCuration repository. Sample metadata for every sample included in curatedMetagenomicData is manually curated by humans and checked against a standardized template for accuracy. The process is labor intensive and information comes from three principle sources: journal articles (and their supplementary materials), metadata included when samples were uploaded to NCBI, and other publications referring to or augmenting the original journal articles. Not only is explicit information (i.e. tables that enumerate the values of variables for each sample) sought out, but implicit information (e.g. “all the subjects in the study were non-smokers”) is also included where it occurs in text. The manually curated sample metadata is continuously updated with the advent of new information (this is common when reanalyses find issues) and these changes propagate from the curation repository to the R package weekly.
Raw data are processed using MetaPhlAn3 and HUMAnN3 and uploaded to a Google Cloud Storage bucket as zipped tar file. The tar file contains directories (one for each data type) of tsv files (one for each sample) that are used as the input of R pipeline.
The collection of tsv files are not obviously or immediately useful –
there is an pipeline written in R to process these outputs into relevant
matrix objects for user consumption. The curatedMetagenomicDataPipeline
repository on GitHub details the methods that are used to construct and
save the six matrices for each study that are uploaded to ExperimentHub.
The gene_families data type is large and stored as a sparse
matrix, while the other data types are store as standard matrix objects.
The upload to ExperimentHub is ultimately done using the aws-cli with
credentials from Bioconductor specific to the curatedMetagenomicData
package.
As for why we construct matrix objects: the process of combining the
samples into a matrix can take hours even on heavy-duty equipment (many
cores, many GB of RAM). This is both the advantage and crux of
curatedMetagenomicData – subseting the matrix takes mere seconds and
simple laptops are powerful enough; however, the ability to download
many single samples across studies is lost. We accomplish the later in
the R package by matrix subsetting and merging in the
returnSamples() method. All alternatives considered, this
is the optimal trade-off, but we are working towards something more
efficient in the future – stay tuned!