Skip to contents

Using DelayedMatrix with MultiAssayExperiment

MultiAssay Special Interest Group

September 10, 2024

Source: vignettes/UsingHDF5Array.Rmd
UsingHDF5Array.Rmd

Integrating an HDF5 backend for MultiAssayExperiment

Dependencies

HDF5Array and DelayedArray Constructor

The HDF5Array package provides an on-disk representation of large datasets without the need to load them into memory. Convenient lazy evaluation operations allow the user to manipulate such large data files based on metadata. The DelayedMatrix class in the DelayedArray package provides a way to connect to a large matrix that is stored on disk.

First, we create a small matrix for constructing the DelayedMatrix class.

smallMatrix <- matrix(rnorm(10e5), ncol = 20)

We add rownames and column names to the matrix object for compatibility with the MultiAssayExperiment representation.

rownames(smallMatrix) <- paste0("GENE", seq_len(nrow(smallMatrix)))
colnames(smallMatrix) <- paste0("SampleID", seq_len(ncol(smallMatrix)))

Here we use the DelayedArray constructor function to create a DelayedMatrix object.

smallMatrix <- DelayedArray(smallMatrix)
class(smallMatrix)
## [1] "DelayedMatrix"
## attr(,"package")
## [1] "DelayedArray"
# show method
smallMatrix
## <50000 x 20> DelayedMatrix object of type "double":
##              SampleID1    SampleID2    SampleID3 ...   SampleID19   SampleID20
##     GENE1 -1.400043517  1.431663588 -1.336606503   . -2.307774531 -2.293199815
##     GENE2  0.255317055  0.171127984  0.089914742   .  0.614199996 -0.335822358
##     GENE3 -2.437263611  2.312015552  0.379995935   . -0.733181166  0.083092829
##     GENE4 -0.005571287 -0.057850207 -0.056034294   .  0.818604632  0.106635015
##     GENE5  0.621552721 -0.975709562  0.700863028   .  0.005259257  1.643080357
##       ...            .            .            .   .            .            .
## GENE49996    0.2027090    0.4424774    0.6336781   .  -0.47392601   0.28854415
## GENE49997    1.2847634    1.2558989    1.2587418   .  -2.52268507  -1.77342923
## GENE49998    0.2648680    1.3883708   -0.3003243   .  -1.76756368   0.76883529
## GENE49999    1.4098422    0.2065422   -0.4955832   .   1.59338432  -0.05131006
## GENE50000   -0.7546331    0.6750468    0.1188484   .  -1.25230132   0.30771323
dim(smallMatrix)
## [1] 50000    20

Writing to a file with dimnames

Finally, the rhdf5 package stores dimnames in a standard location.

In order to make use of this functionality, we would use writeHDF5Array with the with.dimnames argument:

testh5 <- tempfile(fileext = ".h5")
writeHDF5Array(smallMatrix, filepath = testh5, name = "smallMatrix",
    with.dimnames = TRUE)
## <50000 x 20> HDF5Matrix object of type "double":
##              SampleID1    SampleID2    SampleID3 ...   SampleID19   SampleID20
##     GENE1 -1.400043517  1.431663588 -1.336606503   . -2.307774531 -2.293199815
##     GENE2  0.255317055  0.171127984  0.089914742   .  0.614199996 -0.335822358
##     GENE3 -2.437263611  2.312015552  0.379995935   . -0.733181166  0.083092829
##     GENE4 -0.005571287 -0.057850207 -0.056034294   .  0.818604632  0.106635015
##     GENE5  0.621552721 -0.975709562  0.700863028   .  0.005259257  1.643080357
##       ...            .            .            .   .            .            .
## GENE49996    0.2027090    0.4424774    0.6336781   .  -0.47392601   0.28854415
## GENE49997    1.2847634    1.2558989    1.2587418   .  -2.52268507  -1.77342923
## GENE49998    0.2648680    1.3883708   -0.3003243   .  -1.76756368   0.76883529
## GENE49999    1.4098422    0.2065422   -0.4955832   .   1.59338432  -0.05131006
## GENE50000   -0.7546331    0.6750468    0.1188484   .  -1.25230132   0.30771323

To see the file structure we use h5ls:

h5ls(testh5)
##                    group                  name       otype dclass        dim
## 0                      / .smallMatrix_dimnames   H5I_GROUP                  
## 1 /.smallMatrix_dimnames                     1 H5I_DATASET STRING      50000
## 2 /.smallMatrix_dimnames                     2 H5I_DATASET STRING         20
## 3                      /           smallMatrix H5I_DATASET  FLOAT 50000 x 20

Importing HDF5 files

Note that a large matrix from an HDF5 file can also be loaded using the HDF5ArraySeed and DelayedArray functions.

hdf5Data <- HDF5ArraySeed(file = testh5, name = "smallMatrix")
newDelayedMatrix <- DelayedArray(hdf5Data)
class(newDelayedMatrix)
## [1] "HDF5Matrix"
## attr(,"package")
## [1] "HDF5Array"
newDelayedMatrix
## <50000 x 20> HDF5Matrix object of type "double":
##              SampleID1    SampleID2    SampleID3 ...   SampleID19   SampleID20
##     GENE1 -1.400043517  1.431663588 -1.336606503   . -2.307774531 -2.293199815
##     GENE2  0.255317055  0.171127984  0.089914742   .  0.614199996 -0.335822358
##     GENE3 -2.437263611  2.312015552  0.379995935   . -0.733181166  0.083092829
##     GENE4 -0.005571287 -0.057850207 -0.056034294   .  0.818604632  0.106635015
##     GENE5  0.621552721 -0.975709562  0.700863028   .  0.005259257  1.643080357
##       ...            .            .            .   .            .            .
## GENE49996    0.2027090    0.4424774    0.6336781   .  -0.47392601   0.28854415
## GENE49997    1.2847634    1.2558989    1.2587418   .  -2.52268507  -1.77342923
## GENE49998    0.2648680    1.3883708   -0.3003243   .  -1.76756368   0.76883529
## GENE49999    1.4098422    0.2065422   -0.4955832   .   1.59338432  -0.05131006
## GENE50000   -0.7546331    0.6750468    0.1188484   .  -1.25230132   0.30771323

Using a DelayedMatrix with MultiAssayExperiment

A DelayedMatrix alone conforms to the MultiAssayExperiment API requirements. Shown below, the DelayedMatrix can be put into a named list and passed into the MultiAssayExperiment constructor function.

HDF5MAE <- MultiAssayExperiment(experiments = list(smallMatrix = smallMatrix))
sampleMap(HDF5MAE)
## DataFrame with 20 rows and 3 columns
##           assay     primary     colname
##        <factor> <character> <character>
## 1   smallMatrix   SampleID1   SampleID1
## 2   smallMatrix   SampleID2   SampleID2
## 3   smallMatrix   SampleID3   SampleID3
## 4   smallMatrix   SampleID4   SampleID4
## 5   smallMatrix   SampleID5   SampleID5
## ...         ...         ...         ...
## 16  smallMatrix  SampleID16  SampleID16
## 17  smallMatrix  SampleID17  SampleID17
## 18  smallMatrix  SampleID18  SampleID18
## 19  smallMatrix  SampleID19  SampleID19
## 20  smallMatrix  SampleID20  SampleID20
colData(HDF5MAE)
## DataFrame with 20 rows and 0 columns

SummarizedExperiment with DelayedMatrix backend

A more information rich DelayedMatrix can be created when used in conjunction with the SummarizedExperiment class and it can even include rowRanges. The flexibility of the MultiAssayExperiment API supports classes with minimal requirements. Additionally, this SummarizedExperiment with the DelayedMatrix backend can be part of a bigger MultiAssayExperiment object. Below is a minimal example of how this would work:

HDF5SE <- SummarizedExperiment(assays = smallMatrix)
assay(HDF5SE)
## <50000 x 20> DelayedMatrix object of type "double":
##              SampleID1    SampleID2    SampleID3 ...   SampleID19   SampleID20
##     GENE1 -1.400043517  1.431663588 -1.336606503   . -2.307774531 -2.293199815
##     GENE2  0.255317055  0.171127984  0.089914742   .  0.614199996 -0.335822358
##     GENE3 -2.437263611  2.312015552  0.379995935   . -0.733181166  0.083092829
##     GENE4 -0.005571287 -0.057850207 -0.056034294   .  0.818604632  0.106635015
##     GENE5  0.621552721 -0.975709562  0.700863028   .  0.005259257  1.643080357
##       ...            .            .            .   .            .            .
## GENE49996    0.2027090    0.4424774    0.6336781   .  -0.47392601   0.28854415
## GENE49997    1.2847634    1.2558989    1.2587418   .  -2.52268507  -1.77342923
## GENE49998    0.2648680    1.3883708   -0.3003243   .  -1.76756368   0.76883529
## GENE49999    1.4098422    0.2065422   -0.4955832   .   1.59338432  -0.05131006
## GENE50000   -0.7546331    0.6750468    0.1188484   .  -1.25230132   0.30771323
MultiAssayExperiment(list(HDF5SE = HDF5SE))
## A MultiAssayExperiment object of 1 listed
##  experiment with a user-defined name and respective class.
##  Containing an ExperimentList class object of length 1:
##  [1] HDF5SE: SummarizedExperiment with 50000 rows and 20 columns
## Functionality:
##  experiments() - obtain the ExperimentList instance
##  colData() - the primary/phenotype DataFrame
##  sampleMap() - the sample coordination DataFrame
##  `$`, `[`, `[[` - extract colData columns, subset, or experiment
##  *Format() - convert into a long or wide DataFrame
##  assays() - convert ExperimentList to a SimpleList of matrices
##  exportClass() - save data to flat files

Additional scenarios are currently in development where an HDF5Matrix is hosted remotely. Many opportunities exist when considering on-disk and off-disk representations of data with MultiAssayExperiment.

Session info 

## R version 4.4.1 (2024-06-14)
## Platform: x86_64-pc-linux-gnu
## Running under: Ubuntu 22.04.4 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
## LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so;  LAPACK version 3.10.0
## 
## locale:
##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
##  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
##  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
##  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
## 
## time zone: Etc/UTC
## tzcode source: system (glibc)
## 
## attached base packages:
## [1] stats4    stats     graphics  grDevices utils     datasets  methods  
## [8] base     
## 
## other attached packages:
##  [1] HDF5Array_1.32.1            rhdf5_2.48.0               
##  [3] DelayedArray_0.30.1         SparseArray_1.4.8          
##  [5] S4Arrays_1.4.1              abind_1.4-5                
##  [7] Matrix_1.7-0                MultiAssayExperiment_1.30.3
##  [9] SummarizedExperiment_1.34.0 Biobase_2.64.0             
## [11] GenomicRanges_1.56.1        GenomeInfoDb_1.40.1        
## [13] IRanges_2.38.1              S4Vectors_0.42.1           
## [15] BiocGenerics_0.50.0         MatrixGenerics_1.16.0      
## [17] matrixStats_1.4.1           BiocStyle_2.32.1           
## 
## loaded via a namespace (and not attached):
##  [1] sass_0.4.9              lattice_0.22-6          digest_0.6.37          
##  [4] evaluate_0.24.0         grid_4.4.1              bookdown_0.40          
##  [7] fastmap_1.2.0           jsonlite_1.8.8          BiocManager_1.30.25    
## [10] httr_1.4.7              UCSC.utils_1.0.0        textshaping_0.4.0      
## [13] jquerylib_0.1.4         cli_3.6.3               rlang_1.1.4            
## [16] crayon_1.5.3            XVector_0.44.0          cachem_1.1.0           
## [19] yaml_2.3.10             tools_4.4.1             Rhdf5lib_1.26.0        
## [22] GenomeInfoDbData_1.2.12 R6_2.5.1                lifecycle_1.0.4        
## [25] zlibbioc_1.50.0         fs_1.6.4                htmlwidgets_1.6.4      
## [28] ragg_1.3.2              desc_1.4.3              pkgdown_2.1.0          
## [31] bslib_0.8.0             systemfonts_1.1.0       xfun_0.47              
## [34] knitr_1.48              rhdf5filters_1.16.0     htmltools_0.5.8.1      
## [37] rmarkdown_2.28          compiler_4.4.1