qr(..., method = "tsqr"): fixed incorrect results
(loss of orthogonality in Q) when the last row-block had fewer rows than
columns (m mod block_size < n). The last block is now
merged with the previous one, guaranteeing all blocks satisfy the rank
condition required by the local Householder QR step. Affected
configurations: any (m, n, threads) where
m mod max(2n, ceil(m / max(4, threads * 4))) < n.
Discovered via a systematic regression test covering 202 (m, n, threads,
block_size) combinations.
pseudoinverse() now correctly zeroes singular values
at or below the tolerance (1e-9), fixing incorrect results for
rank-deficient or near-singular matrices.
Fixed silent failure on matrix conversion errors in
bdCrossprod(), bdtCrossprod(), and
bd_wproduct(); these now raise informative errors.
Removed unreachable code in the correlation HDF5 backend (no functional effect).
pseudoinverse.matrix() for consistent in-memory
support alongside the existing HDF5Matrix method.pseudoinverse() and
prcomp(rank. =) documentation to match actual
behavior.crossprod(),
tcrossprod(), aggregation functions, scale(),
and solve(): when input fits within 20% of available RAM, a
single HDF5 read + BLAS call + single write replaces the block
loop.scale() PATH 2 (block-wise streaming) is now
parallelized with OpenMP.hdf5matrix_options(paral = TRUE) now forces PATH 2 (OMP
streaming) in %*%, crossprod(),
tcrossprod(), aggregations, and scale(),
giving explicit thread control regardless of matrix size.getAvailableMemoryMB() now uses platform-native
detection (macOS: host_statistics64; Linux:
/proc/meminfo), replacing the hardcoded 4 GB fallback.target_blocks = max(4, nthreads * 4)).rbind() and cbind() with 3 or more
HDF5Matrix arguments no longer produce dataset names
exceeding 1400 characters. Auto-generated names are now
rbind_N_<uid> with a random 8-character identifier,
also preventing “dataset already exists” collisions. Two-argument calls
are unchanged (A_rbind_B).paral = FALSE being silently ignored in
multiplication() PATH 2/3.#pragma omp critical(accessFile) in
crossprod(), tcrossprod(), and aggregation
PATH 2 (race condition fix).Cholesky_decomposition_intermediate_hdf5 on macOS ARM64
(removed throw from OpenMP parallel region).scale(): added paral, threads
parameters; PATH 1/PATH 2 @details;
@seealso hdf5matrix_options().split(), hdf5_apply(),
hdf5_reduce(): clarified calling conventions and two-level
access pattern. Fixed legacy example.hdf5_import
example.This is a major release of BigDataStatMeth. It introduces a new
HDF5Matrix user-facing interface, standard S3 methods for
HDF5-backed matrices, and an extended C++ infrastructure for block-wise
statistical computing with HDF5 files.
HDF5Matrix interface for working with
matrices stored in HDF5 files.dim(), [,
[<-, %*%, crossprod(),
tcrossprod(), scale(), cor(),
svd(), prcomp(), qr(),
chol(), and solve().hdf5matrix_options()
for common settings such as parallel execution, number of threads, block
size, and HDF5 compression.HDF5Matrix objects.%*%, crossprod(), tcrossprod(),
cbind(), and rbind().colSums(), rowSums(), colMeans(),
rowMeans(), colVars(), rowVars(),
colSds(), rowSds(), colMins(),
rowMins(), colMaxs(), rowMaxs(),
mean(), var(), and sd().scale(), sweep(), and cor().svd(), prcomp(),
qr(), chol(), solve(),
eigen(), and pseudoinverse().diag(), diag<-(), diag_op(),
and diag_scale().bd* utilities for
specialized workflows that do not map directly to standard R
generics.list_datasets() for inspecting datasets stored in
HDF5 files.hdf5_close_all() for closing open HDF5 handles
managed by the package.hdf5matrix_options().HDF5Matrix interface and standard R methods.help("BigDataStatMeth").extdata
directory.minimumBlockSize promoted from int to
long in
Cholesky_decomposition_intermediate_hdf5 and
Inverse_of_Cholesky_decomposition_intermediate_hdf5 to
prevent overflow before promotion to double for
sqrt(). Detected by gcc-ASAN on R-hub.matrixSvdBlock.hpp): wrong singular value was evaluated in
the rank-truncation check, causing incorrect nzeros counts.H5SL_insert from H5I_register) when the
number of hierarchical levels q >= 2:
Next_level_SvdBlock_decomposition_hdf5 is now sequential;
the outer block loop already provides sufficient parallelism.nev parameter not being applied to per-block
truncation in First_level_SvdBlock_decomposition_hdf5: the
parameter was declared in the function signature but never used, causing
unnecessarily large intermediate matrices and a final SVD over more
components than requested.bdCCA_hdf5_rcpp().