tsqr(data, compute_svd=False, _max_vchunk_size=None)
As presented in:
Benson, D. Gleich, and J. Demmel.
Direct QR factorizations for tall-and-skinny matrices in MapReduce architectures. IEEE International Conference on Big Data, 2013. https://arxiv.org/abs/1301.1071
This algorithm is used to compute both the QR decomposition and the Singular Value Decomposition. It requires that the input array have a single column of blocks, each of which fit in memory.
With k
blocks of size (m, n)
, this algorithm has memory use that scales as k * n * n
.
The implementation here is the recursive variant due to the ultimate need for one "single core" QR decomposition. In the non-recursive version of the algorithm, given k
blocks, after k
m * n
QR decompositions, there will be a "single core" QR decomposition that will have to work with a (k * n, n)
matrix.
Here, recursion is applied as necessary to ensure that k * n
is not larger than m
(if m / n >= 2
). In particular, this is done to ensure that single core computations do not have to work on blocks larger than (m, n)
.
Where blocks are irregular, the above logic is applied with the "height" of the "tallest" block used in place of m
.
Consider use of the rechunk
method to control this behavior. Taller blocks will reduce overall memory use (assuming that many of them still fit in memory at once).
Whether to compute the SVD rather than the QR decomposition
Used internally in recursion to set the maximum row dimension of chunks in subsequent recursive calls.
Direct Tall-and-Skinny QR algorithm
dask.array.linalg.qr
Powered by this algorithm
dask.array.linalg.sfqr
Variant for short-and-fat arrays
dask.array.linalg.svd
Powered by this algorithm
The following pages refer to to this document either explicitly or contain code examples using this.
dask.array.linalg.qr
dask.array.linalg.sfqr
dask.array.linalg.svd
Hover to see nodes names; edges to Self not shown, Caped at 50 nodes.
Using a canvas is more power efficient and can get hundred of nodes ; but does not allow hyperlinks; , arrows or text (beyond on hover)
SVG is more flexible but power hungry; and does not scale well to 50 + nodes.
All aboves nodes referred to, (or are referred from) current nodes; Edges from Self to other have been omitted (or all nodes would be connected to the central node "self" which is not useful). Nodes are colored by the library they belong to, and scaled with the number of references pointing them