Advances in out-of-core computing

Since size is relative to technology define ``big'' in terms of technology

A data set is large if it exceeds 20% of the available RAM for a single machine

• ``R is not well-suited for working with data structures larger than 10-20% of a computer's RAM.'' -The R Installation and Administration Manual
• You can use standard tools but you need to be careful

A data set is massive if it exceeds 50% of available RAM on a single machine

• Can't afford even a single copy without swapping
• Need an alternative approach

"The data may not contain the answer. The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data." -John W. Tukey

Computations on data are limited by how quickly we can get data to a processor (or computing resource)

The term ``Big Data'' is technological concept

``Computers are like cars. Everyone wants a faster one but no one knows why.'' -unknown

1. Not easily retrofitted to existing applications
   • In-core data structures have to be swapped for out-of-core ones
   • OOC data structures sometimes behave differently
   • OOC data structures have different performance characteristics
2. Development requires system-level expertise
   • Different practice than application development or algorithm delopment
   • Another component to manage
3. Generally platform specific
   • Windows is not Linux is not OS X
   • Maintain parallel versions of functionally equivalent software

If the allocation were small standard memory functions (malloc/new) could be called

If the alloation were large:

1. Create and memory map (mmap) a file with specified size
2. Return the mmap'ed pointer address

Applications could run OOC without modification

Existing data-processing applications could accomodate much larger data sets without modification

A general, transparent tool for OOC computing

Launched as a command line utility taking an application as a argument

All memory allocations larger than a specified threshold are file-backed

Process- and thread-safe

When data is not needed it is stored on disk, when it is it is cached

Tested with R, python, and the GIMP.

Start an application in flexmem by specifying the application as an argument

mike@mike-VirtualBox:~/$ flexmem R

memory-mapped files are stored in /tmp by default

Internals can be interrogated using the Rflexmem package

Works right now for Linux with OS X support on the way

Feedback and contributions are welcome

Project can be found at: https://github.com/kaneplusplus/flexmem

flexmem is a tool for managing data

• Transparently create OOC data structures
• Will not make your analyses faster

Complements ``Big data'' development

• Allows you to focus on algorithm development
• Allows you to easily apply new algorithms larger-than-RAM data

For an \(n \times p\) matrix the computational complexity of the SVD is \(O(n^2p + n p^2 + p^3)\) for \(n \geq p\).

Difficult to calculate for even modest size matrices

Implicitly Restarted Lanczos Bidiagonalization Algorithm (IRLBA) (Baglama and Reichel, 2005)

• Iterative approximation method
• Truncated SVD
• Generally converges in a constant number of steps

1. OS X support

2. Directly map to data structures on disk
   • No need to serialize
   • Planned for R

3. Feedback/testing

https://github.com/kaneplusplus/flexmem