R | disk.frame (larger-than-memory dataframes)

disk.frame [2] is a package that aims to solve the problem of manipulating tables that are larger than the size of the local machine's memory, by saving chunks of the table as FST [1] files and accessing them as needed. It may be suitable for tables that are too big to (comfortably) fit into memory, but not large enough to require a separate database management system like SQLite. In this space, it competes with other read-from-disk solutions such as Apache arrow's dataset construct and Tidyverse's vroom.

Resources:

• [1] R | FST package (fast read/write to disk)

• [2] https://diskframe.com/index.html

• [3] disk.frame supported dplyr verbs: https://diskframe.com/index.html#dplyr-verbs