Threadle: A Memory-Efficient Network Storage and Query Engine for Large, Multilayer, and Mixed-mode Networks

Researchers Carl Nordlund and Yukun Jiao have introduced Threadle, a groundbreaking open-source engine designed to solve a fundamental scaling problem in network science. Existing libraries struggle with massive, multilayer, mixed-mode networks, especially those containing two-mode (bipartite) data derived from sources like national administrative registers. Threadle directly addresses this by providing a high-performance, memory-efficient storage and query system that can handle networks with millions of nodes and billions of edges, a scale that has been a persistent bottleneck for social network analysis, epidemiology, and computational social science.

Threadle's technical breakthrough is its pseudo-projection approach, which allows analysts to query two-mode network layers as if they were in a projected one-mode form, but without ever creating the actual projection, which would be prohibitively large. The team demonstrates that a network with 20 million nodes containing layers equivalent to 8 trillion projected edges can be stored in approximately 20 GB of RAM, achieving a compression ratio exceeding 2000:1. The system includes an integrated node attribute manager, a CLI with 50+ commands, and is supplemented by threadleR, an R frontend for advanced sampling and traversal analyses. This makes large-scale, heterogeneous population network analysis computationally feasible for the first time.