Towards Optimizing Learned Index for High Performance, Memory Efficiency and NUMA Awareness

Learned indexes provide significant performance advantages over classical ordered indexes. However, current learned indexes face challenges regarding tradeoffs between performance and space, as well as scalability issues in platforms with multiple NUMA nodes. These limitations hinder the practical application of learned indexes in production environments. This article presents DiffLex, a learned index with high-performance, memory-efficiency, and NUMA-awareness. The core design of DiffLex is to perform differentiated management based on the popularity of data. For optimal performance, DiffLex stores newly inserted data in sparse delta arrays and frequently accessed data in sparse hot cache arrays. However, for cold data that occupy a majority of the storage space, DiffLex stores them in dense arrays and conducts compression to reduce memory costs. DiffLex ensures NUMA-awareness by partitioning sparse deltas and replicating the hot cache arrays across multiple NUMA nodes. Additionally, we propose a persistent version of DiffLex tailored for emerging persistent memory devices. Our evaluation results demonstrate that DiffLex achieving 3.88× and 1.82× performance improvements compared to state-of-the-art learned indexes, while maintaining a compact index size.