Design and Characterization of InfiniBand Hardware Tag Matching in MPI

Message Passing Interface (MPI) standard uses (source rank, tag, and communicator id) to properly place the incoming data into the application receive buffer. The act of searching through the receive queues and finding the appropriate match is called Tag Matching (TM). In the state-of-the-art MPI libraries, this operation is either being performed by the main thread or a separate communication progress thread. Either way leads to underutilization of the resources and major synchronization overheads leading to less optimal performance. Mellanox ConnectX-5 network architecture has introduced a feature to offload the Tag Matching and communication progress from host to InfiniBand network card. This paper proposes a Hardware Tag Matching aware MPI library and discusses various aspects and challenges of leveraging this feature in MPI library. Moreover, it characterizes hardware Tag Matching using different benchmarks and provides guidelines for the application developers to develop Hardware Tag Matching-aware applications to maximize their usage of this feature. Our proposed designs are able to improve the performance of non-blocking collectives up to 42% on 512 nodes and improve the performance of 3Dstencil application kernel on 7168 processes and Nekbone on 512 processes by a factor 40% and 3.5%, respectively.