Applying Attention to Tabular Data

Compared with the feed-forward, convolutional, and recurrent mechanisms discussed in Chapters 3 , 4 , and 5 , the attention mechanism has been popular in deep learning for a very short amount of time. Despite the brevity of its existence, it has become the basis of modern natural language processing models. Moreover, it is a very natural mechanism to compute relationships not just between tokens in a language sequence but also between features in a tabular dataset – which is why a significant body of recent work on deep learning tabular data methods centers on attention.