Why and when should you pool? Analyzing Pooling in Recurrent\n Architectures

Pooling-based recurrent neural architectures consistently outperform their\ncounterparts without pooling. However, the reasons for their enhanced\nperformance are largely unexamined. In this work, we examine three commonly\nused pooling techniques (mean-pooling, max-pooling, and attention), and propose\nmax-attention, a novel variant that effectively captures interactions among\npredictive tokens in a sentence. We find that pooling-based architectures\nsubstantially differ from their non-pooling equivalents in their learning\nability and positional biases--which elucidate their performance benefits. By\nanalyzing the gradient propagation, we discover that pooling facilitates better\ngradient flow compared to BiLSTMs. Further, we expose how BiLSTMs are\npositionally biased towards tokens in the beginning and the end of a sequence.\nPooling alleviates such biases. Consequently, we identify settings where\npooling offers large benefits: (i) in low resource scenarios, and (ii) when\nimportant words lie towards the middle of the sentence. Among the pooling\ntechniques studied, max-attention is the most effective, resulting in\nsignificant performance gains on several text classification tasks.\n