Improving the Robustness of Transformer-based Large Language Models with Dynamic Attention

Transformer-based models, such as BERT and GPT, have been widely adopted in\nnatural language processing (NLP) due to their exceptional performance.\nHowever, recent studies show their vulnerability to textual adversarial attacks\nwhere the model's output can be misled by intentionally manipulating the text\ninputs. Despite various methods that have been proposed to enhance the model's\nrobustness and mitigate this vulnerability, many require heavy consumption\nresources (e.g., adversarial training) or only provide limited protection\n(e.g., defensive dropout). In this paper, we propose a novel method called\ndynamic attention, tailored for the transformer architecture, to enhance the\ninherent robustness of the model itself against various adversarial attacks.\nOur method requires no downstream task knowledge and does not incur additional\ncosts. The proposed dynamic attention consists of two modules: (I) attention\nrectification, which masks or weakens the attention value of the chosen tokens,\nand (ii) dynamic modeling, which dynamically builds the set of candidate\ntokens. Extensive experiments demonstrate that dynamic attention significantly\nmitigates the impact of adversarial attacks, improving up to 33\\% better\nperformance than previous methods against widely-used adversarial attacks. The\nmodel-level design of dynamic attention enables it to be easily combined with\nother defense methods (e.g., adversarial training) to further enhance the\nmodel's robustness. Furthermore, we demonstrate that dynamic attention\npreserves the state-of-the-art robustness space of the original model compared\nto other dynamic modeling methods.\n