Just Say No: Analyzing the Stance of Neural Dialogue Generation in\n Offensive Contexts

Dialogue models trained on human conversations inadvertently learn to\ngenerate toxic responses. In addition to producing explicitly offensive\nutterances, these models can also implicitly insult a group or individual by\naligning themselves with an offensive statement. To better understand the\ndynamics of contextually offensive language, we investigate the stance of\ndialogue model responses in offensive Reddit conversations. Specifically, we\ncreate ToxiChat, a crowd-annotated dataset of 2,000 Reddit threads and model\nresponses labeled with offensive language and stance. Our analysis reveals that\n42% of human responses agree with toxic comments, whereas only 13% agree with\nsafe comments. This undesirable behavior is learned by neural dialogue models,\nsuch as DialoGPT, which we show are two times more likely to agree with\noffensive comments. To enable automatic detection of offensive language, we\nfine-tuned transformer-based classifiers on ToxiChat that achieve 0.71 F1 for\noffensive labels and 0.53 Macro-F1 for stance labels. Finally, we quantify the\neffectiveness of controllable text generation (CTG) methods to mitigate the\ntendency of neural dialogue models to agree with offensive comments. Compared\nto the baseline, our best CTG model achieves a 19% reduction in agreement with\noffensive comments and produces 29% fewer offensive replies. Our work\nhighlights the need for further efforts to characterize and analyze\ninappropriate behavior in dialogue models, in order to help make them safer.\nOur code and corpus are available at https://github.com/abaheti95/ToxiChat .\n