TrimStream: Adaptive Realtime Video Streaming through Intelligent Frame Retrospection in Adverse Network Conditions

Realtime video streaming (RVS) services are gaining popularity in various applications such as video conferencing, online education, and mixed reality. However, adverse network conditions can significantly damage video transmission, leading to a decline in users' Quality of Experience (QoE). Existing approaches have made considerable efforts to address these problems, including bitrate adaptation, FEC (forward error correction) encoding, and super-resolution techniques. Nevertheless, these methods either focus solely on adjusting transmission configurations (ABR) or consume additional network and computational resources to enhance QoE (FEC or super-resolution), making them suboptimal for adverse network conditions. In this paper, we analyze the limitations of conventional RVS systems when confronted with adverse network conditions and propose TrimStream , a novel RVS solution based on intelligent frame retrospection, to effectively handle such scenarios. Our approach leverages the high similarity observed between frames in realtime video streaming. The core idea is to store a subset of correctly received frames and exploit frame similarity to minimize transmission while breaking down frame-level dependencies. We formulate the frame caching problem to maximize QoE in RVS and present an online frame cache algorithm. Furthermore, we design a vision-transformer-based, cost-effective frame matching framework that combines different levels of frame information. Our evaluation results demonstrate that TrimStream outperforms state-of-the-art solutions by $14.8\% \sim 21.1\%$ improvement in overall QoE.