Sleep loss is associated with myriad decrements in cognitive function and is a ubiquitous risk factor for mood disorders. Nevertheless, little is known about the relations between sleep loss and its recovery and interference control of affective material. Delineating the association between interference control and sleep changes can provide insight into the link between sleep and the maintenance of mood disorders. Thus, the current study examined whether stimulus valence and time of day moderated the association between recovery sleep and cognitive inhibition following partial sleep deprivation. Healthy adults (N = 24) participated in a laboratory-based sleep study with baseline, sleep restriction, and recovery phases. Participants completed the modified Sternberg Task in the morning and afternoon on the final day of each phase. The association between recovery sleep and cognitive inhibition depended on time of day. Additionally, interference control improved from the baseline to the recovery phase, but not the sleep restriction phase, indicating that sleep deprivation may be associated with worse performance. The current study provides additional insight into factors (sleep and time of day) that are associated with interference control of affective information, which might have important implications for cognitive and emotional functioning.