Despite the importance of visual working memory (VWM) in supporting everyday behavior, it can be susceptible to interference: Perceptual distractors appearing during maintenance can impair the integrity of representations stored in VWM. There are mechanisms to help protect representations, such as reinforcing an item with a retroactive cue (retro-cue). Critically, previous work focused on the protection of inactive VWM representations—representations that are maintained over a delay period but are not being actively used during the delay period. Our world, however, is dynamic, often requiring active use/manipulation of items in VWM. Can in-use VWM representations be protected against perceptual interference? In the present study, participants remembered the orientations of two briefly presented colored bars. Subsequently, a colored retro-cue identified one bar as the target. Subjects were then cued to mentally rotate the target: Two sequential rotation cues indicated the amount and direction to mentally rotate. On some trials, a colorful perceptual distractor appeared during the trial. Critically, we manipulated when in the trial sequence the distractor appeared, either during manipulation of the target bar or after manipulation was complete. Perceptual interference was measured as poorer orientation report performance on Distractor-present versus Distractor-absent trials. We observed robust perceptual interference when the distractor appeared while the item was in-use (during VWM manipulation), but not when the distractor appeared once the item was no longer in-use. The results suggest that VWM items are not protected from visual interference while they are in-use, but they can be protected once they are no longer in-use.