This paper proposed a method of analysis the energy of video signal and an interference removal algorithm which bases on it. First, we give the definition of signal energy, which follows by a discussion of the signal average energy and instant energy. Next we construct self-adaptive threshold of the average energy, and then propose an interference removal algorithm. Finally, the experiment results of single pixel show that the algorithm is very effective.