We demonstrate an all optically driven memory device based on vanadium dioxide ( VO2 ) for terahertz (THz) waves. By easily tuning the power of illuminating light, a VO2 memory device is coded in reconfigurable and multi-level states, taking advantage of its hysteretic metal-to-insulator phase transition (MIT). Further, writing with intense femtosecond pulses, the memory effects are performed by non-thermal photo-induced MIT, and resultant 2-bit coding with a write time of 22 µs is demonstrated, yielding a dramatically improved write rate compared to existing thermally controlled VO2 memory device. The proposed all optically driven VO2 -based memory device paves the way for actively manipulating THz waves within robust reconfigurable high-speed memory functionality.