SIFT is a well-known algorithm on image retrieval with good precision and robustness, but its high computation time makes it difficult in practical projects. A fast SIFT algorithm based on color vector (CV-SIFT) is proposed to enhance the time performance and keep precision. A feature extracted method based on region-averaged color vector is introduced to achieve color descriptors, which is a new feature descriptor in CV-SIFT. At the matching stage, a new matching policy is used to reduce false matches by combining the threshold and ratio of similarity. Experiments demonstrate that the time performance of CV-SIFT is boosted greatly, and the precision is retained fairly.