This paper studies the effect of the moduli number within a moduli set on the overall speed of the residue number system (RNS). Choosing a proper moduli set greatly affects the performance of the whole system. The widely known issue is that as the number of moduli increases the speed of the residue arithmetic units (RAUs) increases, whereas the residue-to-binary converters (RCs) become slower and more complex. Thus, we carried out a detailed study on different moduli sets with different moduli number and different dynamic ranges (DRs) and compared timing performance of systems based on them in order to determine the moduli number effect on the overall RNS timing performance and find out the most efficient set for each DR.