Rab GTPases are molecular switches with essential roles in mediating vesicular trafficking and establishing organelle identity. The conversion from the inactive, cytosolic to the membrane-bound, active species and back is tightly controlled by regulatory proteins. Recently, the roles of membrane properties and lipid composition of different target organelles in determining the activity state of Rabs have come to light. The investigation of several Rab guanine nucleotide exchange factors (GEFs) has revealed principles of how the recruitment via lipid interactions and the spatial confinement on the membrane surface contribute to spatiotemporal specificity in the Rab GTPase network. This paints an intricate picture of the control mechanisms in Rab activation and highlights the importance of the membrane lipid code in the organization of the endomembrane system.