While multi-key homomorphic encryption (MKHE) ensures privacy in federated learning (FL) by encrypting model updates, its requirement for aggregate ciphertext decryption and dropout handling increases communication rounds during aggregation. To enable secure multi-key aggregation with low communication rounds, we propose SAMK. SAMK enables the server to compute the sum of model updates from clients participating in FL, while keeping these updates encrypted by different keys throughout the computation. By utilizing the polynomial property of the BFV ciphertext, SAMK successfully implements individual decryption of the aggregated ciphertext (encrypted under multiple keys) by each client using their respective keys, resulting in low communication rounds. It means that in SAMK, all client interactions are avoided and the client-server interaction is only once (ciphertext uploads and downloads) in each round of aggregation computation. In addition, SAMK is robust to any number of clients dropping out at any time, and the client who has dropped out after uploading model updates, can still get the correct aggregation result upon reconnecting. We prove the security of SAMK for semi-honest server and clients, where client collusion is also considered. At last, we implement SAMK and comprehensively evaluate its performance to demonstrate its practicability.