Inorganic aerogels have promising applications in thermal insulation of buildings or industries due to their low thermal conductivity and good thermal stability. In this paper, magnesium borate whiskers (MBOWs) and polyvinyl alcohol (PVA) are used to prepare PVA/MBOWs aerogels by self-assembly combined with the freeze-drying technique. The PVA/MBOWs aerogels develop a new synthesis route and composite aerogels system for a wide range of short fiber scaffolds. The well-developed laminar-like pore structure is obtained with the average pore size of about 100–200 nm. The aerogels show good thermal insulation properties with a low thermal conductivity of about 0.03 W/(m ⋅ K). Thermal insulation is better than commercial thermal insulation, layer clay aerogels, the average temperature difference can reach [Formula: see text]C. In addition, through the formation of a thermal insulation coke layer in the combustion/non-combustible gases/cooling flame retardant synergistic mechanism, to achieve a good fire retardant effect. Moreover, the maximum stress at 0.6 mm/mm strain is 1.8 MPa, which indicates that the aerogels have better mechanical properties. The combination of these properties makes PVA/MBOWs aerogels an effective and practical candidate for building insulation.