Abstract The formation mechanism of AlN whiskers on sapphire substrates during heat treatment in a mixed flow of H2 and N2 was investigated in the temperature range of 980–1380 °C. AlN whiskers grew above 1030 °C after covering the sapphire surface with a thin AlN layer. The existence of pits on the sapphire surface beneath the thin AlN layer was observed. Both AlN whisker and pit densities of samples were on the same order of 108 cm−2. These results suggested the following mechanism. First, the sapphire surface reacts with H2, and the generated Al gas reacts with N2 to form a thin AlN layer on sapphire. Then, the sapphire surface reacts with H2 diffusing to the AlN/sapphire interface. The Al gas escapes through dislocations in the AlN layer to leave pits on the sapphire surface, and finally reacts with N2 to form AlN whiskers on the top surface.