The main challenges faced in detecting infrared small targets include small target sizes, lack of clear shape and texture information, and low signal-to-noise ratio in complex environments. We propose an infrared small target detection network, FSFANet, to overcome these difficulties. The network improves the representation of features for small targets through the integration of a prior knowledgebased attention module (PAM) and a multi-scale dilated convolution module (MDCM). Additionally, the U-Net architecture is improved to effectively integrate low-level details and high-level semantic information through a full-scale skip connection strategy. To further incorporate features of different scales, the axial self-cross attention module (ASCA) is introduced. This module can capture long-range dependencies and encode multi-scale information while maintaining computational efficiency. In addition, this paper introduces a hybrid loss function that combines weighted cross-entropy loss and SoftIoU loss to effectively address the issue of imbalance between target and background pixels, thereby enhancing the segmentation accuracy of the model. Experimental results on multiple datasets validate the outstanding performance of FSFANet with a relatively small number of parameters, demonstrating its effectiveness in applying small infrared target detection.