Zero-shot anomaly detection (ZSAD) has emerged as a promising paradigm for identifying anomalies without category-specific training. However, current vision-language model (VLM)-based ZSAD methods primarily focus on coarse-grained or fine-grained abnormality semantics, while neglecting two critical aspects: (1) the semantic duality between fine-grained normality and abnormality, and (2) the mining of intrinsic fine-grained abnormal and normal visual information within images to enhance semantic sample adaptability, which limits their discriminative power. To address these challenges, we propose MFDPrompt, a novel framework that explicitly models the semantic duality between normal and abnormal representations through Multi-modal Fine-grained Dual Prompts. MFDPrompt incorporates three innovative components: Fine-grained Dual Text Prompts (FDTP) for generating complementary text prompt pairs with semantic disentanglement, Fine-grained Dual Visual Prompts (FDVP) for dynamic adaptation to target datasets via visual query-tuning, and a Multi-modal Prompt Generator (MPG) for fusing text and visual prompts through two-way cross-attention. Extensive experiments across 19 industrial and medical benchmarks demonstrate that MFDPrompt achieves new state-of-the-art results in ZSAD, significantly outperforming the AnomalyCLIP baseline by 2.7-3.6% in image-level AUROC/AP metrics and 0.7-5.2% in pixel-level AUROC/PRO scores, with performance averaged across all datasets. Our work underscores the importance of jointly modeling both normal and abnormal semantics in fine-grained, multi-modal prompts, offering a robust and generalizable solution for ZSAD.