Most existing medical image analysis systems are task-specific, requiring separate models for classification and segmentation, and lack the flexibility to support user-defined workflows expressed in natural language. Developing a prompt-driven medical assistant introduces significant challenges, including the need for large annotated datasets, integration of multimodal data (text and images), high computational demands, and the difficulty of adapting to diverse tasks without retraining. To address these challenges, we present MedPrompt, a unified and extensible framework that combines a few-shot prompted Large Language Model (LlaMA-4-17B) for task planning with a modular CNN architecture (DeepFusionLab) for image analysis. The LLM parses user instructions and generates structured JSON workflows, dynamically selecting and routing task-specific pretrained weights to DeepFusionLab—a strategy we term dynamic weight routing . This eliminates the need for retraining when adding new tasks and supports multi-stage and conditional task execution. We evaluate MedPrompt on 19 public datasets across 12 tasks and 5 imaging modalities. The system achieves 97% end-to-end correctness in interpreting and executing natural language prompts, with an average inference latency of 2.5 s on CPU—suitable for near real-time clinical applications. DeepFusionLab demonstrates strong segmentation accuracy (e.g., Dice ∼ 0.9856 for lungs) and classification performance (F1 ∼ 0.9744 for tuberculosis). Overall, MedPrompt offers a scalable, interpretable, and deployment-ready solution for prompt-driven medical image analysis across a wide range of clinical scenarios.