Emotion Recognition in Conversation (ERC) aims to distinguish the emotion of a target utterance within a conversation from predefined emotion categories. ERC from textual dialogues remains challenging due to the difficulty of distinguishing fine-grained emotion categories. This challenge arises from both label-level ambiguity and expression-level variability in conversational data. Specifically, many emotion categories are close in semantic space. Moreover, Personality differences lead to the same emotion being expressed in text with surface-level variations. Existing ERC methods, including those based on pre-trained models and LLMs, mainly focus on improving contextual modeling, while leaving these two sources of confusion largely unaddressed. To this end, we propose an Implicit Personality Enhanced Hierarchical LLM Framework for ERC, named IMPERIAL, which leverages LLMs to enhance emotion understanding from both the output and input perspectives. Specifically, we first generate label definitions to better distinguish semantically similar emotion labels and then mine implicit personality traits to implicitly model personality-driven expression variability. Furthermore, we design a hierarchical prompting mechanism supported by two selection indicators ($SAI$and$EH$), which jointly enable dynamic stage and prompt selection. This design improves classification accuracy while reducing unnecessary computational overhead. We evaluated IMPERIAL on the MELD, IEMOCAP and GoEmotions datasets. IMPERIAL outperforms existing baselines under the zero-shot setting and achieves comparable results to the supervised SOTAs.