A Wearable Human-Machine-Interface (HMI) System based on Multi-Modal Sensing to Recognise Hand Gestures for Cutting-Edge Applications

Hand Gesture Recognition (HGR) is a vital component of Human-Machine Interface (HMI) systems, enabling intuitive control of prosthetics, assistive devices, and robotics. While surface electromyography (sEMG) is widely used, its reliability is affected by electrode shift, muscle fatigue, inter-subject variability and especially the “limb position effect,” where accuracy drops as arm orientation changes. This thesis presents a comprehensive framework to systematically address the “limb position effect” challenge and other challenges associated with HGR.A core contribution is the design and validation of a wearable multi-modal armband that integrates co-located sEMG and pressure-based force myography (pFMG) sensors. This novel configuration, placing sEMG electrodes directly upon custom-designed, 3D-printed flexible air chambers, enables the simultaneous capture of both electrical and mechanical muscle activity from identical anatomical locations. The pFMG chambers were optimised using Finite Element Analysis (FEA) for a linear force-to-pressure response. This thesis has demonstrated that multi-modal sensing not only improves recognition accuracy in fixed positions, but also enhances robustness to arm position changes, consistently outperforming single-modality systems. The effectiveness of the integrated sensors in reliably collecting data from static and varying/dynamic arm positions is systematically assessed.Advanced machine learning (ML) approaches were then applied. Temporal Convolutional Networks (TCNs) and novel fusion strategies were explored, with a dynamic-static mixed training strategy proposed to further reduce limb position sensitivity. This approach goes beyond conventional methods by incorporating gesture data collected across both multiple static postures and continuous dynamic arm movements, thereby providing the learning model with a richer and more representative dataset. For sensor fusion, we compared feature-level and decision-level fusion architectures and further introduced a novel hybrid architecture combining both, referred to as Decision-level and Feature-level Fusion TCN (DFF-TCN) model. This new architecture effectively combines the strengths of both fusion methods by capturing complex cross-modal dependencies at the feature level while preserving modality-specific robustness at the decision level, thereby optimising classification performance under diverse and challenging conditions. This model, paired with the mixed training strategy, achieved 95.88% accuracy for nine gestures and robust generalisation to unseen arm positions, significantly outperforming single-modality and conventional methods.To ensure the data from this novel hardware is reliable and the system is practical for real-world use, a systematic evaluation based on Data Quality Indices (DQI) was performed. This thesis has proposed and validated two DQI-guided sensing channel selection methods, one ML-based channel selection and another correlation-based channel selection, to identify an optimal, minimal subset of channels or sensors. These methods successfully identified a minimal three-pair sensor set that achieved an accuracy comparable to that of the full eight-channel sEMG + pFMG system, improving wearability and comfort.This thesis makes substantial contributions by providing a robust, integrated framework that includes multi-modal sensor design, advanced ML strategies, novel training methodologies, and signal quality evaluation. Collectively, these contributions significantly advance the capabilities of wearable HGR systems and deepen the understanding of multi-modal fusion of human biosignals, laying the groundwork for future research and development of intuitive, reliable, and user-friendly gesture-based HMI systems suitable for real-life applications.