Integrating Image and Textual Information in Human–Robot Interactions for Children With Autism Spectrum Disorder

Talking and literary reading are important activities for children, especially for children with autism spectrum disorder (ASD). We try to integrate the activities with NAO robots to excite their communication willingness. In this paper, a novel multimodal picture book recommendation framework that combines textual information and image information to calculate the similarity between the picture books and the conversation topics is proposed and evaluated using a testing dataset. In the proposed framework, an image neighbor discovery method to get more relative terms and an near-duplicated keyframes (NDK) friend detection method to get more relative NDKs are proposed. Finally, the booklist generated from the experiment is evaluated by six performance indicators and the experimental results demonstrate that our proposed framework achieves satisfactory and promising performance. With the help of the proposed recommendation framework, an autistic child can talk to the NAO robot in a relaxed and enjoyable environment. Please note that the proposed framework is not evaluated for its performance with the ASD children but for its performance at recommending books based on visual and textual features. Therefore, no tests were performed with either professionals nor diagnosed individuals.