Is learning by reading a book better than watching a movie? A computational analysis of semantic concept network growth during text and multimedia comprehension

In this paper we present a study that compares the semantic networks of text comprehension and multimedia comprehension. This comparison is based on the concept learning (CL) model of comprehension. The model, much like artificial neural networks models, mimics the comprehension processes of the human brain. We conducted a human study for the purpose of revealing the semantic variations in comprehending text and comprehending audio-video multimedia. Each participant in the study created a concept semantic network of what they understand, and these networks were processed by the CL-model. The parameters of the CL-model give us insights into the collective learning of the two groups as well as personal performance of each individual. The model metrics are analyzed to reveal quantitative and qualitative differences. The combination of computational modeling of comprehension with semantic networks analysis, makes us able to measure comprehension performance of reader and watchers in a way that was not possible before. Some of the important results that we found indicate that textual media provided easier integration of newly learned concepts with background information. At the same time, we found that recognizing an overwhelming number of concepts is easier with audio-video multimedia. The presented results are important for media creators and educators, as well as artificial intelligence scientists who aim at creating systems that resemble human learning. Similar to the way biology inspired statistical learning algorithms, studying cognitive tasks, such as comprehension, can help us understand human behavior and build systems that imitate human learning.