Developing a holistic AI literacy assessment matrix – Bridging generic, domain-specific, and ethical competencies

Motivated by a holistic understanding of AI literacy, this work presents an interdisciplinary effort to make AI literacy measurable in a comprehensive way, considering generic and domain-specific AI literacy as well as AI ethics. While many AI literacy assessment tools have been developed in the last 2-3 years, mostly in the form of self-assessment scales and less frequently as knowledge-based assessments, previous approaches only accounted for one specific area of a comprehensive understanding of AI competence, namely cognitive aspects within generic AI literacy. Considering the demand for AI literacy development for different professional domains and reflecting on the concept of competence in a way that goes beyond mere cognitive aspects of conceptual knowledge, there is an urgent need for assessment methods that capture domain-specific AI literacy on each of the three competence dimensions of cognition, behavior, and attitude. In addition, competencies for AI ethics are becoming more apparent, which further calls for a comprehensive assessment of AI literacy for this very matter. This conceptual paper aims to provide a foundation upon which future AI literacy assessment instruments can be built and provides insights into what a framework for item development might look like that addresses both generic and domain-specific aspects of AI literacy as well as AI ethics literacy, and measures more than just knowledge-related aspects based on a holistic approach.