A conversation-analytic hermeneutic rating protocol to assess L2 oral pragmatic competence

Speech act theory-based, second language pragmatics testing (SLPT) poses problems for validation due to a lack of correspondence with empirical conversational data. Since conversation analysis (CA) provides a richer and more accurate account of language behavior, it may be preferred as a basis for SLPT development. However, applying CA methodology in turn poses epistemological and practical challenges to psychometrics-driven language testing. The present pilot study, attempting to resolve this seeming conflict, focuses on the behavior of two CA-trained raters applying a holistic rubric to responses on a test of ESL oral pragmatic competence. Results showed that CA-informed testing (CAIT) could be practical, though statistical reliability was not achieved. However, post-rating hermeneutic dialogues between the raters provided evidence that valid inferences of examinee oral ESL pragmatic ability could be made through iterative, rater recourse to empirical data in a conversation-analytic mode. A model for iterative, CA-informed SLPT development is offered.