Examining Paradoxical Associations Between Students' Questionnaire Responses and Their Achievement Across PISA Cycles: The Case of Teacher Support
Abstract
In the context of Programme for International Student Assessment (PISA), paradoxical findings concerning the relationship between questionnaire scales and student achievement are often documented. These questionnaire scales are found to correlate positively with student achievement within countries at the individual level, but negatively at the between-country level (i.e., when correlating questionnaire scale and achievement values aggregated at the level of countries, the countries being the unit of analysis).
These anomalous findings can be caused by the differences in reporting behavior of students in different countries and might lead educators and educational policy-makers to erroneous conclusions. In this paper, we examine the relationship between the teacher support scale and student achievement across three PISA cycles - 2012, 2015, and 2018.
Our results show that there is a consistent negative between-country correlation between teacher support and student achievement in all three examined PISA cycles, which is in line with the previously documented paradoxes. We have also found that some countries, which participated in all three PISA cycles under study, consistently contribute to this paradox by having quite high levels of student-reported teacher support but rather low achievement scores.
Future research should take into account the differences in reporting behavior between students when making cross-country analyses and consider the application of methodological approaches to identify and adjust for these differences such as the anchoring vignette method and the overclaiming technique.