Investigating the differences in student reporting behavior across different world regions: A synthesis of different reporting behavior measures

Abstract

International large-scale assessment (ILSA) studies like Programme for International Student Assessment (PISA) are an important source of information for educators, educational policy-makers and other stakeholders across the world. Given the importance of ILSA studies' data for contemporary education, it is necessary that we measure different types of student outcomes accurately. To measure various concepts related to student's attitudes, beliefs, and learning experiences, ILSA studies' questionnaires commonly employ self-assessment questions with rating scales. However, there has been a long term concern about the comparability of such data due to the differences in reporting behavior among different groups of respondents (e.g. Kyllonen & Bertling,

2013). Two students with the same objective level of a given concept (e.g. self-concept, motivation for learning) might assess themselves on a scale using different categories - e.g. one as agree, the other as disagree. This might lead researchers to inaccurate conclusions about the actual level of the concept of these students, making the comparison of such data challenging. Several methodological approaches have been suggested to identify and adjust for the differences in reporting behavior among respondents such as the identification of response styles irrespective of item content (hereinafter referred to as response style identification; Baumgartner & Steenkamp,

2001), anchoring vignette method (AVM; King et al.,

2004), and the overclaiming technique (OCT; Paulhus et al.,

2003). Previous studies using these approaches separately indicated that there are marked differences in the reporting behavior between respondents from different countries and world regions (e.g. Buckley, 2009; Vonkova, Papajoanu, & Stipek,

2018). Also, notable similarities in reporting behavior of respondents within certain world regions have been identified (He & van de Vijver, 2016; Vonkova, Papajoanu, & Stipek,

2018). However, there is limited knowledge concerning (a) how the measures of reporting behavior identified using these approaches are mutually related and (b) whether there are any systematic patterns in the reporting behavior in different world regions when analyzing these measures together. In this presentation, we contribute to filling this gap in the current research. The aim of this presentation is to examine the patterns of reporting behavior among different countries and world regions using response style identification, anchoring vignettes method, and the overclaiming technique. We use country-level values of reporting behavior measures identified using these three approaches. The values of the measures were collected from published papers and were all based on the data from PISA 2012 study. Overall, we use data for 63 countries/economies. For response style identification, we use extreme response style (ERS) values from He and van de Vijver (2016). The measure was computed based on student responses to 15 randomly selected attitudinal questions (e.g. math anxiety, self-efficacy) with four-point scale in PISA student questionnaire. Extreme responses were recoded as 1 and mid-point responses were coded as 0 - the mean taken as indicator of ERS. For the anchoring vignette method, we took the threshold parameters estimated using the compound hierarchical ordered probit model from Vonkova, Zamarro, and Hitt (2018), who used PISA anchoring vignette data on teacher's classroom management from student questionnaire. These thresholds between adjacent scale-points

1) "strongly disagree",

2) "disagree",

3) "agree",

4) "strongly agree" represent implicit evaluation standards of students in particular countries. For example, a lower value of the second threshold means that students in a particular country tend to assess the given level of classroom management by the scale category "agree" rather than "disagree". Finally, for the overclaiming technique, we used the values of index of exaggeration (IE) and index of accuracy (IA) from Vonkova, Papajoanu, and Stipek (2018), who used student reports of familiarity with existing OCT items (e.g. cosine, polygon) in the domain of mathematics and non-existing items (e.g. declarative fraction). Higher IA values mean that students in a particular country better discriminate between existing and non-existing OCT items, while higher IE values mean higher overall tendency to report familiarity with both existing and non-existing OCT items. Our preliminary analysis has revealed that the extreme response style identified using 15 different randomly chosen attitudinal questions (ERS) is related to the preference of the use of the extreme scale categories identified by the anchoring vignette method (AVM). Moreover, the higher values of the index of exaggeration (IE) identified by the overclaiming technique relate to the preference of higher scale point categories (especially choosing "disagree" rather than "strongly disagree") identified by the anchoring vignette method (AVM). This preliminary analysis points to an interesting pattern of mutual relationships between these measures, warranting further examination. Further, we used a cluster analysis to reveal systematic patterns in reporting behavior across the world regions. Some world regions show similar patterns of reporting behavior (using all our measures). For example, Northern European countries such as Sweden show similar reporting behavior like some countries in Oceania such as Australia. On the other hand, there are significant differences in reporting behavior across other world regions. The reporting behavior in some countries in South America like Brazil is significantly different from countries in North America like the USA. Countries in South America tend to use, for example, extreme scale category "strongly agree" rather than "agree" (as identified by the AVM) more often than countries in North America. The accurate cross-country comparison of student outcomes related to attitudes, beliefs, and learning experiences requires that we take into account the differences in reporting behavior among students from different countries. We show that: (a) there appears to be a pattern of relationships between different reporting behavior measures which requires further methodological clarification; (b) there are systematic patterns of reporting behavior of students from different world regions as identified by the different reporting behavior measures, which might indeed lead to the problems with data comparability. Investigating the merits of different methodological approaches in the identification of and the adjustment for the differences in reporting behavior and their mutual relationships thus represent a priority in comparative research using ILSA studies' data.