The influence of the selected school characteristics on teacher's classroom management in European countries: student's assessments and anchoring vignettes
Abstract
Teacher's classroom management is an important element of student academic success and its improvement is a major concern among educators, parents and policy makers. One of the method to estimate teacher's classroom management level is to ask the students using a questionnaire.
Students are first hand witnesses of what occurs in the classrooms and so they are in a privileged position to provide feedback on teaching practices (Brimble & Stevenson-Clarke, 2005; Vonkova et al., 2015). In general, data from student surveys is a central source of information in education research.
However, comparisons of self-reported measures across students in different countries, regions, schools, classrooms, or different socio-economic groups can be biased if respondents differ in their use of scales in the provided questions. Several studies have drawn attention to the heterogeneity in reporting behavior among respondents from different cultural backgrounds (e.g., Buckley, 2009; Clarke, 2001; Chen, Lee, & Stevenson, 1995).
The anchoring vignette method has been introduced in the social sciences to adjust for such heterogeneity in reporting behavior and obtain comparable responses across groups (King et al., 2004). Vonkova et al. (2015) show a substantial heterogeneity in student's perceptions of teacher's classroom management reporting behavior across PISA 2012 countries.
In some countries, students have higher standards for judging teacher behavior and therefore such countries improve their relative position in the ranking of teachers' classroom management skills after adjusting for heterogeneity in reporting behavior. In Europe, this is the case of for example Norway, the Netherlands and Denmark.
In contrary, students in other countries have lower standards and therefore such countries worsen their relative position after adjustment. This is the case of, for example, Romania, Albania and Bulgaria.
It has been shown in the literature that the heterogeneity in reporting behavior may play a substantial role not only in between-country level but also on within-country level (Bago d'Uva et al., 2008; Vonkova & Hrabak, 2015; Vonkova, Bendl & Papajoanu, forthcoming). In this paper, we focus on the relationship of selected school variables with teacher's classroom management and show whether they play a substantial / minor role when the heterogeneity in reporting behavior is accounted for.
The selected school variables are the following: a) private versus public school, b) size of school location, c) competition between schools in the school's location (number of schools in the school's area that compete for students), d) classroom size, e) using achievement data in accountability procedures like posting them publicly, f) teacher salary change based on teacher appraisal, g) proportion of maths teachers with ISCED 5A (maths as major). The analysis is done for all 33 European countries participating in PISA 2012.
Our research questions are: 1) To what extent is the heterogeneity in the student's reporting behavior influenced by the selected school factors in European countries participating in PISA 2012? 2) What is the influence of the selected school factors on teacher's classroom management before and after adjustment for the heterogeneity in reporting behavior across European countries? What selected school factors play substantial role in teacher's classroom management across European countries when (not) adjusted for the heterogeneity? 3) In what European countries is the heterogeneity in reporting behavior most / less substantial? Method Responses in questionnaire surveys may reflect not only the objective situation we seek to measure but also differences in the scale usage. The anchoring vignette method is used for the correction of the differences in the scale usage.
The method is based on asking the respondents not only to assess their situation, but also assess hypothetical situations described by anchoring vignettes. Anchoring vignettes are short texts describing hypothetical individuals who manifest the trait of interest.
Since all the respondents assess the same anchoring vignettes, we can interpret the heterogeneity in their assessments as different scale usage. Based on the measured differences, we can adjust the student's responses to their own situations and get more accurate comparisons.
This paper focuses on the study of the relationship between teacher's classroom management and schools with different characteristics among European countries. Specifically, we use PISA 2012 student questionnaire which included the following questions on teacher classroom management: a) my teacher gets students to listen to him or her, b) my teacher keeps the class orderly, c) my teacher starts lessons on time, and d) the teacher has to wait a long time for students to quiet down.
Students were giving their answers on a four-point Likert-type scale: strongly agree, agree, disagree, strongly disagree. As mentioned above, students at schools with different characteristics and in different countries may however interpret the scale categories differently.
To correct for these differences, PISA 2012 introduced the anchoring vignettes related to teacher's classroom management. Students were asked to rate to what extent they agree that the teacher is in control of his/her classroom: Vignette 1 (High level): The students' in Ms. class are calm and orderly.
She always arrives on time to class. Ms. is in control of her classroom.
Vignette 2 (Medium level): The students' in Ms. class frequently interrupt her lessons. She always arrives five minutes early to class.
Ms. is in control of her classroom. Vignette 3 (Low level): The students' in Mr. class frequently interrupt his lessons.
As a result, he often arrives five minutes late to class. Mr. is in control of his classroom.
Expected Outcomes The heterogeneity in student's perceptions of teacher's classroom management reporting behavior across school factors plays a substantial role. Before the adjustment for the reporting behavior, a lower share of math teacher with ISCED 5A seemed to improve students' perception of teacher's classroom management.
In fact, the share of high quality math teachers seemed to be one of the most important factors, but the vector of the influence (good teachers making the situation worse) was counter-intuitive. After the adjustment for the reporting behavior, the share of ISCED 5A math teachers plays much less role.
Our interpretation is that students at schools with low proportion of good math teachers have typically lower standards for evaluating the level of teacher's classroom management (i.e. evaluate a situation as 'excellent' when other would say 'good'). The most important school-level determinants of classroom management level are a) how much the school needs to compete for students with other schools, and b) whether the school is private.
Competition between schools has a significantly positive effect on classroom management level in 11 countries and a negative one in 6 countries. Being a private school has a significant positive effect in 14 countries, significant negative effect was identified only in 1 country.
The least influential factors were teacher salary change based on teacher appraisal and proportion of math teacher with ISCED 5A. Concerning the importance of reporting behavior adjustment in individual countries, there is only one country (Lithuania) where the significance of the school factors remain unchanged after the adjustment of the heterogeneity in reporting behavior.
The biggest change in the significant influence of the selected school factors on classroom management after adjustment for the heterogeneity is found in Slovenia, Montenegro, Germany, Denmark, France, Island, Poland, Portugal, Serbia and Slovakia. References Buckley, J. (2009).
Cross-national response styles in international educational assessments: Evidence from PISA 2006. New York University: Department of Humanities and Social Sciences in the Professions.
Retrieved from https://edsurveys.rti.org/PISA/documents/Buckley PISAresponsestyle.pdf Bago d'Uva, T., van Doorslaer, E., Lindeboom, M., & O'Donnell, O. (2008). Does reporting heterogeneity bias the measurement of health disparities? Health Economics, 17(3), 351-375.
Brimble, M., & Stevenson-Clarke, P. (2005). Perceptions of the prevalence and seriousness of academic dishonesty in Australian universities.
The Australian Educational Researcher, 32(3), 19-44. Clarke, L. (2001).
Extreme response style in cross-cultural research. International Marketing Review, 18(3), 301-324.
Chen, Ch, Lee, S., & Stevenson, H. W. (1995).
Response style and cross-cultural comparisons of rating scales among East Asian and North American students. Psychological Science, 6(3), 170-175.
King, G., Murray, C. J.
L., Salomon, J. A., & Tandon, A. (2004).
Enhancing the validity and cross-cultural comparability of measurement in survey research. American Political Science Review, 98(1), 567-583.
Vonkova, H., Bendl, S., Papajoanu, (2015). O.
How students report dishonest behavior in school: Self-assessment and anchoring vignettes. Journal of Experimental Education, forthcoming.
Vonkova, H., & Hrabak, J. (2015). The (in) comparability of ICT knowledge and skill self-assessments among upper secondary school students: The use of the anchoring vignette method.
Computers & Education, 85 , 191-202. Vonkova, H., Zamarro, G., & DeBerg, V. (2015).
Comparisons of student perceptions of teacher's performance in the classroom: Using parametric anchoring vignette methods for improving comparability. Retrieved from https://www.researchgate.net/publication/273785729_Comparisons_of_Student_Perceptions_of_Teacher's_Performance_in_the_Classroom_Using_Parametric_Anchoring_Vignette_Methods_for_Improving_Comparability_PRELIMINARY_VERSION