Published and Working Papers

Voňková, H., Bendl, S., & Papajoanu, O. (2016). How students report dishonest behavior in school: Self-assessment and anchoring vignettes. The Journal of Experimental Education, DOI: 10.1080/00220973.2015.1094438.

The authors have studied heterogeneity in reporting behavior and its impact on the analysis of self-reports about students’ dishonest behavior in schools. 265 randomly-chosen, seventh-grade students (typically 12 years old) from lower secondary schools in Prague 6, a district in the capital of the Czech Republic, participated in this survey. The results of the self-reports, adjusted for heterogeneity are highly related to students’ levels of academic achievement and their parents’ education and partly to gender while unadjusted self-reports are only slightly related to the level of parents’ education. The authors also show differences in the reporting behavior across diverse subdomains of school behavior and suggest using anchoring vignettes closely related to the domain described in the self-reports.

Vonkova, H., & Hrabak, J. (2015). The (in) comparability of ICT knowledge and skill self-assessments among upper secondary school students: The use of the anchoring vignette method. Computers & Education, 85, 191-202.

Self-reporting of ICT knowledge and skills is commonly used in questionnaire surveys among students and in job search situations to provide information about actual ICT knowledge and skills. The main advantage of this approach is its low cost. The responses can, however, reflect not only the actual level of knowledge and skills but also the self-assessment style. Two students with the same actual knowledge and skills level may give different self-assessments since one might be optimistic and overestimate his/her skills and the other could be pessimistic and underestimate his/her skills. The anchoring vignette method helps to adjust self-reports for the differences in scale usage. We compare the ICT knowledge and skills of two distinct groups of Czech upper secondary school students, ICT students (N = 228) and non-ICT (business and pedagogy) students (N = 147), based on their self-reported general ICT knowledge and skills, before and after the anchoring vignette adjustment for the different usage of scale. We show that the anchoring vignette method helps to distinguish between the two groups of students’ differences in scale usage and that adjusted self-reports correspond to the actual level of students’ ICT knowledge and skills (in contrary to the unadjusted self-reports). The anchoring vignette method is a very promising tool for increasing the validity of surveys that use students’ ICT knowledge and skill self-reports.

Vonkova, H., Zamarro, G., DeBerg, V., & Hitt, C. (2015). Comparisons of student perceptions of teacher’s performance in the classroom: Using parametric anchoring vignette methods for improving comparability. Working paper.

Self-reports are an indispensable source of information in education research but might be affected by reference group bias if the frame of reference (i.e. implicit standards), used to answer the questions, differs across students. The anchoring vignettes method was introduced, in other areas of social science, precisely to correct for this source of bias. However, studies that make use of this approach in education are rare and more research is needed to study its potential. This paper uses data from PISA 2012 to investigate the use of the parametric model of the anchoring vignettes method to correct for differential implicit standards in cross-country comparisons of student’s perceptions of an important dimension of teacher quality: teacher’s classroom management. Our results show significant heterogeneity in implicit standards across countries. We also show how correlations between countries’ average teacher classroom management levels and external variables can be improved substantially when heterogeneity in implicit standards is adjusted for. We conclude that the anchoring vignettes method shows a good potential to enhance the validity and comparability of self-reported measures in education.

Vonkova, H., & Sramkova, E. (2013). Zjišťování porozumění textu v angličtině: Jaké možnosti dává využití metody ukotvujících vinět? In: L. Círus (Eds.), Efektivita vzdělávání v proměnách společnosti: Sborník příspěvků XXI. celostátní konference České asociace pedagogického výzkumu (pp. 92-97). Ústí nad Labem: Univerzita J. E. Purkyně.

To find out knowledge of foreign language, self-assessment questions are sometimes used apart from the didactic tests. Their use, however, is debatable. Somebody may for example assess their knowledge as very good and somebody else as average, although in reality their knowledge is the same. The self-assessment of the people is not comparable then. This article is concerned with the possibility of the use of the anchoring vignette method in the correction of self-assessment of reading comprehension in English, so that it is comparable. With this method, the respondents answer a self-assessment question, and alongside all of them are to assess the same anchoring vignettes, i.e. short stories describing knowledge of a hypothetical person. The differences in their vignette assessment, which reflects respondents’ different perception of scale is then used for self-assessment correction. A nonparametric approach with the use of four vignettes is described. A relation between the self-assessment and corrected self-assessment using anchoring vignettes and a relation between the (corrected) self-assessment and the result of an English reading comprehension didactic test is shown on the example of secondary school students. The assessment of vignettes showed high variability among students, which indicates their different use of particular points on scale. Thus, the corrected self-assessment differs from the original self-assessment. In some cases the corrected self-assessment accords much more with the objective level of reading comprehension – the results of the didactic test.

Vonkova, H. (2013). Subjektivní hodnocení problémů s pohybem: Užití parametrického modelu metody ukotvujících vinět. Orbis Scholae, 7 (1), 49-66.

In survey research, respondents’ self-assessment is often used to compare the objective state across different groups. The problem is that self-assessments may depend on both the respondents’ objective situation and the way in which they interpret scale categories. The anchoring vignette method offers a solution of this problem. Using evaluation of hypothetic persons in short stories (anchoring vignettes) the heterogeneity of the scale use is identified and used for correction of the response scale differences. This article aims at identification of heterogeneity in scale use, when respondents’ problems with moving around are being studied. The sample consists of people aged fifty or older from ten European countries. Data collection was done in 2006−2007. It is shown that Belgian and Italian respondents are more optimistic when evaluating problems with moving around. Czech and Dutch respondents use the end points of the scale less often. After the correction for heterogeneity in reporting behavior, the best actual situation is found in Denmark, France and Sweden, while Czech and Polish seniors are the least mobile. Younger men with higher education have less problems with moving around. After the correction of reporting behavior the differences between both men and women and respondents with higher and lower education increase because men and respondents with higher education perceive a given problem as worse with higher probability.

Vonkova, H. (2012). Metoda ukotvujicích vinět a možnosti využití v pedagogice. Orbis Scholae, 6 (1), 27-40.

Self-assessment questions are used often in survey research. A direct comparison of countries or different socioeconomic groups based on answers to such questions may lead to an incorrect conclusion about the respondents’ actual situation. One of the reasons may be that respondents from different groups use the response scale in different ways, for example Americans may evaluate a situation as “excellent” more often compared to other nationalities. The anchoring vignette method solves the problem by letting respondents evaluate both their situation and hypothetical situations of other people (anchoring vignettes), Since the hypothetical situation is invariant over respondents, the differences in their evaluations of the hypothetical situation may be interpreted as the differences in their use of response scale. This is then used to adjust (anchor) their self-assessments. The anchoring vignettes method has not yet been discussed in Czech language academic literature. This paper introduces the basic idea of the method both intuitively and using mathematics and presents several applications of the method in social sciences. Least but not last, it suggests how to use this method in educational science.

Contact

Pedagogická fakulta Univerzita Karlova
M. Rettigové 4, Praha 1
2. a 4. patro

Institution: Charles University, Faculty of Education
Street: Magdaleny Rettigove 4
City: Prague
ZIP/Postal code: 116 39
State: Czech Republic

 

Contact person:
Director: Hana Vonkova
Email: hana.vonkova@pedf.cuni.cz