Validation of the Strategy for Determining the Numerical Rating of the Cognitive Complexity of Exam Items in the Field of Chemical Kinetics
Abstract
The main goal of this study was to validate the strategy for the assessment of the cognitive complexity of chemical kinetics exam items. The strategy included three steps: 1) assessment of the difficulty of concepts, 2) assessment of distractor value. and 3) assessment of concepts’ interactivity. One of the tasks was to determine whether there were misconceptions by students that might have influenced their achievement. Eighty-seven students in the first year of secondary school participated in the study. A knowledge test was used as a research instrument to assess the performance, and a five-point Likert-type scale was used to evaluate the perceived mental effort. The strategy was validated using regression analysis from which significant correlation coefficients were obtained between selected variables: students’ achievement and invested mental effort (dependent variables) and a numerical rating of cognitive complexity (independent variable).
Downloads
References
Bain, K., & Towns, M. (2016). A review of research on the teaching and learning of chemical kinetics. Chemistry Education Research and Practice, 17(2), 246–262. https://doi.org/10.1039/C5RP00176E
Banerjee, A. (1991) Misconceptions of students and teachers in chemical equilibrium. International Journal of Science Education, 13(3), 355–362. https://doi.org/10.1080/0950069910130411
Banks, P. J. (1997). Students’ understanding of chemical equilibrium. Unpublished MA thesis. Department of Educational Studies, University of York.
Barczi, K. (2013). Applying cooperative techniques in teaching problem solving. Center for Educational Policy Studies Journal, 3(4), 61–78. https://doi.org/10.26529/cepsj.223
Bieri, J. (1955). Cognitive complexity–simplicity and predictive behaviour. Journal of Abnormal and Social Psychology, 51(2), 263–268.
BouJaoude, S., & Barakat, H. (2000). Secondary school students’ difficulties with stoichiometry, School Science Review, 81(296), 91–98.
Brace, N., Kemp, R., & Snelgar, R. (2006) SPSS for Psychologists: A guide to data analysis using SPSS for Windows (3rd ed.). Routledge.
Çakmakci, G. (2010). Identifying alternative conceptions of chemical kinetics among secondary school and undergraduate students in Turkey. Journal of Chemical Education, 87(4), 449–455. https://doi.org/10.1021/ed8001336
Çam, A., Topçu, M. S., & Sülün, Y. (2015). Preservice science teachers’ attitudes towards chemistry and misconceptions about chemical kinetics. Asia-Pacific Forum on Science Learning and Teaching, 16(2), 1–6.
Cliff, W. H. (2009). Chemistry misconceptions associated with understanding calcium and phosphate homeostasis. Advance in Physiology Education, 33(4), 323–328. https://doi.org/10.1152/advan.00073.2009
Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences. Lawrence Erlbaum Associates.
Daniel, R. C.. & Embretson, S.E. (2010). Designing cognitive complexity in mathematical problem-solving items. Applied Psychological Measurement, 34(5), 348–364. https://doi.org/10.1177/0146621609349801
Embretson, S. E., & Daniel, R. C. (2008). Understanding and quantifying cognitive complexity level in mathematical problem solving items. Psychological Science, 50, 328–344.
Evans, J. D. (1996). Straightforward statistics for the behavioral sciences. ‎Brooks/Cole.
Goldreich, O. (2008). Computational complexity: A conceptual perspective. Cambridge University Press.
Gorodetsky, M., & Gussarsky, E. (1986) Misconceptualisation of the chemical equilibrium concept as revealed by different evaluation methods. European Journal of Science Education, 8(4), 427–441. https://doi.org/10.1080/0140528860080409
Hackling, M.W., & Garnett, P. (1985). Misconceptions of chemical equilibria. European Journal of Science Education, 7(2), 205–214. https://doi.org/10.1080/0140528850070211
Horvat, S. A, Mihajlović, J., RonÄević, T. N., & Rodić, D. D. (2021). Procedure for the assessment of cognitive complexity: Development and implementation in the topic "Hydrolysis of Salts". Macedonian Journal of Chemistry and Chemical Engineering, 40(1), 119–130. https://doi.org/10.20450/mjcce.2021.2240
Horvat, S. A. , RonÄević, T. N., Arsenović, D. Z , Rodić, D. D., & Segedinac, M. D. (2020). Validation of the procedure for the assessment of cognitive complexity of chemical technology problem tasks. Journal of Baltic Science Education, 19(1), 64–75. https://doi.org/10.33225/jbse/20.19.64
Horvat, S., Rodić, D. D., Segedinac, M. D., & RonÄević, T. N. (2017). Evaluation of cognitive complexity of tasks for the topic hydrogen exponentin the solutions of acids and bases. Journal of Subject Didactics, 2(1), 33–45. https://doi.org/10.5281/zenodo.1238972
Horvat, S., Segedinac, M. D., Milenković, D. D., & Hrin, T.N. (2016). Development of procedure for the assessment of cognitive complexity of stoichiometric tasks. Macedonian Journal of Chemistry and Chemical Engineering, 35(2), 275–284. https://doi.org/10.20450/mjcce.2016.893
Johnstone, A. H., Macdonald, J. J., & Webb, G. (1977). Chemical equilibria and its conceptual difficulties, Education in Chemistry, 14(6), 169– 171.
Jonsson, A., & Svingby, G. (2002). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research and Reviews, 2(2), 130–144. https://doi.org/10.1016/j.edurev.2007.05.002
Justi, R. (2002), Teaching and learning chemical kinetics. In J. K. Gilbert, De O. Jong, R. Justi, D. Treagust , & J. H. Van Driel (Eds.), Chemical Education: Towards Research based Practice (pp. 293–315). Kluwer.
Kalainoff, M., Lachance, R., Riegner, D., & Biaglow, A. A. (2012). Computer algebra approach to solving chemical equilibria in general chemistry. PRIMUS, 22(4), 284–302.
Kalyuga, S. (2008). Managing cognitive load in adaptive multimedia learning. information science reference. Hershey.
Kelly, G. A. (1955). The psychology of personal construct. A theory of personality. Taylor & Francis.
Kim, S. J., Aleven, V., & Dey, A. K. (2014). Understanding expert-novice differences in geometry problem-solving tasks. CHI '14 Extended Abstracts on Human Factors in Computing Systems. https://doi.org/10.1145/2559206.2581248
Knaus, K., Murphy, K., Blecking, A., & Holme, T. (2011). A valid and reliable instrument for cognitive complexity rating assignment of chemistry exam items. Journal of Chemistry Education, 88(5), 554–560. https://doi.org/10.1021/ed900070y
Kousathana, M., & Tsaparlis, G., (2002), Students’ errors in solving numerical chemical equilibrium problems. Chemistry Education Research and Practice, 3(1), 5–17 https://doi.org/10.1039/B0RP90030C
Loewenthal, K. M. & Lewis, C. A. (2001). An introduction to psychological tests and scales. Psychology Press. https://doi.org/10.4324/9781315782980
Mayers, A. (2013). Introduction to statistics and SPSS in psychology. Pearson Education.
Maskill, R., & Cachapuz, A. F. C. (1989). Learning about the chemistry topic of equilibrium: The use of word association tests to detect developing conceptualisations. International Journal of Science Education, 11(1), 57–69. https://doi.org/10.1080/0950069890110106
Morgan, G. A., Leech, N. L., Gloackener, G. W., & Barret, K. C. (2011). BM SPSS for introductory statistics: Use and interpretation. Routledge.
Moss, S., Prosser, H., Costello, H., Simpson, N., Patel, P., Rowe, S.,Tuner, S., & Hatton, C. (1998). Reliability and validity of the PAS-ADD checklist for detecting psychiatric disorders in adults with intellectual disability. Journal of Intellectual Disability Research, 42(2), 173–183. https://doi.org/10.1046/j.1365-2788.1998.00116.x
Å urjanović, M., & Nikolajević, R. (2011). Hemija - Zbirka zadataka iz hemije - za 1. i 2. razred gimnazije prirodno-matematiÄkog smera, medicinsku i poljoprivrednu Å¡kolu [Chemistry - Collection of tasks in chemistry - for the 1st and 2nd grade of the gymnasium of natural mathematics, medical and agricultural school]. Zavod za udžbenike i nastavna sredstva.
Pande, S. S., Pande, R. P., Parate, V. P., Nikam, A. N., & Agrekar, S. H. (2013). Correlation between difficulty and dis-crimination indices of MCQs in formative exam in physiology. South-East Asian Journal of Medical Education, 7(1), 45–50. https://doi.org/10.4038/seajme.v7i1.149
Pippenger, N. (1978). Complexity theory. Scientific American, 238(6), 114–125.
Raker, J. R., Trate, J. M., Holme, T. A.. & Murphy, K. (2013). Adaptation of an instrument for measuring the cognitive complexity of organic chemistry exam Items. Journal of Chemical Education, 90(10), 1290–1295. https://doi.org/10.1021/ed400373c
Segedinac, M., Segedinac, M., Konjović, Z., & Savić, G. (2011). A formal approach to organization of educational objectives. Psihologija, 44(4), 307–323. https://doi.org/10.2298/PSI1104307S
Segedinac, M. T., Horvat, S., Rodić, D. D., RonÄević, T. N., & Savić, G. (2018). Using knowledge space theory to compare expected and real knowledge spaces in learning stoichiometry. Chemistry Education Research and Practice, 19(3), 670–680. https://doi.org/10.1039/C8RP00052BC
Soeharto, S., Csapó, B., Sarimanah, E., Dewi, F.I., & Sabri, T. (2019). A review of students’ common misconceptions in science and their diagnostic assessment tools. Jurnal Pendidikan IPA Indonesia, 8(2), 247–266.
Soeharto, S., & Csapó, B. (2021). Evaluating item difficulty patterns for assessing student misconceptions in science across physics, chemistry, and biology concepts. Heliyon, 7(11), e08352. https://doi.org/10.1016/j.heliyon.2021.e08352
Sözbilir M., Pinarbasi T., & Canpolar N. (2010). Prospective chemistry teachers’ conceptions of chemical thermodynamics and kinetics. Eurasian Journal of Mathematics, Science and Technology Education, 6(2), 111–120. https://doi.org/10.12973/ejmste/75232
Sweller, J., Ayres, P., & Kalyuga, S. (2011). Cognitive load theory. Springer.
Taber, K. S., (2002). Chemical misconceptions – prevention, diagnosis and cure: Theoretical background. Royal Society of Chemistry.
Taber, K. S. (2018). The use of Cronbach’s alpha when developing and reporting research instruments in science education. Research in Science Education, 48(6), 1273–1296. https://doi.org/10.1007/s11165-016-9602-2
Tavakol, M., & Dennick, R. (2011). Making sense of Cronbach’s alpha. International Journal of Medical Education, 2, 53–55. https://doi.org/10.5116/ijme.4dfb.8dfd
Turányi, T., & Tóth, Z., (2013). Hungarian university students’ misunderstandings in thermodynamics and chemical kinetics. Chemistry Education Research and Practice, 14(1), 105–116. https://doi.org/10.1039/C2RP20015E
Tóth, Z. (1999), Egy kémiai tévképzet nyomában. Az egyensúlyi állandó bevezetésének lehetöségei és problémái [Tracing a chemical misconception. The challenges and problems of the introduction of the chemical equilibrium constant]. Iskolakultúra, 9, 108–112.
Towns, M. H. (2014). Guide to developing high-quality, reliable, and valid multiple-choice assessments. Journal of Chemical Education, 91(9), 1426−1431. https://doi.org/10.1021/ed500076x
Van Driel, J. H. (2002) Students’ corpuscular conceptions in the context of chemical equilibrium and chemical kinetics. Chemistry Education: Research and Practice, 3(2), 201–213. https://doi.org/10.1039/B2RP90016E
Van Gog, T., Kester., L., & Paas, F. (2011). Effects of concurrent monitoring on cognitive load and performance as a function of task complexity. Applied Cognitive Psychology, 25(4), 584–587. https://doi.org/10.1002/acp.1726
Yan, Y. K., & Subramaniam, R. (2018). Using a multi-tier diagnostic test to explore the nature of students’ alternative conceptions on reaction kinetics. Chemistry Education Research and Practice, 19(1), 213–226 https://doi.org/10.1039/C7RP00143F
Yunus, W. M. D. Z. W., & Ali, Z .M. (2012). Urban Students’ Attitude towards Learning Chemistry. Procedia-Social and Behavioral Sciences, 68, 295–304. https://10.1016/j.sbspro.2012.12.228
Zubairi, A., Lide, N., & Abu Kassim, N. L. (2006). Classical And Rasch Analyses Of Dichotomously Scored Reading Comprehension Test Items. Malaysian Journal of ELT Research, 2, 1–20.
Authors who publish with this journal agree to the following terms:
- Authors are confirming that they are the authors of the submitted article, which will be published online in the Center for Educational Policy Studies Journal (for short: CEPS Journal) by University of Ljubljana Press (University of Ljubljana, Faculty of Education, Kardeljeva ploščad 16, 1000 Ljubljana, Slovenia). The Author’s/Authors’ name(s) will be evident in the article in the journal. All decisions regarding layout and distribution of the work are in the hands of the publisher.
- The Authors guarantee that the work is their own original creation and does not infringe any statutory or common-law copyright or any proprietary right of any third party. In case of claims by third parties, authors commit themselves to defend the interests of the publisher, and shall cover any potential costs.
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under https://creativecommons.org/licenses/by/4.0/deed.en that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.
