Using Kappa Cohen for Expected Agreement of Math X-Quest Learning Media for in Support of SDGs-4 (Quality Education) Context
Main Article Content
Abstract
Media quality needs to be proven in certain ways, one way is by expert assessment. In this paper, we examine the quality of Math X-Quest media which is assessed by two experts using the Cohen Kappa method. This activity aims to obtain validity in the form of feasibility of the contextual game-based Math X-Quest media that helps students understand the concept of three-variable linear equations through balancing activities, connecting abstract mathematical ideas with concrete experiences. The rater assessment consisted of aspects of the material and learning media. Analysis of the rater results was carried out using descriptive statistics with the Crosstabs technique. The results obtained showed that 2 raters had the same view on the quality of the assessment with a value of 0.69 spearman correlation. Meanwhile, the quality of X-Quest obtained a suitability score of 0.872 from the two expert assessors, indicating a near-perfect Cohen Kappa Agreement index. Thus, it can be concluded that X-Quest is suitable for use in mathematics learning which can support improving the quality of education as envisioned in SDG 4.
Article Details
How to Cite
ARIANTI, Putri Dwi; MUKTI, Taufiq Satria; NUR’AINI, Luthfiah Hamidah.
Using Kappa Cohen for Expected Agreement of Math X-Quest Learning Media for in Support of SDGs-4 (Quality Education) Context.
Proceeding of International Conference on Islamic Education (ICIED), [S.l.], v. 10, n. 1, p. 650-655, dec. 2025.
ISSN 2613-9804.
Available at: <https://conferences.uin-malang.ac.id/index.php/icied/article/view/3644>. Date accessed: 03 feb. 2026.
doi: https://doi.org/10.18860/icied.v10i1.3644.
Section
Articles
References
Abdulrahaman, M. D., Faruk, N., Oloyede, A. A., Surajudeen-Bakinde, N. T., Olawoyin, L. A., Mejabi, O. V, Imam-Fulani, Y. O., Fahm, A. O., & Azeez, A. L. (2020). Multimedia tools in the teaching and learning processes: A systematic review. Heliyon, 6(11), e05312. https://doi.org/10.1016/j.heliyon.2020.e05312
Aiken, L. R. (1985). Three coefficients for analyzing the reliability and validity of ratings. Educational and Psychological Measurement, 45(1), 131–142.
Amalia, L., Makmuri, M., & Hakim, L. El. (2024). Learning Design: To Improve Mathematical Problem-Solving Skills Using a Contextual Approach. JIIP - Jurnal Ilmiah Ilmu Pendidikan, 7(3), 2353–2366. https://doi.org/10.54371/jiip.v7i3.3455
Arsyad, M., Mujahiddin, M., & Syakhrani, A. W. (2024). The efficiency of using visual learning media in improving the understanding of science concepts in elementary school students. Indonesian Journal of Education (INJOE), 4(3), 775–787.
Carvajal, A. L. P., Fernandez, T. M., Pangilinan, A. M., Obod, M. M., Amihan, S. R., Sanchez, R. D., Sanchez, A. M. P., & Sanchez, J. J. D. (2025). Future-Proofing Teachers in Reframing Teacher Education Curriculum in the Philippines: Basis for Policy Recommendations. International Journal of Open-Access, Interdisciplinary and New Educational Discoveries of ETCOR Educational Research Center (IJOINED ETCOR), 4(2), 235–252.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46.
Farani, R. (2013). The Evaluation Of Instructional Multimedia Product For English Learning. Journal of English and Education (JEE).
February, P., Alter, G., Buendgens-Kosten, J., Müller, F. J., & Di Paolo, A. (2025). (DIGITAL) MEDIA AND MATERIALS FOR LEARNING. In All means all!–OpenTextbook for diversity in education (pp. 1115–1148). All-means-all.
Hana, A. Y., Sa’diyah, H., Nahar, Z. K., & Fakhriyana, D. (2024). Pendidikan Berkualitas untuk Mewujudkan Sustainable Development Goals (SDGs) Melalui Pembelajaran Matematika: Systematic Literature Review. ARITHMETIC: Academic Journal of Math, 6(1), 79–100.
Landis, J. R., & Koch, G. G. (1977). The Measurement of Observer Agreement for Categorical Data. Biometrics, 33(1), 159. https://doi.org/10.2307/2529310
Ling, J. H. (2025). A Review of Rubrics in Education: Potential and Challenges. Indonesian Journal of Innovative Teaching and Learning, 2(1), 1–14. https://doi.org/10.64420/ijitl.v2i1.197
MacKay, R. S., Kenna, R., Low, R. J., & Parker, S. (2017). Calibration with confidence: a principled method for panel assessment. Royal Society Open Science, 4(2), 160760.
Manik, I. M. S. B., Rizal, C., & Kurniawan, H. (2025). ANALISIS AKURASI DAN KEANDALAN METODE CERTAINTY FACTOR PADA SISTEM PAKAR DIAGNOSTIK MEDIS. Jurnal Multimedia Dan Teknologi Informasi (Jatilima), 7(03), 702–707.
McHugh, M. L. (2012). Interrater reliability: the kappa statistic. Biochemia Medica, 22(3), 276–282.
Olasoji, O. V., Ugwu, U. F., & Onoh, D. O. (2023). The Role of Mathematics Education in Achieving Sustainable Development Goals (SDGs). ESUT Journal of Education, 6(1), 247–255.
Prasetyowati, D. (2020). Development of Learning Videos for Junior High School Math Subject to Enhance Mathematical Reasoning. International Journal of Education and Practice, 8(1), 18–25.
Prawiro, F., & Vereen, A. (2024). REPRESENTASI SOSIAL TENTANG GANGGUAN JIWA PADA MASYARAKAT YANG INGIN MENCARI BANTUAN PSIKOLOGIS FORMAL (STUDI PADA MASYARAKAT BANTEN YANG RELATIF MODERN). MANASA, 13(2), 74–91. https://doi.org/10.25170/manasa.v13i2.6099
Reinke, L. T., & Casto, A. R. (2022). Motivators or conceptual foundation? Investigating the development of teachers’ conceptions of contextual problems. Mathematics Education Research Journal, 34(1), 113–137.
Sachdeva, S., & Eggen, P.-O. (2021). Learners’ Critical Thinking About Learning Mathematics. International Electronic Journal of Mathematics Education, 16(3), em0644. https://doi.org/10.29333/iejme/11003
Saharani, T., & Abadi, A. M. (2024). The Development of Interactive Learning Media Based on Discovery Learning Oriented to Students’ Concept Understanding and Mathematical Disposition. AL-ISHLAH: Jurnal Pendidikan, 16(4). https://doi.org/10.35445/alishlah.v16i4.5681
Sahu, S., & Demirtas, H. (2024). A Linear Relationship between Correlation and Cohen’s Kappa for Binary Data and Simulating Multivariate Nominal and Ordinal Data with Specified Kappa Matrix. ArXiv Preprint ArXiv:2404.14149.
Smith, N. M., Hoal, K. E. O., & Thompson, J. F. H. (2020). Ensure inclusive and equitable quality education and promote lifelong learning opportunities for all. In Mining, materials, and the sustainable development goals (SDGs) (pp. 29–38). CRC Press.
United Nations. (2024). Sustainable Development Goal 4: Quality Education. https://www.un.org/sustainabledevelopment/education
Warrens, M. J. (2014). New Interpretations of Cohen’s Kappa. Journal of Mathematics, 2014, 1–9. https://doi.org/10.1155/2014/203907
Warrens, M. J. (2015). Five ways to look at Cohen’s kappa. Journal of Psychology & Psychotherapy, 05(04). https://doi.org/10.4172/2161-0487.1000197
Widiana, I. W., Tegeh, I., & Parwata, I. (2020). Improving Students’ Factual Knowledge with Concrete Media through Observing Activities in Scientific Approaches in Elementary Schools. Journal of Education and E-Learning Research, 7(3), 293–299.
Zamanzadeh, V., Ghahramanian, A., Rassouli, M., Abbaszadeh, A., Alavi-Majd, H., & Nikanfar, A.-R. (2015). Design and Implementation Content Validity Study: Development of an instrument for measuring Patient-Centered Communication. Journal of Caring Sciences, 4(2), 165–178. https://doi.org/10.15171/jcs.2015.017
Zec, S., Soriani, N., Comoretto, R., & Baldi, I. (2017). High Agreement and High Prevalence: The Paradox of Cohen’s Kappa. The Open Nursing Journal, 11(1), 211–218. https://doi.org/10.2174/1874434601711010211
Aiken, L. R. (1985). Three coefficients for analyzing the reliability and validity of ratings. Educational and Psychological Measurement, 45(1), 131–142.
Amalia, L., Makmuri, M., & Hakim, L. El. (2024). Learning Design: To Improve Mathematical Problem-Solving Skills Using a Contextual Approach. JIIP - Jurnal Ilmiah Ilmu Pendidikan, 7(3), 2353–2366. https://doi.org/10.54371/jiip.v7i3.3455
Arsyad, M., Mujahiddin, M., & Syakhrani, A. W. (2024). The efficiency of using visual learning media in improving the understanding of science concepts in elementary school students. Indonesian Journal of Education (INJOE), 4(3), 775–787.
Carvajal, A. L. P., Fernandez, T. M., Pangilinan, A. M., Obod, M. M., Amihan, S. R., Sanchez, R. D., Sanchez, A. M. P., & Sanchez, J. J. D. (2025). Future-Proofing Teachers in Reframing Teacher Education Curriculum in the Philippines: Basis for Policy Recommendations. International Journal of Open-Access, Interdisciplinary and New Educational Discoveries of ETCOR Educational Research Center (IJOINED ETCOR), 4(2), 235–252.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46.
Farani, R. (2013). The Evaluation Of Instructional Multimedia Product For English Learning. Journal of English and Education (JEE).
February, P., Alter, G., Buendgens-Kosten, J., Müller, F. J., & Di Paolo, A. (2025). (DIGITAL) MEDIA AND MATERIALS FOR LEARNING. In All means all!–OpenTextbook for diversity in education (pp. 1115–1148). All-means-all.
Hana, A. Y., Sa’diyah, H., Nahar, Z. K., & Fakhriyana, D. (2024). Pendidikan Berkualitas untuk Mewujudkan Sustainable Development Goals (SDGs) Melalui Pembelajaran Matematika: Systematic Literature Review. ARITHMETIC: Academic Journal of Math, 6(1), 79–100.
Landis, J. R., & Koch, G. G. (1977). The Measurement of Observer Agreement for Categorical Data. Biometrics, 33(1), 159. https://doi.org/10.2307/2529310
Ling, J. H. (2025). A Review of Rubrics in Education: Potential and Challenges. Indonesian Journal of Innovative Teaching and Learning, 2(1), 1–14. https://doi.org/10.64420/ijitl.v2i1.197
MacKay, R. S., Kenna, R., Low, R. J., & Parker, S. (2017). Calibration with confidence: a principled method for panel assessment. Royal Society Open Science, 4(2), 160760.
Manik, I. M. S. B., Rizal, C., & Kurniawan, H. (2025). ANALISIS AKURASI DAN KEANDALAN METODE CERTAINTY FACTOR PADA SISTEM PAKAR DIAGNOSTIK MEDIS. Jurnal Multimedia Dan Teknologi Informasi (Jatilima), 7(03), 702–707.
McHugh, M. L. (2012). Interrater reliability: the kappa statistic. Biochemia Medica, 22(3), 276–282.
Olasoji, O. V., Ugwu, U. F., & Onoh, D. O. (2023). The Role of Mathematics Education in Achieving Sustainable Development Goals (SDGs). ESUT Journal of Education, 6(1), 247–255.
Prasetyowati, D. (2020). Development of Learning Videos for Junior High School Math Subject to Enhance Mathematical Reasoning. International Journal of Education and Practice, 8(1), 18–25.
Prawiro, F., & Vereen, A. (2024). REPRESENTASI SOSIAL TENTANG GANGGUAN JIWA PADA MASYARAKAT YANG INGIN MENCARI BANTUAN PSIKOLOGIS FORMAL (STUDI PADA MASYARAKAT BANTEN YANG RELATIF MODERN). MANASA, 13(2), 74–91. https://doi.org/10.25170/manasa.v13i2.6099
Reinke, L. T., & Casto, A. R. (2022). Motivators or conceptual foundation? Investigating the development of teachers’ conceptions of contextual problems. Mathematics Education Research Journal, 34(1), 113–137.
Sachdeva, S., & Eggen, P.-O. (2021). Learners’ Critical Thinking About Learning Mathematics. International Electronic Journal of Mathematics Education, 16(3), em0644. https://doi.org/10.29333/iejme/11003
Saharani, T., & Abadi, A. M. (2024). The Development of Interactive Learning Media Based on Discovery Learning Oriented to Students’ Concept Understanding and Mathematical Disposition. AL-ISHLAH: Jurnal Pendidikan, 16(4). https://doi.org/10.35445/alishlah.v16i4.5681
Sahu, S., & Demirtas, H. (2024). A Linear Relationship between Correlation and Cohen’s Kappa for Binary Data and Simulating Multivariate Nominal and Ordinal Data with Specified Kappa Matrix. ArXiv Preprint ArXiv:2404.14149.
Smith, N. M., Hoal, K. E. O., & Thompson, J. F. H. (2020). Ensure inclusive and equitable quality education and promote lifelong learning opportunities for all. In Mining, materials, and the sustainable development goals (SDGs) (pp. 29–38). CRC Press.
United Nations. (2024). Sustainable Development Goal 4: Quality Education. https://www.un.org/sustainabledevelopment/education
Warrens, M. J. (2014). New Interpretations of Cohen’s Kappa. Journal of Mathematics, 2014, 1–9. https://doi.org/10.1155/2014/203907
Warrens, M. J. (2015). Five ways to look at Cohen’s kappa. Journal of Psychology & Psychotherapy, 05(04). https://doi.org/10.4172/2161-0487.1000197
Widiana, I. W., Tegeh, I., & Parwata, I. (2020). Improving Students’ Factual Knowledge with Concrete Media through Observing Activities in Scientific Approaches in Elementary Schools. Journal of Education and E-Learning Research, 7(3), 293–299.
Zamanzadeh, V., Ghahramanian, A., Rassouli, M., Abbaszadeh, A., Alavi-Majd, H., & Nikanfar, A.-R. (2015). Design and Implementation Content Validity Study: Development of an instrument for measuring Patient-Centered Communication. Journal of Caring Sciences, 4(2), 165–178. https://doi.org/10.15171/jcs.2015.017
Zec, S., Soriani, N., Comoretto, R., & Baldi, I. (2017). High Agreement and High Prevalence: The Paradox of Cohen’s Kappa. The Open Nursing Journal, 11(1), 211–218. https://doi.org/10.2174/1874434601711010211