All Issue

2026 Vol.33, Issue 2

Research Article

Reliability Evaluation of LLM-as-a-Judge–Based Literature Classification: A Comparative Analysis of Zero-Shot and Few-Shot Approaches Using Domestic BEMS Research

LLM as a Judge 기반 문헌 분류 기법의 신뢰도 검증: 국내 BEMS 연구 문헌 대상의 제로샷과 퓨샷 비교 분석

Seon-Woo Kim1
,
Byeong-Il Kwon1
,
Shin-Kyu Kang1
,
Dong-Won Kim1
,
Chang-Heon Cheong2*
김 선우1
,
권 병일1
,
강 신규1
,
김 동원1
,
정 창헌2*
1Undergraduate student, School of Architectural Engineering, Gyeongsang National University, Jinju, Korea
2Professor, School of Architectural Engineering, Gyeongsang National University, Jinju, Korea
1경상국립대 건축공학부 학사과정
2경상국립대 건축공학부 교수
*Corresponding Author
30 April 2026. pp. 105-116
PDF XML Citation
Abstract

This study aims to examine the applicability and reliability of the LLM-as-a-Judge approach for literature classification using large language models (LLMs). To this end, a systematic literature review was conducted on domestic research related to Building Energy Management Systems (BEMS), and automated literature classification was performed using ChatGPT-5.1 under both zero-shot and few-shot settings. Subsequently, four researchers independently evaluated the appropriateness of the LLM-generated classification results. Inter-rater agreement and classification reliability were analyzed using final consensus rates, Fleiss’ kappa, and Gwet’s AC1 statistics. The results indicate that, even under the zero-shot setting, a meaningful level of agreement was observed between the LLM-based classification and human evaluations. However, under the few-shot setting—where predefined classification criteria established by the researchers were provided—the final consensus rate improved substantially to 95.83%. Despite an increase in observed inter-rater agreement in the few-shot condition, a decrease in Fleiss’ kappa was observed, corresponding to the well-known kappa paradox. Complementary analysis using Gwet’s AC1 confirmed the performance improvement of the few-shot approach. Overall, this study demonstrates that, rather than directly adopting automatically generated LLM outputs, combining clear, user-defined classification criteria with appropriate human intervention is effective in enhancing classification reliability and consistency in LLM-based literature classification. These findings may serve as a methodological foundation for future research trend analysis and automated literature classification in the field of building energy, including BEMS applications.

Keywords
BEMS
Literature classification
Large language model
Fleiss’ kappa
Gwet’s AC1
References
1

Ahn, Y. -Y., & Jung, S. -J. (2022). Trends in ICT-based building energy management technology standardization for carbon neutrality. The Journal of the Korean Institute of Communication Sciences, 39(7), 75-79.

2

Ahn, Y., Hong, G., & Kim, B. S. (2020). Predicting supply air temperature in air handling unit using machine learning-based automation algorithm. Korean Journal of Air-Conditioning and Refrigeration Engineering, 32(1), 37-45. https://doi.org/10.6110/KJACR.2020.32.1.037

10.6110/KJACR.2020.32.1.037
3

Bae, S., & Song, J. (2024). A study on the development of BACnet-based system control application using RTU terminal. Journal of Knowledge Information Technology and Systems, 19(1), 209-225. https://doi.org/10.34163/jkits.2024.19.1.020

10.34163/jkits.2024.19.1.020
4

Choi, H., Han, S., Han, G., Jung, D., Jin, S., & Kim, J. (2025). Analysis of Korea’s building energy codes and certification systems for the implementation of the grid-interactive efficient building (GEB). Korean Journal of Air-Conditioning and Refrigeration Engineering, 37(10), 469-479. https://doi.org/10.6110/KJACR.2025.37.10.469

10.6110/KJACR.2025.37.10.469
5

Choi, J. Y. (2025). Analysis of standard trends and standardization of energy management systems. The Journal of Korean Institute of Communications and Information Sciences, 50(2), 331-340. https://doi.org/10.7840/kics.2025.50.2.331

10.7840/kics.2025.50.2.331
6

Choi, S., & Kim, S. (2025). Reinforcement learning based energy control method for smart buildings integrated with renewable energy. The Journal of The Korea Institute of Electronic Communication Sciences, 20(1), 173-180. https://doi.org/10.13067/JKIECS.2025.20.1.173

10.13067/JKIECS.2025.20.1.173
7

Choi, S., Kim, K., & Won, A. (2022). An analysis of energy consumption characteristics and energy self‐sufficiency rate of national park management offices using BEMS data. Journal of Korean Institute of Architectural Sustainable Environment and Building Systems, 16(6), 509-518.

8

Choi, S., Lee, G., & Kim, K. (2023). Predicting model energy consumption in multi-family homes during the summer period measured by BEMS(building energy management systems) using the SARIMAX (seasonal auto-regressive integrated moving average with eXogenous factors) Model. Journal of the Regional Association of Architectural Institute of Korea, 25(6), 109-117.

9

Diamond, I. R., Grant, R. C., Feldman, B. M., Pencharz, P. B., Ling, S. C., Moore, A. M., & Wales, P. W. (2014). Defining consensus: A systematic review recommends methodologic criteria for reporting of delphi studies. Journal of Clinical Epidemiology, 67(4), 401-409. https://doi.org/10.1016/j.jclinepi.2013.12.002

10.1016/j.jclinepi.2013.12.002
10

Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378-382. https://doi.org/10.1037/h0031619

10.1037/h0031619
11

Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H., Wang, S., Zhang, K., Wang, Y., Gao, W., Ni, L., & Guo, J. (2024). A survey on LLM-as-a-Judge. arXiv. https://arxiv.org/abs/2411.15594

12

Gwet, K. L. (2008). Computing inter-rater reliability and its variance in the presence of high agreement. The British Journal of Mathematical and Statistical Psychology, 61(Pt 1), 29-48. https://doi.org/10.1348/000711006X126600

10.1348/000711006X126600
13

Hwang, J., Kim, J. & Yun, S., (2024). Building energy efficiency through the application of digital Twin-based BEMS. Journal of the Korean Solar Energy Society, 44(3), 33-42. https://doi.org/10.7836/kses.2024.44.3.033

10.7836/kses.2024.44.3.033
14

Jeon, J. B., & Kim, Y. I. (2025). A study on recovering lost data of office building electric power consumption using outdoor air enthalpy variation patterns. Korean Journal of Air-Conditioning and Refrigeration Engineering, 37(3), 143-150. https://doi.org/10.6110/KJACR.2025.37.3.143

10.6110/KJACR.2025.37.3.143
15

Jung, H., Kim, J., Kim, K., & Shin, Y. (2020). An empirical study on the validity of the construction project of overseas public building energy management system. Journal of The Korea Society of Information Technology Policy & Management, 12(4), 1861-1868.

16

Jung, H., Kim, J., Lee, S., & Kim, S. (2021). An empirical study of building energy consumption prediction simulator using machine learning techniques. Journal of The Korea Society of Information Technology Policy & Management, 13(4), 2543-2551.

17

Kim, D., Oh, S., Yang, H., & Lee, S. (2020). Improvements on public buildings energy information system. Journal of the Architectural Institute of Korea, 36(9), 109-118. https://doi.org/10.5659/JAIK.2020.36.9.109

10.5659/JAIK.2020.36.9.109
18

Kim, J., & Kim, S. (2022). Reinforcement learning-based illuminance control method for building lighting system. Journal of IKEEE, 26(1), 56-61. https://doi.org/10.7471/IKEEE.2022.26.1.56

10.7471/IKEEE.2022.26.1.56
19

Kim, J., Kim, S., Lee, D., & Kim, K. (2024). Performance comparisons on learning based prediction models considering electricity demand patterns of buildings. The Journal of Society for e-Business Studies, 29(2), 1-22. https://doi.org/10.7838/jsebs.2024.29.2.001

10.7838/jsebs.2024.29.2.001
20

Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159-174. https://doi.org/10.2307/2529310

10.2307/2529310
21

Lee, I. -H., Kim, S. -M., Kim, J. -H., & Kim, J. -T. (2025). Performance analysis of BIPV system and power consumption modeling with ESS integration in a library building. Journal of the Korean Solar Energy Society, 45(5), 13-25. https://doi.org/10.7836/kses.2025.45.5.013

10.7836/kses.2025.45.5.013
22

Lee, M. G., & Jung, K. (2021). Design of IoT-based energy monitoring system for residential building. The Journal of The Korea Institute of Electronic Communication Sciences, 16(6), 1223-1230. https://doi.org/10.13067/JKIECS.2021.16.6.1223

10.13067/JKIECS.2021.16.6.1223
23

Lee, Y., Kwag, B. C., Ha, H., & Kim, G. (2025). Analysis of BEMS applicability in existing multi-family housing based on facility and communication infrastructure. Journal of The Korean Society of Living Environmental System, 32(5), 541-552. https://doi.org/10.21086/ksles.2025.10.32.5.541

10.21086/ksles.2025.10.32.5.541
24

Lee, Y. -J., Jung, K., & Choi, B. J. (2020). Prediction and evaluation for power of photovoltaic system based on dynamic time warping hierarchical clustering for building energy management. Journal of The Korean Institute of Culture Architecture, 69, 235-241.

25

Oh, J., Chae, H., & Nam, Y. (2025). Comparative study of system capacity based on renewable energy installation standards in Korea. Transactions of the Korean Society for Geothermal and Hydrothermal Energy, 21(2), 33-39.

26

Park, D., Choi, S., Jian, S., Han, B., & Rho, D. (2022). A study on ems operation algorithm for peak demand management of small zero energy building. Journal of the Korea Academia-Industrial Cooperation Society, 23(5), 580-589. https://doi.org/10.5762/KAIS.2022.23.5.580

10.5762/KAIS.2022.23.5.580
27

Quan, J., Shin, J. -W., Ko, J. -L., & Shin, S. -K. (2021). A study on energy consumption prediction from building energy management system data with missing values using SSIM and VLSW algorithms. The Transactions of The Korean Institute of Electrical Engineers, 70(10), 1540-1547. https://doi.org/10.5370/KIEE.2021.70.10.1540

10.5370/KIEE.2021.70.10.1540
28

Ra, S., Aum, T., & Son, J. (2020). Analysis of annual operation status of central heating and cooling system in a public office building. Journal of The Architectural Institute of Korea Structure & Construction, 36(2), 175-180. https://doi.org/10.5659/JAIK_SC.2020.36.2.175

10.5659/JAIK_SC.2020.36.2.175
29

Yun, W., Ryu, W., & Seo, H. (2025). Development and data-driven evaluation of internal heat gain prediction-based MPC for HVAC systems. Journal of the Architectural Institute of Korea, 41(7), 353-364. https://doi.org/10.5659/JAIK.2025.41.7.353

10.5659/JAIK.2025.41.7.353
Information
  • Publisher :The Korean Society of Living Environmental System
  • Publisher(Ko) :한국생활환경학회
  • Journal Title :Journal of The Korean Society of Living Environmental System
  • Journal Title(Ko) :한국생활환경학회
  • Volume : 33
  • No :2
  • Pages :105-116
  • Received Date : 2026-02-03
  • Revised Date : 2026-02-23
  • Accepted Date : 2026-03-02
  • DOI :https://doi.org/10.21086/ksles.2026.4.33.2.105
Journal Informaiton Journal of The Korean Society of Living Environmental System Journal of The Korean Society of Living Environmental System
Online Submission & Review System
  • NRF
  • KOFST
  • crossref crossmark
  • crossref cited-by
  • crossref funder-registry
  • crosscheck
  • orcid
  • open access
  • ccl
Journal Informaiton Journal Informaiton - close