Evaluating ChatGPT’s Competency in BIM Related Knowledge via the Korean BIM Expertise Exam

Jiwon Choi; Bonsang Koo; Youngsu Yu; Yujeong Jeong; Namhyuk Ham

doi:10.13161/kibim.2023.13.3.021

All Issue

2023 Vol.13, Issue 3 Preview Page Next Page

Research Article

Evaluating ChatGPT’s Competency in BIM Related Knowledge via the Korean BIM Expertise Exam BIM 운용 전문가 시험을 통한 ChatGPT의 BIM 분야 전문 지식 수준 평가

30 September 2023. pp. 21-29

PDF XML

Abstract

ChatGPT, a chatbot based on GPT large language models, has gained immense popularity among the general public as well as domain professionals. To assess its proficiency in specialized fields, ChatGPT was tested on mainstream exams like the bar exam and medical licensing tests. This study evaluated ChatGPT's ability to answer questions related to Building Information Modeling (BIM) by testing it on Korea’s BIM expertise exam, focusing primarily on multiple-choice problems. Both GPT-3.5 and GPT-4 were tested by prompting them to provide the correct answers to three years' worth of exams, totaling 150 questions. The results showed that both versions passed the test with average scores of 68 and 85, respectively. GPT-4 performed particularly well in categories related to 'BIM software' and 'Smart Construction technology'. However, it did not fare well in 'BIM applications'. Both versions were more proficient with short-answer choices than with sentence-length answers. Additionally, GPT-4 struggled with questions related to BIM policies and regulations specific to the Korean industry. Such limitations might be addressed by using tools like LangChain, which allow for feeding domain-specific documents to customize ChatGPT’s responses. These advancements are anticipated to enhance ChatGPT’s utility as a virtual assistant for BIM education and modeling automation.

Keywords

ChatGPT

대규모 언어 모델

제너레이티브 AI

BIM 운용 전문가 시험

빌딩정보모델

키워드

ChatGPT

Large Language Model

Generative AI

BIM Expertise Exam

BIM

References

Beetz, J., Van Leeuwen, J., De Vries, B. (2009). IfcOWL : A Case of Transforming EXPRESS Schemas into Ontologies. Ai Edam, 23(1), pp. 89-101. 10.1017/S0890060409000122

Bommarito II, M., Katz, D. M. (2022). GPT Takes the Bar Exam. arXivpreprint arXiv:2212.14402. 10.2139/ssrn.4314839

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P. Sastry, G. Askell, A., Agarwal, S., Herbert-Voss, A. Krueger, G, Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D. (2020).Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems, 33, pp. 1877-1901.

Choi, J. H., Hickman, K. E., Monahan, A., Schwarcz, D. (2023). ChatGPT Goes to Law School. Available at SSRN. 10.2139/ssrn.4335905

Kasai, J., Kasai, Y., Sakaguchi, K., Yamada, Y., Radev, D. (2023). Evaluating GPT-4 and ChatGPT on Japanese- Medical Licensing Examinations. JMIR Medical Education, 9.

KIBIM (2023). A study on countermeasures for construction engineering in accordance with the digitalization of the construction industry, Final Report.

Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepano, C., Madriaga, M., Aggabao, R., Diaz- Candido, G., Maningo, J., Tseng, V. (2023). Performance of ChatGPT on USMLE: Potential for AI-assisted Medical Education Using Large Language Models. PLOS Digital Health, 2(2). 10.1371/journal.pdig.000019836812645PMC9931230

Mazairac, W., Beetz, J. (2013). BIMQL-An Open Query language for Building Information Models. Advanced Engineering Informatics, 27(4), pp. 444-456. 10.1016/j.aei.2013.06.001

OpenAI (2023). GPT-4 Technical Report. ArXiv, abs/2303. 08774.

Seo, M. B., Ju, K. B. (2012). Strategies to Revitalize BIM (Building Information Modeling) by the Survey Question- Naires from Design Experts in Field of Civil Engineering. Journal of the Korea Contents Association, 12(11), pp. 446-457. 10.5392/JKCA.2012.12.11.446

Singh, A. K., Pal, A., Kumar, P., Lin, J. J., Hsieh, S. H. (2023). Prospects of Integrating BIM and NLP for Automatic Construction Schedule Management. ISARC. Proceedings of the International Symposium on Automation and Robotics in Construction. IAARC Publications, 40, pp. 238-245. 10.22260/ISARC2023/0034

Zheng, J., Fischer, M. (2023). BIM-GPT: A Prompt-based Virtual Assistant Framework for BIM Information Retrieval. arXiv preprint, DOI: 10.48550/arXiv.2304.09333 10.1016/j.autcon.2023.105067

Information

Publisher :Journal of KIBIM
Publisher(Ko) :한국BIM학회
Journal Title :Journal of KIBIM
Journal Title(Ko) :한국BIM학회논문집
Volume : 13
No :3
Pages :21-29
Received Date : 2023-06-16; 2023-09-05
Accepted Date : 2023-09-10
DOI :https://doi.org/10.13161/kibim.2023.13.3.021

Journal of KIBIM ISSN:2288-1697(Print) 2288-1697(Online) 한국BIM학회논문집

All Issue