From Job Site to Insight: Evolving Data Pipelines for LLM-Driven Construction Analytics
DOI:
https://doi.org/10.63282/3117-5481/AIJCST-V8I5P101Keywords:
Construction Analytics, Data Pipelines, Large Language Models, Kafka, Apache Spark, Retrieval-Augmented Generation, Data Lakes, SaaS, Predictive Safety, Automated RFI ResolutionAbstract
With advances in construction, such as those from IoT sensors, equipment, BIM systems, RFIs, inspections, safety logs, field communication and project documents, a massive amount of data is created during a project. While both Apache Kafka and Spark work well with structured and semi-structured data, they lack the ability to interpret unstructured text, which impedes timely decision-making, coordination, risk detection, and project optimization. Setting the groundwork for a pipeline that transforms Kafka's data into operational insights in real time, powered by an LLM based on RAG and Spark. The architecture provides for automated analysis of all types of RFIs, field reports, safety observations and communications and leverages lakehouse technologies and tenant-aware retrieval to provide secure SaaS deployment. The proposed framework offers enhanced processing times, contextual accuracy and insight generation, and decision support, when compared to conventional pipelines, by leveraging automated knowledge extraction and secure and contextually relevant analytics.
References
[1] Tang, S., Shelden, D. R., Eastman, C. M., Pishdad-Bozorgi, P., & Gao, X. (2019). A review of building information modeling (BIM) and the internet of things (IoT) devices integration: Present status and future trends. Automation in construction, 101, 127-139.
[2] Wu, C., Li, X., Guo, Y., Wang, J., Ren, Z., Wang, M., & Yang, Z. (2022). Natural language processing for smart construction: Current status and future directions. Automation in Construction, 134, 104059.
[3] Ding, Y., Ma, J., & Luo, X. (2022). Applications of natural language processing in construction. Automation in Construction, 136, 104169.
[4] Kreps, J., Narkhede, N., & Rao, J. (2011, June). Kafka: A distributed messaging system for log processing. In Proceedings of the NetDB (Vol. 11, No. 2011, pp. 1-7).
[5] Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J., McCauly, M., ... & Stoica, I. (2012). Resilient distributed datasets: A {Fault-Tolerant} abstraction for {In-Memory} cluster computing. In 9th USENIX symposium on networked systems design and implementation (NSDI 12) (pp. 15-28).
[6] Zaharia, M., Xin, R. S., Wendell, P., Das, T., Armbrust, M., Dave, A., ... & Stoica, I. (2016). Apache spark: a unified engine for big data processing. Communications of the ACM, 59(11), 56-65.
[7] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.
[8] Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019, June). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) (pp. 4171-4186).
[9] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., ... & Amodei, D. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33, 1877-1901.
[10] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., ... & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33, 9459-9474.
[11] Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., ... & Liang, P. (2021). On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258.
[12] Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the association for computational linguistics, 12, 157-173.
[13] Afzal, M. (2024). Improving Request for Information (RFI) Processing in Construction Projects using Natural Language Processing (NLP) Techniques (Doctoral dissertation, University of Technology Sydney (Australia)).
[14] Fialho, B. C., Codinhoto, R., & Fabricio, M. M. (2020). BIM and IoT for the AEC Industry: A systematic literature mapping. Procedia Engineering, 2(4), 6.
[15] Guu, K., Lee, K., Tung, Z., Pasupat, P., & Chang, M. (2020, November). Retrieval augmented language model pre-training. In International conference on machine learning (pp. 3929-3938). PMLR.
[16] Izacard, G., Lewis, P., Lomeli, M., Hosseini, L., Petroni, F., Schick, T., ... & Grave, E. (2023). Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, 24(251), 1-43.
[17] Our Evolutionary Journey of Building the LLM-powered Data Agent, Online. https://medium.com/@meghasaini/our-evolutionary-journey-of-building-the-llm-powered-data-agent-e6109b943923
[18] Asai, A., Wu, Z., Wang, Y., Sil, A., & Hajishirzi, H. (2024, May). Self-rag: Learning to retrieve, generate, and critique through self-reflection. In International conference on learning representations (Vol. 2024, pp. 9112-9141).
[19] Es, S., James, J., Anke, L. E., & Schockaert, S. (2024, March). Ragas: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th conference of the european chapter of the association for computational linguistics: system demonstrations (pp. 150-158).
[20] van der Heijden, J. (2023). Construction 4.0 in a narrow and broad sense: A systematic and comprehensive literature review. Building and Environment, 244, 110788.
[21] Statsenko, L., Samaraweera, A., Bakhshi, J., & Chileshe, N. (2023). Construction 4.0 technologies and applications: A systematic literature review of trends and potential areas for development. Construction Innovation, 23(5), 961-993.
[22] Klinc, R., & Turk, Ž. (2019). Construction 4.0–digital transformation of one of the oldest industries. Economic and Business Review, 21(3), 4.
[23] Data-Driven Construction. Navigating the data age in the construction industry, datadrivenconstruction, Online. https://datadrivenconstruction.io/product/data-driven-construction/
[24] Craveiro, F., Duarte, J. P., Bartolo, H., & Bartolo, P. J. (2019). Additive manufacturing as an enabling technology for digital construction: A perspective on Construction 4.0. Automation in construction, 103, 251-267.
[25] Begić, H., & Galić, M. (2021). A Systematic Review of Construction 4.0 in the Context of the BIM 4.0 Premise. Buildings, 11(8), 337.
