A Three-Layer Reference Architecture for Agentic Engineering Integrating Runtime Harnesses, Evidence-Driven Execution Loops, and Explicit Workflow Graphs for Production Reliability
DOI:
https://doi.org/10.63282/3117-5481/AIJCST-V8I6P101Keywords:
Agentic Engineering, Agentic AI, Three-Layer Reference Architecture, Runtime Harnesses, Evidence-Driven Execution, Workflow Graphs, Production Reliability, Autonomous Systems, Execution ValidationAbstract
A new paradigm for engineering autonomous systems that are able to plan, execute, monitor and complete complicated tasks with very little human interaction is gaining traction, and it is known as agentic engineering. The adoption of agentic systems in production systems, however, is certainly not without its difficulties with regard to reliability of execution, state management, task coordination, evidence checks, error recovery, and transparency of operations. Frequently, there is no common design for coordinating runtime execution, validating intermediate results and systematically managing complex workflows in existing agentic architectures. In this paper, a three-layer reference architecture for agentic engineering, combining run-time harnesses, evidence-driven execution loops and explicit workflow graphs, is proposed to tackle these challenges, with the aim of making production more reliable. The first layer is for managing the environments that are run, integration of tools, tracking of state and exception handling. The second layer is for setting up evidence-based execution mechanisms, linking agent actions to observations, verification, feedback and corrective operations. The third layer is based on explicit workflow graphs to specify task dependencies, executions, conditional transitions and recovery paths. The proposed architecture makes it possible to coordinate the three layers but also ensures that traceability, observability, fault tolerance and systematic execution validation are achieved. An evaluation methodology is proposed that provides a structured approach to measuring task completion, task execution success, whether evidence is verified correctly, whether the workflow is consistent, how long the response takes and how effective the failure recovery. The architecture serves as a systematic basis for the design, assessment, and implementation of reliable agentic engineering systems in complex production systems.
References
[1] Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022). React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629.
[2] Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Hambro, E., ... & Scialom, T. (2023). Toolformer: Language models can teach themselves to use tools. Advances in neural information processing systems, 36, 68539-68551.
[3] Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., & Yao, S. (2023). Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems, 36, 8634-8652.
[4] Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., ... & Clark, P. (2023). Self-refine: Iterative refinement with self-feedback. Advances in neural information processing systems, 36, 46534-46594.
[5] Gou, Z., Shao, Z., Gong, Y., Yang, Y., Duan, N., & Chen, W. (2024, May). Critic: Large language models can self-correct with tool-interactive critiquing. In International Conference on Learning Representations (Vol. 2024, pp. 57734-57811).
[6] Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., ... & Neubig, G. (2023, July). Pal: Program-aided language models. In International conference on machine learning (pp. 10764-10799). PMLR.
[7] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., ... & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33, 9459-9474.
[8] Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023, October). Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology (pp. 1-22).
[9] Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., ... & Wen, J. (2024). A survey on large language model based autonomous agents. Frontiers of computer science, 18(6), 186345.
[10] Li, X., Wang, S., Zeng, S., Wu, Y., & Yang, Y. (2024). A survey on LLM-based multi-agent systems: workflow, infrastructure, and challenges. Vicinagearth, 1(1), 9.
[11] Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L., Lu, Y., ... & Sun, M. (2024, May). Toolllm: Facilitating large language models to master 16000+ real-world apis. In International Conference on Learning Representations (Vol. 2024, pp. 9695-9717).
[12] Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., ... & Wang, C. (2023). Autogen: Enabling next-gen llm applications via multi-agent conversation. arXiv preprint arXiv:2308.08155.
[13] Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Wang, J., ... & Schmidhuber, J. (2024, May). MetaGPT: Meta programming for a multi-agent collaborative framework. In International Conference on Learning Representations (Vol. 2024, pp. 23247-23275).
[14] Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., ... & Tang, J. (2024, May). Agentbench: Evaluating llms as agents. In International Conference on Learning Representations (Vol. 2024, pp. 52989-53046).
[15] Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., ... & Neubig, G. (2024, May). Webarena: A realistic web environment for building autonomous agents. In International Conference on Learning Representations (Vol. 2024, pp. 15585-15606).
[16] Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2024, May). Swe-bench: Can language models resolve real-world github issues?. In International Conference on Learning Representations (Vol. 2024, pp. 54107-54157).
[17] Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., ... & Su, Y. (2023). Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36, 28091-28114.
[18] Karpas, E., Abend, O., Belinkov, Y., Lenz, B., Lieber, O., Ratner, N., ... & Tenenholtz, M. (2022). Mrkl systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning. arXiv preprint arXiv:2205.00445.
[19] Wooldridge, M. (2009). An introduction to multiagent systems. John wiley & sons.
[20] Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., ... & Schulman, J. (2021). Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332.
[21] Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., & Narasimhan, K. (2023). Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36, 11809-11822.
[22] Li, G., Hammoud, H., Itani, H., Khizbullin, D., & Ghanem, B. (2023). Camel: Communicative agents for" mind" exploration of large language model society. Advances in neural information processing systems, 36, 51991-52008.
[23] Baylor, D., Breck, E., Cheng, H. T., Fiedel, N., Foo, C. Y., Haque, Z., ... & Zinkevich, M. (2017, August). Tfx: A tensorflow-based production-scale machine learning platform. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining (pp. 1387-1395).
[24] Yu, J., & Buyya, R. (2005). A taxonomy of workflow management systems for grid computing. Journal of grid computing, 3(3), 171-200.
[25] Deelman, E., Vahi, K., Juve, G., Rynge, M., Callaghan, S., Maechling, P. J., ... & Wenger, K. (2015). Pegasus, a workflow management system for science automation. Future Generation Computer Systems, 46, 17-35.
[26] Taufer, M., & Rosenberg, A. L. (2017). Scheduling DAG-based workflows on single cloud instances: High-performance and cost effectiveness with a static scheduler. The International Journal of High Performance Computing Applications, 31(1), 19-31.
[27] Benoit, A., Çatalyürek, Ü. V., Robert, Y., & Saule, E. (2013). A survey of pipelined workflow scheduling: Models and algorithms. ACM Computing Surveys (CSUR), 45(4), 1-36.
[28] Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., ... & Dennison, D. (2015). Hidden technical debt in machine learning systems. Advances in neural information processing systems, 28.
[29] AI, N. (2023). Artificial intelligence risk management framework (AI RMF 1.0). URL: https://nvlpubs. nist. gov/nistpubs/ai/nist. ai, 1(0), 0-1.
