Impact of Automated Software Development Using Large Language Models: Capabilities, Limitations, and Future Evolution
Downloads
The rapid integration of Large Language Models (LLMs) in the field of software engineering is very much changing the methods of coding, which, at the same time, are also being maintained and optimized. Through this article the journey of the coming of capabilities and restrictions as well as the direction of the future of software development with LLM is monitored. The authors of this article have given a detailed survey of LLM utilization in various stages of the life cycle of development organization, a number of them being code generation, bug detection, automated testing, documentation, and translation of natural language into code, productivity, quality, and accessibility being among the improvements indicated. It is demonstrated through comparative analyses that LLMs’ performance is more favorable than that of traditional and rule-based approaches, while the positions of developers, project managers, and executives as stakeholders reveal that they are both excited about the efficiency gains and, on the other hand, concerned about issues of technical reliability, data privacy, and over-reliance on automation. The main problems that LLMs are facing today—like hallucinations, being out-of-date, not having a very large context, and depending too much on prompt engineering—are plainly revealed along with the proposed solutions. Next, we consider progress in the field of multimodal and context-aware systems, autonomous software agents, continuous learning, and human-AI co-creation platforms. LLM-assisted development is likely to change the face of challenges if it is going to be fully materialized, thus bringing about the birth of a new era of collaborative, efficient, and innovative software engineering.
Terragni V, Vella A, Roop P, Blincoe K. The Future of AI-Driven Software Engineering. ACM Transactions on Software Engineering and Methodology 2025;34:1–20.
https://doi.org/10.1145/3715003.
Peng X. Software development in the age of intelligence: embracing large language models with the right approach. Frontiers of Information Technology & Electronic Engineering 2023;24:1513–9. https://doi.org/10.1631/FITEE.2300537.
Haque MdA. LLMs: A game-changer for software engineers? BenchCouncil Transactions on Benchmarks, Standards and Evaluations 2025;5:100204. https://doi.org/10.1016/j.tbench.2025.100204.
Husein RA, Aburajouh H, Catal C. Large language models for code completion: A systematic literature review. Comput Stand Interfaces 2025;92:103917. https://doi.org/10.1016/j.csi.2024.103917.
Fan A, Gokkaya B, Harman M, Lyubarskiy M, Sengupta S, Yoo S, et al. Large Language Models for Software Engineering: Survey and Open Problems. 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE), IEEE; 2023, p. 31–53.https://doi.org/10.1109/ICSE-FoSE59343.2023.00008.
Hou X, Zhao Y, Liu Y, Yang Z, Wang K, Li L, et al. Large Language Models for Software Engineering: A Systematic Literature Review. ACM Transactions on Software Engineering and Methodology 2024;33:1–79. https://doi.org/10.1145/3695988.
Wang J, Chen Y. A Review on Code Generation with LLMs: Application and Evaluation. 2023 IEEE International Conference on Medical Artificial Intelligence (MedAI), IEEE; 2023, p. 284–9. https://doi.org/10.1109/MedAI59581.2023.00044.
Yu S, Fang C, Ling Y, Wu C, Chen Z. LLM for Test Script Generation and Migration: Challenges, Capabilities, and Opportunities. 2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security (QRS), IEEE; 2023, p. 206–17. https://doi.org/10.1109/QRS60937.2023.00029.
Rasnayaka S, Wang G, Shariffdeen R, Iyer GN. An Empirical Study on Usage and Perceptions of LLMs in a Software Engineering Project. Proceedings of the 1st International Workshop on Large Language Models for Code, New York, NY, USA: ACM; 2024, p. 111–8. https://doi.org/10.1145/3643795.3648379.
Nunes H, Figueiredo E, Rocha L, Nadi S, Ferreira F, Esteves G. Evaluating the Effectiveness of LLMs in Fixing Maintainability Issues in Real-World Projects. 2025 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), IEEE; 2025, p. 669–80. https://doi.org/10.1109/SANER64311.2025.00069.
Jahić J, Sami A. State of Practice: LLMs in Software Engineering and Software Architecture. 2024 IEEE 21st International Conference on Software Architecture Companion (ICSA-C), IEEE; 2024, p. 311–8.https://doi.org/10.1109/ICSA-C63560.2024.00059.
Zhang Q, Fang C, Xie Y, Zhang Y, Yang Y, Sun W, et al. A Survey on Large Language Models for Software Engineering 2024.
Wang R, Guo J, Gao C, Fan G, Chong CY, Xia X. Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering 2025. https://doi.org/10.1145/3728963.
Meedeniya D, Perera I, Bowles J. Transformation and composition of software design models for Model Driven Development. 2015 IEEE 10th International Conference on Industrial and Information Systems (ICIIS), IEEE; 2015, p. 31–6. https://doi.org/10.1109/ICIINFS.2015.7398981.
Somogyi FA. Merging Textual Representations of Software Models. The publications of the MultiScience - XXX. MicroCAD International Scientific Conference, University of Miskolc; 2016. https://doi.org/10.26649/musci.2016.060.
Hrycyk JC, Soares IW, Wiedermann Agner LT. Definition of Object Constraint Language Rules in Models to Support the Development of Android Applications. Journal of Computer Science 2018;14:253–9. https://doi.org/10.3844/jcssp.2018.253.259.
A AA, D H, Nivedha MrsS. ProgAI: Enhancing Code Generation with LLMs For Real World Challenges. INTERANTIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT 2024;08:1–11.
https://doi.org/10.55041/IJSREM36242.
Banerjee S, Dutta A, Layek S, Sahoo A, Joyce SC, Hazra R. Redefining Developer Assistance: Through Large Language Models in Software Ecosystem 2024.
Eghbali A, Pradel M.De-Hallucinator: Mitigating LLM Hallucinations in Code Generation Tasks via Iterative Grounding 2024.
Bo L, He Y, Sun X, Ji W, Wu X. A Software Bug Fixing Approach Based on Knowledge-Enhanced Large Language Models. 2024 IEEE 24th International Conference on Software Quality, Reliability and Security (QRS), IEEE; 2024, p. 169–79. https://doi.org/10.1109/QRS62785.2024.00026.
Zhang Y, Jin Z, Xing Y, Li G. STEAM: Simulating the InTeractive BEhavior of ProgrAMmers for Automatic Bug Fixing 2023.
https://doi.org/10.1145/3718739.
Weng G, Andrzejak A. Automatic Bug Fixing via Deliberate Problem Solving with Large Language Models. 2023 IEEE 34th International Symposium on Software Reliability Engineering Workshops (ISSREW), IEEE; 2023, p. 34–6. https://doi.org/10.1109/ISSREW60843.2023.00040.
Ouedraogo WC, Kabore K, Tian H, Song Y, Koyuncu A, Klein J, et al. LLMs and Prompting for Unit Test Generation: A Large-Scale Evaluation. Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, New York, NY, USA: ACM; 2024, p. 2464–5. https://doi.org/10.1145/3691620.3695330.
Deng Y, Chen R, Xiao C, Yang Z, Luo Y, Zhao J, et al. LLM - TG: Towards Automated Test Case Generation for Processors Using Large Language Models. 2024 IEEE 42nd International Conference on Computer Design (ICCD), IEEE; 2024, p. 389–96. https://doi.org/10.1109/ICCD63220.2024.00066.
Feng S, Lu H, Jiang J, Xiong T, Huang L, Liang Y, et al. Enabling Cost-Effective UI Automation Testing with Retrieval-Based LLMs: A Case Study in WeChat 2024.
Wang W, Yang C, Wang Z, Huang Y, Chu Z, Song D, et al. TESTEVAL: Benchmarking Large Language Models for Test Case Generation 2025.
Vu MN, Nguyen T, Jeter TR, Thai MT. Analysis of Privacy Leakage in Federated Large Language Models 2024.
Chen Z, Wang C, Sun W, Yang G, Liu X, Zhang JM, et al. Promptware Engineering: Software Engineering for LLM Prompt Development 2025.
Taherkhani H, Sepindband M, Pham HV, Wang S, Hemmati H. EPiC: Cost-effective Search-based Prompt Engineering of LLMs for Code Generation 2024.
Weber T, Brandmaier M, Schmidt A, Mayer S. Significant Productivity Gains through Programming with Large Language Models. Proc ACM Hum Comput Interact 2024;8:1–29. https://doi.org/10.1145/3661145.
Wang G, Sun Z, Gong Z, Ye S, Chen Y, Zhao Y, et al. Do Advanced Language Models Eliminate the Need for Prompt Engineering in Software Engineering? 2024.
Son M, Won Y-J, Lee S. Optimizing Large Language Models: A Deep Dive into Effective Prompt Engineering Techniques. Applied Sciences 2025;15:1430. https://doi.org/10.3390/app15031430.
Fan A, Gokkaya B, Harman M, Lyubarskiy M, Sengupta S, Yoo S, et al. Large Language Models for Software Engineering: Survey and Open Problems. 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE), IEEE; 2023, p. 31–53.https://doi.org/10.1109/ICSE-FoSE59343.2023.00008.
Yang J. Large language models privacy and security. Applied and Computational Engineering 2024;76:177–88. https://doi.org/10.54254/2755-2721/76/20240584.
Tabarsi B, Reichert H, Limke A, Kuttal S, Barnes T. LLMs’ Reshaping of People, Processes, Products, and Society in Software Development: A Comprehensive Exploration with Early Adopters 2025.
Ehsan U, Harrison B, Chan L, Riedl MO. Rationalization. Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, New York, NY, USA: ACM; 2018, p. 81–7. https://doi.org/10.1145/3278721.3278736.
Fakhoury S, Chakraborty S, Musuvathi M, Lahiri SK. Towards Generating Functionally Correct Code Edits from Natural Language Issue Descriptions 2023.
Zrelli R, Misson HA, Ben Attia M, de Magalhaes FG, Shabah A, Nicolescu G. Advancing Formal Verification: Fine-Tuning LLMs for Translating Natural Language Requirements to CTL Specifications. 2024 International Workshop on Rapid System Prototyping (RSP), IEEE; 2024, p. 21–7. https://doi.org/10.1109/RSP64122.2024.10870993.
Dahl M, Magesh V, Suzgun M, Ho DE. Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models 2024.
https://doi.org/10.1093/jla/laae003.
Tian Y, Yan W, Yang Q, Zhao X, Chen Q, Wang W, et al. CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verification 2025.
Sun W, Li B, Zhang GL, Yin X, Zhuo C, Schlichtmann U. Classification-Based Automatic HDL Code Generation Using LLMs 2024.
Bhattacharya R. Strategies to mitigate hallucinations in large language models. Applied Marketing Analytics: The Peer-Reviewed Journal 2024;10:62. https://doi.org/10.69554/NXXB8234.
Tang W, Cao Y, Deng Y, Ying J, Wang B, Yang Y, et al. EvoWiki: Evaluating LLMs on Evolving Knowledge 2024.
Sun Z, Liu Y, Wang J, Meng F, Xu J, Chen Y, et al. Outdated Issue Aware Decoding for Reasoning Questions on Edited Knowledge 2024.
Gupta L, Sharma S, Zhao Y. Systematic Evaluation of Long-Context LLMs on Financial Concepts. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, Stroudsburg, PA, USA: Association for Computational Linguistics; 2024, p. 1163–75. https://doi.org/10.18653/v1/2024.emnlp-industry.88.
Almorsi A, Ahmed M, Gomaa W. Guided Code Generation with LLMs: A Multi-Agent Framework for Complex Code Tasks 2025.
Zhang Y, Li J, Liu P. Extending LLMs’ Context Window with 100 Samples 2024.
Venditti D, Ruzzetti ES, Xompero GA, Giannone C, Favalli A, Romagnoli R, et al. Enhancing Data Privacy in Large Language Models through Private Association Editing 2024.
Zhan X, Seymour W, Such J. Beyond Individual Concerns: Multi-user Privacy in Large Language Models. ACM Conversational User Interfaces 2024, New York, NY, USA: ACM; 2024, p. 1–6. https://doi.org/10.1145/3640794.3665883.
Peng S, Kalliamvakou E, Cihon P, Demirer M. The Impact of AI on Developer Productivity: Evidence from GitHub Copilot 2023.
Mavrych V, Yaqinuddin A, Bolgova O. Claude, ChatGPT, Copilot, and Gemini performance versus students in different topics of neuroscience. Adv Physiol Educ 2025;49:430–7. https://doi.org/10.1152/advan.00093.2024.
Wulcan JM, Jacques KL, Lee MA, Kovacs SL, Dausend N, Prince LE, et al. Classification performance and reproducibility of GPT-4 omni for information extraction from veterinary electronic health records. Front Vet Sci 2025;11. https://doi.org/10.3389/fvets.2024.1490030.
Kim S, Baudru J, Ryckbosch W, Bersini H, Ginis V. Early evidence of how LLMs outperform traditional systems on OCR/HTR tasks for historical records 2025.
Rasnayaka S, Wang G, Shariffdeen R, Iyer GN. An Empirical Study on Usage and Perceptions of LLMs in a Software Engineering Project 2024.
Karnouskos S. The Relevance of Large Language Models for Project Management. IEEE Open Journal of the Industrial Electronics Society 2024;5:758–68. https://doi.org/10.1109/OJIES.2024.3412222.
Khomh F. Harnessing Predictive Modeling and Software Analytics in the Age of LLM-Powered Software Development (Invited Talk). Proceedings of the 19th International Conference on Predictive Models and Data Analytics in Software Engineering, New York, NY, USA: ACM; 2023, p. 1–1. https://doi.org/10.1145/3617555.3634736.
Zhang L-J, Chen H, He S, Li C, Chen J, Zhang H, et al. COSIS: An AI-Enabled Digital Transformation Framework Integrating Large Language Models and Key Performance Indicators, 2025, p. 74–99. https://doi.org/10.1007/978-3-031-77000-5_6.
Pashchenko DS. Early Formalization of AI-tools Usage in Software Engineering in Europe: Study of 2023. International Journal of Information Technology and Computer Science 2023;15:29–36. https://doi.org/10.5815/ijitcs.2023.06.03.
Copyright (c) 2025 MD Raziul Hasan Nayon, S M Shezan Ahmed, MD TARIQ UZ ZAMAN, MST Ashima Alom Shova, Fatin Khan
This work is licensed under a Creative Commons Attribution 4.0 International License.
All Content should be original and unpublished.
