An empirical analysis of generative artificial intelligence models for enhancing software development efficiency

Abstract

The influence of Generative Artificial Intelligence (GAI) on modern software development is undeniable, offering intelligent code generation, automated debugging, on-the-go programming support, and more. Despite the widespread adoption of AI tools such as GitHub Copilot and ChatGPT in coding and their increasing use in software development, there has been a dearth of empirical studies that rigorously assess the efficiency and quality of software development when these tools are at play, using the same benchmark datasets. The study employs an empirical approach to compare the performance of generative AI models in enhancing software development efficiency, using a novel transformer-based large language model (LLM) framework and innovative techniques such as retrieval-augmented generation (RAG) and reinforcement learning from human feedback (RLHF). The framework proposed here is based on a comparison of the workflow of traditional software development with that of AI-assisted programming environments. The models presented are evaluated against the HumanEval and Mostly Basic Programming Problems (MBPP) benchmark datasets for programming ability, functional correctness, and software quality. Multiple evaluation metrics, including code generation accuracy, task completion time, defect density, pass@k score, maintainability index, and developer productivity, are analyzed using various validation techniques, including statistical methods. The experimental results prove the capability of the proposed AI-based framework to attain high coding efficiency, fast software development, and decrease manual programming efforts in comparison to the traditional approaches.