Generative AI (GenAI) is increasingly shaping the landscape of software development, promising enhancements in efficiency and productivity for developers. A recent study conducted by Uplevel, an engineering intelligence platform, offers insights into this technology's actual impact, particularly focusing on tools like GitHub Copilot.
According to Matt Hoffman, product manager and data analyst at Uplevel, who spoke to BetaNews, GenAI’s use in software development is varied and depends on individual developers as well as the engineering organisations they belong to. A survey by GitHub highlighted that 92 per cent of developers are now utilising AI tools. Despite this widespread adoption, the technology is still in its infancy, and a clear consensus regarding best practices or applications has yet to be established. Hoffman noted, "We've even heard reports of training for these tools becoming outdated within weeks of creation."
Current applications of GenAI in programming are diverse; developers leverage these tools for generating code, obtaining immediate explanations for existing code blocks, detecting and debugging issues during code review, and even for drafting documentation and release notes. However, the varying ways AI is applied make the landscape seem somewhat chaotic.
When assessing the impacts of GenAI, organisations typically focus on traditional metrics such as efficiency and quality. The use of AI is expected to reduce cycle times in the review process and expedite bug detection, leading to quicker approval and merging of code. Hoffman describes other qualitative measurements that could come into play, including the perceived collaboration facilitated by AI and whether it alleviates developers from tedious tasks.
Despite expectations of improved productivity from GenAI tools, Uplevel’s investigation yielded unexpected results. Upon comparing the performance of about 800 developers before and after gaining access to GitHub Copilot, the data revealed no significant enhancement in metrics such as pull request (PR) cycle time or throughput. More concerning, there was a 41 per cent increase in bug rates among the developers studied, indicating a potential decline in code quality. Hoffman stated, "If GenAI is improving the developer experience... it’s likely happening in ways that don’t substantially change the amount of work they’re doing or how they’re collaborating."
For organisations contemplating the integration of GenAI into their development processes, Hoffman suggests a measured approach. With the predominance of AI tool usage among developers, he believes the era of questioning its adoption has passed. He advises engineering leaders to focus on experimentation: "Set goals, figure out a baseline metric to test against, and measure the impact over time." This iterative process allows teams to identify which applications of GenAI yield the most positive results, whilst also establishing guidelines to mitigate risks associated with experimentation.
Looking ahead, predicting the trajectories of GenAI in software development remains challenging due to the fast-paced evolution of the technology. Hoffman anticipates that like in other sectors, the most meaningful changes will result in enabling teams to concentrate on strategic initiatives by reducing the monotony of menial tasks. He encapsulates this vision: "You want it to do the refactoring so you can be creative." As GenAI continues to mature, it is expected to usher in further advancements within the realm of software development, although its full potential may take time to realise.
Source: Noah Wire Services