How to be your AI's boss (based on AWS, Google, IBM, and other giants)

The question isn’t whether your developers are ready to use AI. It’s whether they’re ready to manage it.

Over the past two years, some of the most heavily researched questions in software development have revolved around the impact of generative AI on developer productivity. The results? AI-assisted development is delivering measurable gains:

• Google engineers reported a 21% improvement in task completion time with AI-powered code suggestions and automation.

• Thoughtworks observed a 10-15% increase in developer productivity after rolling out GitHub Copilot.

• A large Brazilian enterprise saw a 23% reduction in cycle time after implementing generative AI.

The list of examples goes on, and it’s clear that AI tools like GitHub Copilot, Cursor, and ChatGPT enable developers to move faster. But speed isn’t the only metric that matters. Without proper oversight, AI-assisted development introduces significant risks to software quality and engineering processes.

For instance, GitClear’s annual research report highlights an alarming trend:

• Code churn has doubled, from 3.31% in 2023 to a projected 6.87% in 2025.

• Developers are reproducing existing code more frequently than refactoring, increasing technical debt.

So, how can engineering leaders reap the benefits of AI while avoiding its pitfalls? The key is actively managing it.

AI Is Like an Army of Junior Developers; Treat It That Way

Imagine hiring a team of junior developers and letting them loose on your codebase without oversight. Terrifying, right? That’s similar to what happens when you adopt AI coding assistants without a structured approach.

IBM conducted research as part of their recent rollout of the watsonx Code Assistant to more than 12,000 engineers, and they revealed some hard truths about AI adoption. To start, 42.6% of IBM developers felt AI made them less effective due to the extra effort required to refine and correct AI-generated code. Engineers frequently compared AI assistants to junior developers, requiring close supervision.

One developer had this to say:

“I tend to view it as a junior developer helping me out. It can generate code much more quickly than a junior developer, and usually of higher quality, but there are still often mistakes, edge cases, etc., that need to be fixed.”

This stigma isn’t unique to IBM. The University of Oslo uncovered a similar phenomenon when engineering leadership at Sparebank Utvikling introduced GitHub Copilot to the organization. The most commonly cited disadvantage of Copilot was that it provided inaccurate answers, with 50% of study participants indicating this was the biggest problem.

AI tools don’t automatically boost productivity without proper integration and buy-in. They introduce a new layer of management responsibility. To get the most out of AI, developers need to develop a new skill set that includes:

• Critical evaluation – Assessing AI suggestions for accuracy, quality, and alignment with project goals.

• Error correction – Identifying and fixing issues introduced by AI-generated code.

• Cognitive load management – Balancing AI oversight with their own problem-solving responsibilities.

Overcoming AI’s Crisis of Confidence

Beyond technical challenges, AI adoption introduces cultural and psychological barriers. IBM’s research uncovered an unexpected issue: developers feared peer judgment for relying on AI.

One IBM engineer admitted:

“I just don’t use code generated by [AI]... obviously, I would not want to be seen with generated code in my PRs. So embarrassing!”

Additionally, engineers at Sparebank Utvikling had a starkly different approach to AI based on seniority. Junior developers were more likely to accept AI-generated code unquestioningly, even when it didn’t compile. They sometimes spent extra time working with Copilot to debug and refine the auto-generated code. Senior developers used AI to make better decisions and navigate blockers, and they did most of the code creation work without AI. They treated AI as a sparring partner, requesting multiple solutions before choosing the best one.

This highlights a key takeaway: AI adoption is both a technical and cultural challenge. Engineering leaders must proactively shape how AI is perceived and used within their teams.

Here are some key steps to mitigate AI’s confidence crisis:

• Position AI as a productivity tool, not a replacement for developer expertise.

• Encourage developers to iterate on AI-generated code rather than unquestioningly accepting it.

• Normalize AI usage by integrating it into existing workflows.

• Provide automated oversight to mitigate code quality risks.

Where AI Delivers Real Productivity Gains

Imagine having a junior engineer who may not write perfect code but can rapidly analyze documentation, explain complex concepts, and synthesize insights from disparate resources. That’s where AI truly shines.

Recent research from Google, the University of California, and Carnegie Mellon identified specific areas where generative AI is particularly compelling:

• Idea generation – AI can suggest new coding approaches, architectural patterns, and creative problem-solving strategies.

• Cross-domain inspiration – AI exposes engineers to solutions from different fields, broadening problem-solving capabilities.

• Faster debugging – AI-powered assistants can quickly surface potential causes of bugs and suggest fixes.

• Understanding stakeholders – AI can simulate user or tester perspectives to validate feature requirements.

• Choice optimization – AI can evaluate multiple alternatives and apply engineering heuristics to recommend the best option.

The lesson is that AI is most effective when used to enhance human decision-making, not replace it.

The Future of Software Development Will Be Autonomous Agent Orchestration

Today, AI tools assist developers when prompted. Tomorrow, AI agents will execute tasks autonomously.

Recent research from AWS provides a glimpse of what’s ahead. They tested a handful of popular AI models to see how they performed when specialized agents handled different functions, including execution, analysis, retrieval-augmented generation (RAG), and external search.

The agentic development system used at AWS features agents for setting up environments, analyzing logs, and retrieving relevant internal and external information. This agent-based approach significantly improved the ability of all of the tested AI models to handle all of the coding tasks AWS engineers gave them. Specifically, more complex tasks like inference and evaluation almost universally failed in a single-agent environment, and adding a second agent increased the success rate by many orders of magnitude.

In the future, developers will spend less time producing code and more time managing this team of agents to ensure they all adequately fulfill their roles. Developers will likely be responsible for managing a portfolio of specialized AI models, guiding their interactions, and validating outputs. Over time, this application will extrapolate into higher-order challenges like writing tests, producing documentation, building boilerplate code, and replicating standardized features.

Software developers’ roles become more like a manager who maximizes the amount of work they outsource to their agent team. Rather than fearing AI, forward-thinking engineering leaders should help teams embrace their new role as AI managers.

[Sponsor] Stay Ahead of the Agentic Curve with LinearB

LinearB is a developer productivity platform that helps engineering organizations navigate the uncertain future of AI. LinearB recently launched Engineering Intelligence: a LinkedIn newsletter that covers cutting-edge research on developer productivity, DevEx, and software engineering leadership. Can’t get enough of what we cover on Dev Interrupted? We can’t blame you, the world is moving fast! But now you can keep the insights flowing every week both on Substack and LinkedIn.

Each edition dives into how your organization can effectively adopt cutting-edge technologies and maximize engineering effectiveness. The latest edition covers how AI agents will take (some of) your job and what you can do to stay ahead of the curve. It cuts through the hype to provide practical guidance on how to introduce AI into your software delivery workflows.

If you want to keep up on the best software engineering practices, subscribe to Engineering Intelligence today.

Be the Boss of AI, Not Its Servant

AI isn’t a magic bullet or an existential threat. It’s a tool, and like any tool, its effectiveness depends on how well it’s managed. To successfully integrate AI into your engineering organization, focus on establishing best practices for AI-assisted development:

• Train engineers to critically evaluate AI-generated code.

• Remove the stigma around AI usage and foster a culture of responsible adoption.

• Prepare for an agentic AI future where developers orchestrate autonomous AI workers.

Organizations that learn to manage AI effectively will gain a competitive edge in speed, quality, reliability, and innovation. Are you ready to be the boss of AI?

Get Inside Tips From an IBM AI Expert

In this week’s episode of Dev Interrupted, we sat down with Dr. Maryam Ashoori, Senior Director of Product Management for IBM watsonx.ai, to break down the challenges and opportunities of AI in software development, especially the growing skills gap among developers.

“Software engineers using AI-assisted development aren’t replacing engineers—they’re replacing engineers who aren’t using AI.”

Dr. Ashoori gives a powerful playbook to help engineers focus on building new skills to manage AI. She shares first-hand research from IBM and practical advice on how to stay ahead of AI’s evolution that every developer can put into action today.

If you enjoyed this article, be sure to check it out!