pixel

AI-powered coding for everyone

Engineers like Anton Mirhorodchenko are overcoming accessibility issues in unprecedented ways.

Anton Mirhorodchenko’s path to becoming a proficient software developer was anything but conventional. Diagnosed with cerebral palsy, Anton faced physical challenges from a young age. After his parents bought him a book about QBasic, he learned the language in one month, and a new world of possibilities opened up. Despite the limited functionality and lack of accessibility features of his laptop, Anton immersed himself in the digital realm, teaching himself to code.

Over time, Anton developed a shorthand typing method to accommodate his limited finger mobility. However, coding efficiently remained a challenge. That was until he discovered GitHub Copilot, an AI-powered tool that seemed tailor-made for his unique situation. Initially skeptical, Anton soon realized the transformative potential of AI in coding. GitHub Copilot expanded his shorthand, predicted his intentions, and simplified his workflow, making his code more accessible and encouraging him to strive for excellence and inclusivity.

Anton recently created a guide to coding with AIa testament to what he has learned—which he’s eager to share with others. In addition, he’s ventured into practical projects, like designing a simple feeding robot and actively developing open-source eye-tracking software that leverages neural networking to transform standard webcams into cost-effective alternatives. 

Anton’s journey epitomizes the transformative power of AI in coding. But his story is just one of many, reflecting a more significant shift in the developer community, especially during the challenging times of the pandemic.

With AI and Copilot, I can code my intention more precisely.
Anton Mirhorodchenko
Software Developer, Mentor, Microsoft

A need for better communication

The COVID-19 pandemic reshaped the landscape of work, particularly for software developers. As remote work became the new normal, many developers, especially newcomers, grappled with isolation and communication barriers. Connecting with managers and colleagues was fraught with delays and misunderstandings, severely hampering productivity.

Building for accessibility, building for everyone

Enter Priyankar Kumar, a software engineer at Microsoft. In response to the burgeoning communication challenges, Priyankar conceptualized “Hello Cody,” a voice assistant designed for developers’ favorite platform, Visual Studio (VS) Code. Initially, it was meant to be a programming assistant you could communicate with, like asking for help from a colleague. It would allow developers to control code with their voice, receive audio explanations, and interact with an assistant for help with tasks. By using simple voice commands, Cody was set to empower users to define custom commands and streamline their daily activities.

However, it was only a short time before Priyankar’s team stumbled upon a revelation. A team member with visual impairments illuminated how Cody could revolutionize Integrated Development Environment (IDE) navigation, code comprehension, and product development for individuals like him. Another engineer coping with a genetic connective tissue disorder voiced how Cody could alleviate the strain caused by keyboard dependency for users with repetitive stress injuries and other physical disabilities.

We realized there was a strong accessibility aspect to this project, and that we could potentially solve a lot of challenges for those who are left out of using many of the tools we have.
Priyankar Kumar
Microsoft Software Engineer

Hack the future

The insight prompted the team to re-envision “Hello Cody” as a VS Code extension, focusing on improving accessibility and productivity through voice commands. Their efforts were recognized with three awards at the 2021 Microsoft Global Hackathon, leading to a pitch opportunity with product development leaders at Microsoft in March 2022.

“As we got together to discuss the Hackathon project, it became apparent there were still several problems that needed to be solved to help make us, the developer community, more productive,” states the “Hello Cody” Hackathon team.

Since that pivotal moment, the team has collaborated with GitHub Next to evolve “Hello Cody” into Copilot Voice for Copilot X, a suite of GitHub Copilot products designed for AI-powered software development.

Helping developers with disabilities

Beatrice Tohni, a Microsoft SharePoint software engineer, benefits from Copilot Voice. She has hypermobile Ehlers-Danlos Syndrome (hEDS), a genetic connective tissue disorder that affects joints and ligaments, making everyday tasks like typing painful and difficult. Copilot Voice significantly reduces repetitive keystrokes and boilerplate code, improving her daily pain management and reducing the risk of joint dislocation. She acknowledges that Copilot Voice is still in beta but notices significant improvements with each use.

“Anybody who has trouble using a traditional keyboard or mouse will benefit hugely from Copilot Voice,” says Beatrice. “It cuts down on a lot of the busy work that goes into coding.”

Once you start building for accessibility, you also end up building for everyone. And that’s the beauty of this project.
Priyankar Kumar
Microsoft Software Engineer

The future of AI-empowered coding is bright

What distinguishes Copilot Voice and GitHub Copilot products from other platforms is their intelligence. AI enables users to express their coding intentions naturally in everyday language, allowing the tool to generate code accordingly. GitHub Copilot serves as a coding companion, enhancing developers’ abilities and making them more efficient. While users with disabilities benefit greatly, the tool is designed for all developers.

Copilot and Copilot Voice are powered by Microsoft technologies like Microsoft Azure Cognitive Speech Services, VS Code APIs, and Azure Open AI services. By combining these and Generative AI, we can unleash better productivity for all. For example, with Copilot Voice, users can control their IDE (Integrated Development Environment) with just their voice instead of memorizing key-bindings, and converse with Copilot to understand pieces of code. For developers with disabilities like RSI, dependency on a keyboard and mouse is greatly reduced. 

“There’s an appetite from customers who need it, especially people with disabilities,” says Rahul Pandita, researcher for GitHub Next.

Research shows that GitHub Copilot helps developers code faster, stay in the flow longer, and feel more fulfilled with their work. And there’s much more to come as GitHub’s extensive open-source software repository continues to enrich Copilot’s capabilities, with Github Copilot X representing its vision for the future.