Why should I measure GitHub Copilot usage?

As more engineering teams incorporate GitHub Copilot into their daily workflows, the ability to measure the usage and adoption rates of GitHub CoPilot, along with its impact on development teams, becomes more important. Allstacks allows you to understand the ROI of this generative AI coding tool across your teams. 

Learn how you can track, view adoption trends, and optimize your Copilot development cycle in our overview video below.

Prerequisites to get started

• Copilot metrics will only appear under “Configure” → “Metrics” after you connect GitHub with your Allstacks environment. You will need to have an existing GitHub account.

• Access to one of the supported GitHub environments:

  • GitHub Cloud

  • GitHub Enterprise Cloud

  • GitHub Enterprise Server

• Appropriate access permissions:

  • For OAuth connections: A User account with the ‘Owner’ role in the GitHub organization

  • For GitHub App connections:  A User account with the ‘Owner’ role in the GitHub organization

• Copilot Metrics API is enabled for your GitHub organization

  • You can verify that this setting is enabled by checking your organization settings, under Copilot -> Policies.

Connection Methods

Option 1: OAuth App Connection

1. Ensure you have the Owner role in your GitHub organization

2. Navigate to Allstacks

3. Go to the Tools page in the top right organization menu

4. Click the green Add Tools button, then select “Github” in the tool list

5. Click the blue Add Tool button

6. Grant the requested permissions

7. Once the list of repositories has loaded, review the list of enabled repositories and disable any that aren’t needed for your use case.

Option 2: GitHub App Connection

1. Ensure you have the Owner role in your GitHub organization

   1. If you do not have the Owner role, this process will submit a request to install the app that an organization owner will need to approve.

2. Navigate to Allstacks

3. Go to the Tools page in the top right organization menu

4. Click the green Add Tools button, then select “Github App” in the tool list

5. Click the blue Add Tool button

6. Install the Allstacks GitHub App

7. Once the list of repositories has loaded, review the list of enabled repositories and disable any that aren’t needed for your use case.

Open

Where to find GitHub Copilot metrics in Allstacks

Click “Configure” in the lefthand sidebar and select “Metrics” in the pop-out menu, like below:

Open

On the “Metrics” page, scroll down until you see “Copilot Usage” at the bottom of the page. There, you’ll see the option to select either the “Copilot Adoption Rate” or “Copilot Suggestion Acceptance” metrics:

How metrics are generated

Usage metrics are generated by pulling data from Copilot API.
A “suggestion” by Copilot is defined by either a prompt or autocomplete.

Copilot Adoption Rate

The adoption rate metric measures the effectiveness of the roll out of Copilot at your organization. You can see what team members are using Copilot, who has not yet tried Copilot, and who does not yet have an invitation to use Copilot.

• Active - A user who accepted or rejected a suggestion from Copilot.

• Invite - A user who was invited to use Copilot but has not accepted or rejected a suggestion yet.

• Not invited - A user that do not have access to Copilot

Copilot Suggestion Acceptance Rate

This metric measures the number of suggestions accepted and suggestions rejected over time. This data is pulled from users that are defined as “active.”

Additional Reports

With the copilot integration, your contributors will get an auto-generated tag that can help you take a new perspective on the metrics your team uses to measure productivity, code quality, and coding productivity. This will allow you to compare the output of your cohort that uses copilot against those who are not yet using copilot to speed their development.

Troubleshooting

1. Unable to connect

   • OAuth App

     1. Verify that the organization you wish to connect doesn’t have OAuth app restrictions enabled, or that Allstacks is marked as an approved app.
        About OAuth app access restrictions - GitHub Docs  

        1. You can request approval to install the app with these steps: Requesting organization approval for OAuth apps - GitHub Docs  

        2. The app access can be approved by an organization owner with these steps: Approving OAuth apps for your organization - GitHub Docs  

   • GitHub App

     1. When attempting to install the Github App, if you’re taken to a screen that looks like this, it means that your Github account is not a member of any organizations. Ensure that you’re a member of an organization before attempting to connect.

2. Metrics Not Appearing

   • Verify that Copilot is properly connected under “tools” 

   • Check that your GitHub connection is active

   • Ensure you have the correct permissions

3. Access Issues

   • If connected with the OAuth app method, confirm the authorizing user has the Owner role in the org.

   • Verify the GitHub App installation status and permissions (if using GitHub App). This can be done in the org settings, go to the installed Github Apps list, and click Allstacks Analytics. Ensure that access is granted to the needed repos and that no permissions updates are pending approval.
     https://github.com/organizations/%3Cyour%20org%20name%3E/settings/installations  

   • Ensure that “Copilot Metrics API access” is enabled for your organization. This is found on the org settings page under Copilot -> Policies.

4. Data Sync Problems

   • The initial data synchronization may take up to 24 hours for the nightly sync to complete. Check the metrics library after 24 hours to see if the copilot metrics are available.

Support

If you encounter any issues or have questions:

1. Contact your Customer Success Manager (CSM)

2. Provide details about your GitHub environment

3. Include any error messages or specific issues you're experiencing

For additional support or questions about unsupported GitHub versions, please reach out to your CSM directly.