As more engineering teams incorporate GitHub Copilot into their daily workflows, the ability to measure the usage and adoption rates of GitHub CoPilot, along with its impact on development teams, becomes more important. Allstacks allows you to understand the ROI of this generative AI coding tool across your teams.
Learn how you can track, view adoption trends, and optimize your Copilot development cycle in our overview video below.
Copilot metrics will only appear under “Configure” → “Metrics” after you connect GitHub with your Allstacks environment. You will need to have an existing GitHub account.
Access to one of the supported GitHub environments:
GitHub Cloud
GitHub Enterprise Cloud
GitHub Enterprise Server
Appropriate access permissions:
For OAuth connections: A User account with the ‘Owner’ role in the GitHub organization
For GitHub App connections: A User account with the ‘Owner’ role in the GitHub organization
Copilot Metrics API is enabled for your GitHub organization
You can verify that this setting is enabled by checking your organization settings, under Copilot -> Policies.
Ensure you have the Owner role in your GitHub organization
Navigate to Allstacks
Go to the Tools page in the top right organization menu
Click the green Add Tools button, then select “Github” in the tool list
Click the blue Add Tool button
Grant the requested permissions
Once the list of repositories has loaded, review the list of enabled repositories and disable any that aren’t needed for your use case.
Ensure you have the Owner role in your GitHub organization
If you do not have the Owner role, this process will submit a request to install the app that an organization owner will need to approve.
Navigate to Allstacks
Go to the Tools page in the top right organization menu
Click the green Add Tools button, then select “Github App” in the tool list
Click the blue Add Tool button
Install the Allstacks GitHub App
Once the list of repositories has loaded, review the list of enabled repositories and disable any that aren’t needed for your use case.
Click “Configure” in the lefthand sidebar and select “Metrics” in the pop-out menu, like below:
On the “Metrics” page, scroll down until you see “Copilot Usage” at the bottom of the page. There, you’ll see the option to select either the “Copilot Adoption Rate” or “Copilot Suggestion Acceptance” metrics:
Usage metrics are generated by pulling data from Copilot API.
A “suggestion” by Copilot is defined by either a prompt or autocomplete.
The Tag Behavior
These tags behave like any other tags. They can be used to filter or group data in any metric. Specifically this set of tags will help users understand the differences in behavior and output between copilot and non-copilot using developers.
“Copilot Activity 0-7 days ago”
“Copilot Activity 8-30 days ago”
“Copilot Activity 31-60 days ago”
“Copilot Activity 61-90 days ago”
“No Copilot Activity in past 90 days”
“Not Invited in Copilot” (never invited, therefore never accepted)
Old Tag | New Tags |
Copilot User | “Copilot Activity 0-7 days ago” |
“Copilot Activity 8-30 days ago” | |
“Copilot Activity 31-60 days ago” | |
“Copilot Activity 61-90 days ago” | |
Not a Copilot User | “No Copilot Activity in past 90 days” |
“Not Invited in Copilot” (never invited, therefore never accepted, there) |
The adoption rate metric measures the effectiveness of the rollout of Copilot at your organization. You will be able to drill down to see the contributor list in each category.
Active - A user who took action that week.
When you actively use Copilot to generate code suggestions.
When Copilot processes your context to provide inline suggestions.
When you accept a Copilot suggestion.
When you interact with Copilot chat.
Inactive - A user who took no action that week.
This metric measures the number of suggestions accepted and suggestions rejected over time. This data is pulled from users that are defined as “active.”
With the copilot integration, your contributors will get an auto-generated tag that can help you take a new perspective on the metrics your team uses to measure productivity, code quality, and coding productivity. This will allow you to compare the output of your cohort that uses copilot against those who are not yet using copilot to speed their development.
Unable to connect
OAuth App
Verify that the organization you wish to connect doesn’t have OAuth app restrictions enabled, or that Allstacks is marked as an approved app.
https://docs.github.com/en/organizations/managing-oauth-access-to-your-organizations-data/about-oauth-app-access-restrictions
You can request approval to install the app with these steps: https://docs.github.com/en/account-and-profile/setting-up-and-managing-your-personal-account-on-github/managing-your-membership-in-organizations/requesting-organization-approval-for-oauth-apps
The app access can be approved by an organization owner with these steps: https://docs.github.com/en/organizations/managing-oauth-access-to-your-organizations-data/approving-oauth-apps-for-your-organization
GitHub App
When attempting to install the Github App, if you’re taken to a screen that looks like this, it means that your Github account is not a member of any organizations. Ensure that you’re a member of an organization before attempting to connect.
Metrics Not Appearing
Verify that Copilot is properly connected under “tools”
Check that your GitHub connection is active
Ensure you have the correct permissions
Access Issues
If connected with the OAuth app method, confirm the authorizing user has the Owner role in the org.
Verify the GitHub App installation status and permissions (if using GitHub App). This can be done in the org settings, go to the installed Github Apps list, and click Allstacks Analytics. Ensure that access is granted to the needed repos and that no permissions updates are pending approval.
https://github.com/organizations/%3Cyour%20org%20name%3E/settings/installations
Ensure that “Copilot Metrics API access” is enabled for your organization. This is found on the org settings page under Copilot -> Policies.
Data Sync Problems
The initial data synchronization may take up to 24 hours for the nightly sync to complete. Check the metrics library after 24 hours to see if the copilot metrics are available.
If you encounter any issues or have questions:
Contact your Customer Success Manager (CSM)
Provide details about your GitHub environment
Include any error messages or specific issues you're experiencing
For additional support or questions about unsupported GitHub versions, please reach out to your CSM directly.