Feedback Requested: Actions Usage Metrics

[cgtechcommunity]

What’s new:

Today, we are thrilled to announce that Actions Usage Metrics is available in public beta for GitHub Enterprise Cloud plans. 🎉

Highlights:

• Actions Usage Metrics provide detailed insights into where time is spent across workflows, jobs, repositories, runtime OS, and runner types, aiding in identifying improvement opportunities.
• Users can customize data views by filtering through various parameters like workflows, jobs, and more, with the ability to download this data as a .csv file.
• Access to Actions Usage Metrics is default for organization owners but can be extended to others through fine-grained permissions, facilitating data-driven decisions and enhancements.

Do you have questions, comments, concerns, or want to share some kudos? Share your thoughts here. We would love to hear your input! 🙏

Read more about Actions Usage Metrics in the Changelog post or in our docs.

[reiniertimmer]

Looks great already!

It would be very nice if the repositories could be filtered on custom properties, especially when managing 100s of repositories this would be very convenient

[reiniertimmer]

To give some idea about our case, we are dealing with a single GitHub organisation with 10k+ repositories.

These are from different teams, departments and even different companies. We use repository custom properties to identify the owners of all repos.

It would be great to get some insight in the costs per department/team by using these properties.

[reiniertimmer]

It would also be very helpful if the values of Total minutes for last 30 days and Total job runs for last 30 days would show the information based on the filter selection.

If I view the amount of minutes filtered on type=self-hosted (for example), I would expect the summary to show the total metrics for the active filter, instead of always showing the summary for all repositories in the organization.

[tonyjcamp]

Thanks for the feedback @reiniertimmer - I'm wondering if you'd expect to see different rollups per tab?

[reiniertimmer]

Hi @tonyjcamp you mean a different kind of summary when a different tab is opened?

In that case, I don't really think so. Using total minutes (or total cost) for the given query seems ok to me. As long as it represents the currently active filter, that would already be a large win.

[reiniertimmer]

It would be interesting to save some views though (so you can see some pre-populated dashboards based on some pre-defined queries). Right now, these need to be assembled by hand (or shared via a link).

Having a list of "curated" dashboards would seem like an interesting thing to have here as well

[usmonster]

Really nice! There are three essential things missing that we'd love to see:

• The same insights, but at the Enterprise level
• A column/filter for cost (I am aware of Updated billing overview page  github/roadmap#217, but I believe this is also a relevant Actions insight)
• Some kind of graphic visualization(s) (nice to have, but wasn't there something like that in the private beta? was it removed?)

This is especially interesting since we're using a mix of self-hosted and GitHub-hosted runners across multiple organizations.

Thanks!

[lukasz-mitka]

Same here.
Missing cost makes it useless for me. We have a lot of larger runners generating most of the cost for org, but they would go unnoticed when minutes are the core focus metric.

[tonyjcamp]

Thank you for the feedback here!

The same insights, but at the Enterprise level

We do want to bring this up to the Enterprise level. I do wonder who all should have access to these metrics at the Enterprise level?

A column/filter for cost (I am aware of Updated billing overview page github/roadmap#217, but I believe this is also a relevant Actions insight)

We made an explicit decision to keep cost separate from usage. We learned that DX or DevOps teams who manage Actions are primarily interested in how much time is being used. They use this information to identify outliers and optimize those workflows which helps their teams move faster.

Some kind of graphic visualization(s)

We hear you on this one. What types of graphs would help you the most? What would help your teams the most?

[qaisjp]

We (Stripe) would like to be able to grant everyone access to all metrics.

(If you have read access to the repo, you should be able to see its Actions metrics.)

[usmonster]

Thanks for the swift response, @tonyjcamp!

We do want to bring this up to the Enterprise level. I do wonder who all should have access to these metrics at the Enterprise level?

I imagine this can be granted to enterprise owners since they'd be the ones most interested in this comprehensive view that allows them to compare usage across organizations. It could also eventually be based on a custom Enterprise role.

We made an explicit decision to keep cost separate from usage. We learned that DX or DevOps teams who manage Actions are primarily interested in how much time is being used. They use this information to identify outliers and optimize those workflows which helps their teams move faster.

I understand the rationale behind this decision, but the optimization responsibilities of these DevOps teams includes optimizing costs. It is a big factor when considering a technical change that has financial impact, and so I still believe it would be extremely valuable here.

What types of graphs would help you the most? What would help your teams the most?

Time series of daily/weekly/monthly usage per "facet" (i.e. org, repo name/custom property, workflow, runner type/label, etc.) would be a great start. I have some examples I can share privately of something we've been doing with exported data if it helps.

Thanks again for engaging!

[shankarRaman]

Great work !!

[shankarRaman]

Is there an option to filter workflow/jobs based on custom runner labels?

[tonyjcamp]

Hi @shankarRaman - thanks for the feedback.

Is there an option to filter workflow/jobs based on custom runner labels?

Are you more interested in the label (name) or are you looking for more detailed information about the runner (cores, ram, etc)?

[shankarRaman]

Interested in Label (name) - we have a bunch of self-hosted runners that have different labels and I'd like to have the ability to see how different runner groups are performing.

[fortmate]

Thank you for such a great jobs!!
Could you filter data by default to show only the info the viewer has priviledge to access to?
There are several repos in my org which should not be shown their repo's names nor the workflow names.
We'd like to let all developers in my org to see their own usage data.
Is there anyone suffering from the same problem?

[tonyjcamp]

We don't have plans to create permissions for viewing specific repositories in the organization but we are exploring bringing this up to the enterprise level and down to repositories. I could see a repo view helping but you'd have to go to view each repo's metrics individually.

[fortmate]

Thank you for your answer.
Then I'll let non-Organization-owner fetch the repository name list and
filter csv (which has been created Organization owner) with the list.

[ebickle]

@tonyjcamp We recently received requests from our developers asking for the same thing. Instead of having a per-repository permissions model, could the UI be filtered so that "repo admins" can view their own repositories and statistics?

Using GitHub Advanced Security as an example: the organizational "Security" tab is accessible by everyone, but filtered based on the level of access to repositories.

We don't want to grant a global org-level permission to individual developers, but we still want them to be able to monitor their actions usage and help drive down costs.

[tonyjcamp]

Thanks for the feedback @ebickle! We plan to get down to the repository level in the future. I did want to ask about this comment:

We don't want to grant a global org-level permission to individual developers

Actions fine grained permissions would allow you to create a role and only grant access to Actions Usage Metrics. This would include all workflows in the organization but people with this permission would only see Actions Usage Metrics, they would not have full organization owner permissions. Is this still too much for individuals to see? If so, do you mind sharing why? Thanks again!

[bobo]

Thank you!
This gives a great overview on an organizational level, and helps us find where to put in effort.
Some thoughts:

• This is exactly what i would like an api for, is that also planned? We would love to show this statistics in our developer portal.
• I would also like to have this statistics easily availible in the actions tab on a repository, showing the usage trend for a repository for the repository owners to be able to take action on it.
• I would love to be able to see the usage for a shared workflow. I don't see a way to find what shared workflows that use a lot of time today, since that will be spread out over many repoes and probably hidden.

[samdbmg]

👍 for an API - I think that would be enough for internal cross-charging? E.g. you could build a system that grabs Actions usage metrics per-repo, checks that against a list, and then charges the relevant internal budget for the team who own the repo? Need some way of finding owners for repos not in that list though, so maybe it's a different feature

[bobo]

For me, i think https://docs.github.com/en/rest/actions/workflows?apiVersion=2022-11-28#get-workflow-usage is fairly close, but i would like an endpoint per repository instead of per workflow. Calculating the cost from minutes is ok worst case.
Would be nice though to get the same data as the csv (https://docs.github.com/en/billing/managing-billing-for-github-actions/viewing-your-github-actions-usage) though.

[tonyjcamp]

APIs are definitely on our list but are still a ways out.

I would love to be able to see the usage for a shared workflow. I don't see a way to find what shared workflows that use a lot of time today, since that will be spread out over many repoes and probably hidden.

Do you mean a filter to show reusable workflows in the org?

[bobo]

Do you mean a filter to show reusable workflows in the org?

exactly.

[Flasheh]

Looks promising! But in its current state not that useful in our case where we use exclusively self-hosted runners. We're not that interested in total minutes/cost but more about performance per run.

• I'd love to see more insights on average runtime, min/max runtime, queue time, ... So I guess in general more focus on performance/efficiency instead of cost.
• A way to sort the table. As it's just sorted on Total minutes currently it seems. Especially useful if more metrics are included in the future.
• It's already been mentioned but filtering based on runner labels would be great.

[tonyjcamp]

Hi @Flasheh 👋 - We have plans to address performance in the future. For sorting, are you looking for column sorting?

[Flasheh]

Hey @tonyjcamp, looking forward to more performance insights! And yes I do mean column sorting. Within the currently available metrics, sorting by total amount of runs instead of minutes could also be interesting.

Additionally, I was playing around with the filters a bit and found some funky stuff.

• Filtering on certain qualifiers such as job name, workflow name, source repository shows a value for "Exclude ...". I'm confused as to what the purpose of that is. Or if it's even supposed to be there.
  

• When configuring a filter where you can select certain values from the dropdown, such as the source repository, it always shows the same values in the dropdown before you type anything to search. Not sure what that list is based on but it does mean that when you have previously configured a filter and then want to go change it, you'll first have to search for your values again if you want to untick a certain one. It would make sense to have the list of values you previously selected be visible to quickly untick a value.

• If the time period for the metrics can only go back 90 days, then the value search for filters should preferably not return archived repositories, old/deleted job or workflow names etc... which fall outside that 90 day period.

• This last one might be an actual bug. In the Jobs tab, I think filtering on job names where the name contains whitespace is currently broken. When you're using matrix jobs or jobs triggered from a reusable workflow the name will contain whitespace. Filtering on the specific repo returns all the jobs, yet when I try filtering on those job names containing whitespace I get no results.

[tonyjcamp]

@Flasheh - thank you for the feedback here. We'll get those bugs filed.

[tonyjcamp]

@Flasheh - The bug that shows "Exclude ..." as the first result in the filter dropdown should be fixed. Let me know if you're still seeing that one.

We're working on the issue with names that contain spaces

When configuring a filter where you can select certain values from the dropdown, such as the source repository, it always shows the same values in the dropdown before you type anything to search. Not sure what that list is based on but it does mean that when you have previously configured a filter and then want to go change it, you'll first have to search for your values again if you want to untick a certain one. It would make sense to have the list of values you previously selected be visible to quickly untick a value.

I believe it's ordered by recent activity.

[fortmate]

Hi, again.
Is it possible to see the increase and decrease over time?
I'm to check the usage abuse but I can't tell if the usage is abnormal( or not), simply by looking at sum.