- The Mess: Grafana Labs lost control of part of its GitHub environment after attackers leveraged a compromised workflow token that survived an emergency token rotation following the TanStack npm supply-chain attack.
- The Damage: Attackers downloaded Grafana source code and internal repositories, then demanded a ransom. Customer production systems and Grafana Cloud were not compromised.
- The Fix: Rotate every credential after a supply-chain incident, verify the rotation independently, and treat CI/CD tokens as production-grade secrets.
The breach did not start with Grafana Cloud.
It did not start with a vulnerable Grafana server.
It started in the software supply chain — and then survived because one GitHub workflow token was missed.
Grafana Labs confirmed that attackers gained unauthorized access to its GitHub environment in May 2026 and downloaded company code. The incident was subsequently traced to the TanStack npm supply-chain attack associated with the Mini Shai-Hulud campaign.
That alone would have been bad.
The interesting part came next.
Grafana’s security team rotated a significant number of GitHub workflow tokens after detecting the initial compromise.
One token was missed.
That token was enough.
The attack chain
The basic chain looked like this:
TanStack npm supply-chain compromise
↓
Grafana CI/CD exposure
↓
GitHub workflow token
↓
token rotation begins
↓
one token missed
↓
attacker retains access
↓
Grafana GitHub environment
↓
source code downloaded
↓
ransom demand
This is a much more interesting failure than another story about someone forgetting to patch a server.
The initial compromise was external.
The persistence was internal.
And the mistake was operational.
The token that survived
Grafana said its investigation found that attackers entered through a compromised GitHub workflow.
The company immediately initiated incident response and rotated a large number of automation tokens.
But the rotation process relied on an assessment of which workflows were affected.
One workflow was originally considered unaffected.
It wasn’t.
The associated token therefore survived the first rotation.
That gave the attackers continued access to Grafana’s GitHub environment.
This is where incident response gets ugly.
After a supply-chain attack, defenders have to answer a deceptively simple question:
What credentials can the attacker possibly have seen?
Not:
Which credentials do we think they saw?
Those are different questions.
CI/CD credentials are production secrets
Developers sometimes treat GitHub Actions credentials as temporary automation plumbing.
That is a mistake.
A CI/CD token can potentially access:
- repositories;
- package registries;
- release infrastructure;
- cloud APIs;
- deployment systems;
- signing mechanisms;
- build artifacts;
- secrets;
- internal automation.
The token doesn’t need to be an administrator account to be dangerous.
It only needs enough permissions to move the attacker from one trusted system to another.
Grafana’s incident demonstrates precisely this problem.
The compromised environment wasn’t Grafana Cloud.
It was the development infrastructure around it.
But that infrastructure contained valuable intellectual property and internal information.
The attackers downloaded the code
Grafana confirmed that attackers accessed its GitHub repositories and downloaded the company’s codebase.
The affected repositories included both public and private repositories.
They also contained internal GitHub repositories used by teams to collaborate and store operational information and business details.
Grafana subsequently received a ransom demand threatening publication of the stolen code.
The company refused to pay.
That’s significant because the attacker didn’t need to encrypt anything.
They already had the valuable asset.
The extortion model was:
steal source code
↓
prove access
↓
demand money
↓
threaten publication
This is increasingly common in modern supply-chain attacks.
Availability doesn’t have to be destroyed.
Confidentiality is enough.
The important thing Grafana did not lose
There was no evidence that customer production systems were compromised.
Grafana’s subsequent investigation found no unauthorized access to customer production systems or the Grafana Cloud platform.
That distinction matters.
A compromise of a company’s GitHub organization does not automatically mean its customers were compromised.
The blast radius depends on what trust relationships exist between:
GitHub → CI/CD → cloud → production.
In Grafana’s case, the attackers were able to reach source repositories but did not obtain evidence of access to customer production infrastructure.
The company also reported that its source code was downloaded but not modified.
That’s a major difference between:
code theft
and:
supply-chain poisoning.
The nightmare scenario was code modification
Imagine the same attacker had obtained write access to a repository used to build a production package.
The attack would look very different.
Instead of:
GitHub → download source → ransom
you could get:
GitHub
↓
modify source
↓
CI/CD build
↓
signed artifact
↓
package registry
↓
customer deployment
At that point the attacker doesn’t need to compromise every customer individually.
The vendor’s own build pipeline becomes the delivery mechanism.
That is why CI/CD security deserves the same attention as production infrastructure.
The missed-token problem is bigger than Grafana
The most useful lesson here isn’t:
“Don’t get hacked through TanStack.”
It’s:
When you rotate credentials after an incident, prove that the old credentials are dead.
A rotation spreadsheet isn’t proof.
A script that says “rotated 99% of tokens” isn’t proof.
A list of repositories believed to be affected isn’t proof.
The only meaningful question is whether an old credential still works.
That should be tested.
Assume your inventory is wrong
Modern GitHub organizations can contain enormous numbers of:
- repositories;
- GitHub Actions;
- workflow files;
- deploy keys;
- app installations;
- personal access tokens;
- organization secrets;
- environment secrets;
- cloud federation identities.
Some are actively used.
Some haven’t been touched for months.
Some belong to abandoned workflows.
Some are connected to repositories nobody remembers.
That’s exactly where credential rotation becomes dangerous.
Security teams tend to focus on the infrastructure they know exists.
Attackers benefit from the infrastructure nobody remembers.
Grafana brought in Mandiant
Grafana later engaged Mandiant for an independent audit.
The post-incident review published on June 23 reported that Mandiant found no evidence of code tampering or repository poisoning affecting public organizations or production repositories delivered to end users.
That’s an important finding.
The incident remained a source-code and internal GitHub compromise rather than a demonstrated customer-facing software supply-chain compromise.
But the investigation also highlights how close these environments are to becoming something much worse.
The difference between:
read access
and:
write access
can determine whether a breach becomes an embarrassing source-code leak or a global supply-chain incident.
GitHub is now part of the security perimeter
For organizations building software, GitHub is no longer merely where developers store code.
It’s part of the production infrastructure.
It can control:
- builds;
- releases;
- deployment;
- credentials;
- package publication;
- infrastructure automation;
- cloud authentication.
A compromised GitHub organization can therefore be more valuable than a compromised application server.
The application server contains the application.
The CI/CD environment can contain the keys to rebuild it.
What should defenders do?
After any supply-chain compromise, don’t stop at rotating the credentials explicitly identified by the vendor.
Perform a complete credential reset.
Audit:
GitHub Actions tokens
Invalidate and recreate automation credentials.
Deploy keys
Check every repository and remove unused keys.
GitHub Apps
Review installations, permissions and tokens.
Cloud identities
Check whether workflows can authenticate to AWS, Azure or GCP.
Package publishing
Review npm, PyPI, container registry and other publishing credentials.
Signing keys
Treat package and release signing infrastructure as potentially exposed.
Repository permissions
Remove unnecessary write access.
Workflow files
Look for unexpected modifications and newly introduced actions.
GitHub audit logs
Search for unusual repository access, token use, workflow execution and permission changes.
The supply-chain lesson
TanStack was the initial trigger.
Grafana’s missed token was the persistence mechanism.
GitHub was the target.
Source-code theft was the payoff.
And the attackers didn’t need to compromise Grafana Cloud.
That is the important architecture lesson.
Modern software companies have multiple security boundaries:
developer workstation
↓
GitHub
↓
CI/CD
↓
package registry
↓
deployment infrastructure
↓
production
Breaking any one of these doesn’t automatically compromise the entire chain.
But every additional trust relationship creates another route.
The job of defenders is to make those routes as narrow as possible.
Bugstoday’s take
Grafana didn’t lose its source code because somebody forgot to patch Grafana.
It lost the code because a CI/CD credential survived an emergency rotation.
That’s much more uncomfortable.
The first lesson from supply-chain incidents is usually:
Don’t trust third-party packages.
The second lesson should be:
Don’t trust your own credential inventory either.
If a security team cannot prove exactly which automation tokens exist, where they are used, what they can access and whether they were actually revoked, then the organization doesn’t really control its CI/CD environment.
It only hopes it does.
And hope is a terrible secrets-management system.
Today’s Bugs. Tomorrow’s Breaches.
Technical Sources
- Grafana Labs — official security update describing the GitHub compromise, missed workflow token, source-code theft and response.
- Grafana Labs — official post-incident review and independent Mandiant assessment.
- Grafana Labs — security advisory archive and current security disclosures.
