You Don’t Know Where Your Secrets Are

Do you know where your secrets are? If not, I can tell you: you are not alone.

Hundreds of CISOs, CSOs, and security leaders, whether from small or large companies, don’t know either. No matter the organization’s size, the certifications, tools, people, and processes: secrets are not visible in 99% of cases.

It might sound ridiculous at first: keeping secrets is an obvious first thought when thinking about security in the development lifecycle. Whether in the cloud or on-premise, you know that your secrets are safely stored behind hard gates that few people can access. It is not just a matter of common sense since it’s also an essential compliance requirement for security audits and certifications.

Developers working in your organization are well-aware that secrets should be handled with special care. They have put in place specific tools and procedures to correctly create, communicate, and rotate human or machine credentials.

Still, do you know where your secrets are?

Secrets sprawl everywhere in your systems, and faster than most realize. Secrets are copied and pasted into configuration files, scripts, source code, or private messages without much thought. Think about it: a developer hard-codes an API key to test a program quickly and accidentally commits and pushes their work on a remote repository. Are you confident that the incident can be detected in a timely manner?

Insufficient audit and remediation capabilities are some of the reasons why secrets management is hard. They are also the least addressed by security frameworks. Yet these grey areas—where unseen vulnerabilities remain hidden for a long time—are blatant holes in your defense layers.

Recognizing this gap, we developed a self-assessment tool to evaluate the size of this unknown. To take stock of your real security posture regarding secrets in your organization, take five minutes to answer the eight questions (it’s completely anonymous).

So, how much do you not know about your secrets?

Secrets Management Maturity Model

Sound secrets management is a crucial defensive tactic that requires some thought to build a comprehensive security posture. We built a framework (you can find the white paper here) to help security leaders make sense of their actual posture and adopt more mature enterprise secrets management practices in three phases:

1. Assessing secrets leakage risks
2. Establishing modern secrets management workflows
3. Creating a roadmap to improvement in fragile areas

The fundamental point addressed by this model is that secrets management goes well beyond how the organization stores and distributes secrets. It is a program that not needs to align people, tools, and processes, but also to account for human error. Errors are not evitable! Their consequences are. That’s why detection and remediation tools and policies, along with secrets storage and distribution, form the pillars of our maturity model.

The secrets management maturity model considers four attack surfaces of the DevOps lifecycle:

• Developer environments
• Source code repositories
• CI/CD pipelines & artifacts
• Runtime environments

We then built a maturity ramp-up over five levels, going from 0 (Uninitiated) to 4 (Expert). Going 0 to 1 is mostly about assessing the risks posed by insecure software development practices, and starting auditing digital assets for hardcoded credentials. At the intermediate level (level 2), secrets scanning is more systematic, and secrets are cautiously shared across the DevOps lifecycle. Levels 3 (Advanced) and 4 (Expert) are focused on risk mitigation with clearer policies, better controls, and increased shared responsibility for remediating incidents.

Another core consideration for this framework is that making it hard to use secrets in a DevOps context will inevitably lead to the bypassing of the protective layers in place. As with everything else in security, the answers lay between protection and flexibility. This is why the use of a vault/secrets manager starts at the intermediate level only. The idea is that the use of a secrets manager should not be seen as a stand-alone solution but as an additional layer of defense. To be effective, it requires other processes, like continuous scanning of pull requests, to be mature enough.

Here are some questions that this model should raise in order to help you evaluate your maturity: how frequently are your production secrets rotated? How easy is it to rotate secrets? How are secrets distributed at the development, integration, and production phase? What measures are put in place to prevent the unsafe dissemination of credentials on local machines? Do CI/CD pipelines’ credentials adhere to the least privileges principle? What are the procedures in place for when (not if) secrets are leaked?

Reviewing your secrets management posture should be top of mind in 2023. First, everyone working with source code has to handle secrets, if not daily, at least once in a while. Secrets are no longer the prerogative of security or DevOps engineers. They are required by more and more people, from ML engineers, data scientists, product, ops, and even more. Second, if you don’t find where your secrets are, hackers will.

Hackers will find your secrets

The risks posed to organizations failing to adopt mature secrets management practices cannot be overstated. Development environments, source code repositories and CI/CD pipelines have become favorite targets for hackers, for whom secrets are a gateway to lateral movement and compromise.

Recent examples highlight the fragility of secrets management even in the most technologically mature organizations.

In September 2022, an attacker got access to Uber’s internal network, where he found hardcoded admin credentials on a network drive. The secrets were used for logging in to Uber’s privileged access management platform, where many more plaintext credentials were stored throughout files and scripts. The attacker was then able to take over admin accounts in AWS, GCP, Google Drive, Slack, SentinelOne, HackerOne, and more.

In August of the same year, the password manager LastPass fell victim of an attacker who gained access toits development environment by stealing the credentials of a software developer and impersonating that individual. Later in December, the firm disclosed that someone used that information to steal source code and customer data.

In fact, in 2022, source code leaks have proven to be a true minefield for organizations: NVIDIA, Samsung, Microsoft, Dropbox, Okta, and Slack, among others, have been victims of source code leaks. In May, we were warning about the important volume of credentials that could be harvested by analyzing these codebases. Armed with these, attackers can gain leverage and pivot into hundreds of dependent systems in what is known as supply chain attacks.

Finally, even more recently, in January 2023, the continuous integration provider CircleCI was also breached, leading to the compromise of hundreds of customer environment variables, tokens, and keys. The company urged its customers to immediately change their passwords, SSH keys, or any other secrets stored on or managed by the platform. Still, victims need to find out where these secrets are and how they are being used to press the emergency button!

This was a strong case for having an emergency plan ready to go.

The lesson from all these incidents is that attackers have realized that compromising machine or human identities gives a higher return on investment. They are all warning signs of the urgency to deal with hardcoded credentials and to dust off secrets management in general.

Final word

We have a saying in cybersecurity: “encryption is easy, but key management is hard.” This still holds true today, although it isn’t just about encryption keys anymore. Our hyper-connected services world relies on hundreds of types of keys, or secrets, to function properly. These could be as many potential attack vectors if mismanaged.

Knowing where your secrets are, not just in theory but in practice, and how they are used along the software development chain is crucial for security. To help you, we created a maturity model specifically about secrets distribution, leak detection, remediation process, and rotation habits.

The first step is always to get a clear audit of the organization’s security posture regarding secrets: where and how are they used? Where do they leak? How to prepare for the worst? This alone could prove to be a lifesaver in an emergency situation. Find out where you stand with the questionnaire and learn where to go from there with the white paper.

In the wake of recent attacks on development environments and business tools, companies that want to defend themselves effectively must ensure that the grey areas of their development cycle are cleared as soon as possible.