While working on a new requirement for a page I had taken over from my predecessor, I happened to find one line of code.
I wondered if I had misread it.
I checked the browser developer tools, hoping it was not what I thought. It was plainly visible.
$.ajax({
url: 'https://some-api.com/items/recommendations',
data: JSON.stringify(data),
beforeSend: function(xhr){xhr.setRequestHeader('x-api-key', 'k8nSt...')},
...
})
I immediately revoked the key and made a hotfix that moved the logic to the server.
The usage history showed no signs of suspected misuse. But because the key had been exposed on an old JSP page, it was difficult to determine whether it had actually leaked.
I was lucky to find it this time, but other pages could have the same problem. We decided to inspect all related repositories as quickly as possible.
How Do We Find Hidden API Keys?
API keys are generally long and contain irregular mixtures of many characters so that attackers cannot guess them.
Shannon entropy can quantify how irregular a string is.
For example, if we calculate the entropy of the following two strings, the first has a higher value.
### Comparing Shannon entropy values
1. A0D28bba7b1287ff432d0b : 3.5161 # Irregularly mixed string
2. aaaaaaabbbbbbbbccccccc : 1.5820 # Three repeated lowercase letters
This is because the first string uses more kinds of characters than the second, and the frequency of each character is relatively more distributed.
But a high Shannon entropy value does not necessarily mean that a string is an API key.
A UUID, for example, also has fairly high entropy.
c1b100db-f254-493f-8b53-f0fcab2437d0: 3.7391
Entropy alone therefore cannot determine whether a string is an API key. It can, however, help narrow down suspicious candidates.
For a quick inspection, I wrote a script that calculated the entropy of strings in the source code and produced a report. It extracted about 100 suspicious strings from ten repositories. We found and addressed one more key that had been externally exposed.
================= Example List of Detected Strings ================
EXAMPLE-KEY-8bba7b1287ff432d0b7777532f52fd59ef75cc56d3818b8ed5a8d06 : 4.3197
5b8621ee0cf2475f66709a5d : 3.7721
eventEntry1Remark : 3.4548
1842692849310691 : 2.9056
P2018043031858 : 2.8424
...
Add the Check to the Process
The script was a temporary tool for quickly inspecting all related repositories.
At the time, we were already considering introducing static analysis with SonarQube, which also had a secret detection feature. Instead of operating the script separately, we decided to include this check in the existing build flow.
I configured the build process to perform static analysis on the source code. If it found issues above our internal threshold, both the Quality Gate and the build failed. If the same kind of problem occurs again, this should help us find and address it before it reaches production.
Retrospective
When I first found the API key, I thought only about removing the immediate problem. Establishing a new check that would prevent the same problem from recurring was a separate task.
There was a lot to consider: whether the system could operate reliably for a long time in an enterprise environment, and how to convince team members that the new process was necessary. SonarQube was unfamiliar to everyone, so I explained the changed deployment flow and the Quality Gate thresholds in detail. I also documented the restart procedure needed during a failure, focusing on making the information part of the team’s shared knowledge.
It may seem that having the system work now is enough. But it needs to keep working for a long time even after the person who introduced it is gone. A system that the company cannot maintain becomes an operational risk. Building or introducing something new also seems to come with responsibility for it.
Thank you for reading this long article.