📜 Jelle Pelgrims

That one time I spent a whole day debugging dollar signs

Every once in a while you stumble upon a bug that really stops you in tracks and keeps you busy for a few hours. I recently had this happen while working on some kubernetes deployments. While creating a new pod (in Node.js, using the k8s API) that connected to a remote postgresql database I suddenly found out that the database password - which I knew to be correct, as I had used it too many times before - didn't work anymore.

It just didn't work. The password was given to the pod through environment variables in the kubernetes deployment. Some Node.js code then took the password from the environment variables and used it to log in to the database. Not much room for things to go wrong, right?

Slightly annoyed by the fact that something so simple could still go so wrong, I grabbed the password from the k8s secret to test it directly on the database. Big surprise, it worked just fine. But then, why didn't it work? Maybe I could reason the bug away. The issue had to exist somewhere between the kubernetes deployment and the Node.js code that handed the password over to some postgreSQL client library.

Going for a top-down approach, I used kubectl describe to check what environment variables the pod had available. No problem there. It had to be the Node.js app then. I strategically placed some log statements and ran the application, only to come to the conclusion that the password that was logged to the screen was not the password that was present in the k8s secret. The password stored in the kubernetes secret contained two consecutive dollar signs. Much to my amazement I only saw one dollar sign in the password shown on the screen.

While frantically looking through the kubernetes documentation for anything that might explain why this was happening to me, I stumbled onto the documentation page for the kubernetes API EnvVar entity. As it turns out, kubernetes environment variables are not static but can refer to other environment variables using the $(VARIABLE) syntax. This feature, which was not at all well documented in the kubernetes documentation, somehow malformed my password.

As for the question of why it malformed my password; I'm still not sure (and so are other people). The documentation states that the $(VAR_NAME) syntax can be escaped with a double "$$". That seems logical enough, but why does it already escape sequences consisting of "$$" instead of sequences consisting of "$$(" ? I can only imagine how many people will end up going down the same rabbit hole as I did.

In the end I solved the issue by wrapping every environment variable with a javascript function that doubled every dollar sign. This is the javascript monstrosity I came up with to fix the issue:

function EscapeK8sEnvVariable(string) {
    return string.replace('$', '$$$$');

Note how the function uses four consecutive dollar signs in the replace function.

I can already hear you thinking: "Wait, why four? Didn't he just say we needed to double every dollar sign, not quadruple?" Well, we want to replace each dollar sign in the string with two dollar signs. This is to accomodate the kubernetes environment variable substitution scheme, as explained before. So you'd think we'd just use string.replace('', '**'), right? WRONG! Javascript also has its own string substitution scheme that uses dollar signs. So, to get the correct number of dollar signs in the string we need to escape every dollar sign in the replace function with another dollar sign, turning two dollar signs into four.

I still distinctly remember questioning certain choices I had made in my life when writing that function.