“Who are you, how do I know you?” I thought that the person on the other end of the phone was a coworker, somebody that I desperately needed to talk with. Our top ten internet site had been hacked and we didn’t know who or how, beyond knowing that it must have been a compromised account somewhere in the system. It was the holidays and nobody was in the office meaning that everything had to be done over the phone. And it was almost impossible to know if the phone numbers were real given that an attacker could have easily changed them in the Google Doc if they had compromised an employee’s Google account.
I was rapidly resetting and locking down credentials, but I needed to share the new credentials with some select, key individuals, which of course required me to know for sure that I was talking with the right people. In a perfect world we would all have trusted PGP keys that we had pre-shared and signed, but that would have only helped a select few. Eventually I needed to communicate with developers or support staff that wouldn’t have understood how to use PGP well enough given how notoriously complicated it can be to use it right.
Eventually we got through the chaos of the moment and cleaned up our vulnerability (which turned out to be a leapfrog from a compromised IT vendor into our Google Apps account, and eventually our DNS provider). In order to handle the identity issue highlighted above we added a new step, a code word that rotated and was shared between all the critical staff weekly. No major change could be done over the phone without this codeword. That was good, but there is a much better way…
My current preferred method for identity is to setup a TOTP, such as Google’s Authenticator system. Generate a QR code that can be shared in person only, ideally at on-boarding time and semi regular intervals from then on. This can be done easily here. Everybody adds that QR code to their phone and when needed they can be asked “what is the current rotating code?” Both employees should have the same code at the same time in their TOTP application. You can repeat this step for smaller groups if needed having one for all employees, and another for production staff. A generalization and improvement is to use multiple TOTP code generators, giving limited numbers to each person so that its harder for an attacker that has compromised a coworkers phone to to imitate the person. This requires a little more infrastructure and pre-setup though. The best possible version of this is to create a TOTP code for each person pairing, so that it becomes extremely difficult to impersonate somebody else.
[Edit from 2024: Its gotten much easier to setup and maintain a zoom bridge or Google Meet room with video than it was in 2016 when I wrote this, let alone 2010 when the incident that inspired it occurred. This can mitigate much of the worry, however it doesn’t help when trying to bridge into departments and teams you have not ever met before, and also doesn’t mitigate the possibility of a deep fake attack.]
This approach would have made my life vastly simpler during that chaos by helping us to establish trust much faster! No matter what method you use its important to make sure you know how you would figure out who you could trust before a major hack of this sort happens.