I didn't choose key management. It chose me.
Early in my career I was a one-man IT show at a company that printed health and financial cards — mag stripe, embossing, the whole thing. We had a data center, a rack full of servers, a handful of HSMs, and a compliance calendar that kept getting more complicated every year. When PCI became a real thing for us, my boss at the time started writing policies and needed help. I was curious by nature, so I got pulled in.
That's how I became a key custodian. Not because I studied cryptography. Not because I was the security guy. Because someone had to do it and I was there.
I didn't understand why we received the keys that we did, or how they were actually used. I mean that literally. I knew we received them in the mail — three parts, three different carriers, UPS, DHL, FedEx, you couldn't send all three via the same medium — and I knew we had to enter them into the HSM in a certain sequence, in a freezing server room, with at least two people, motion sensors armed, no phones, door locked. An hour and a half minimum. No exceptions.
I knew the procedure. I just didn't understand what I was protecting or why any of it mattered.
That's more dangerous than it sounds.
The Ceremony
If you've never done a key ceremony, here's what it looks like: the keys arrive in a sealed envelope, sometimes formatted like a Scantron sheet, often handwritten. You take that paper into the server room, you lock the door, you type what's written, you generate a check value, you call the customer, you compare. If they match, you're done. If they don't — and about 10% of the time, they didn't — you start over. Wait two or three more days for corrected keys, reschedule the staff, and pray that the key parts arrive as scheduled.
And it never got faster. I don't care how many times you'd done it. The process still took the same amount of time, or longer, because your brain treated it like the first time every time. You'd forget the sequence. You'd second-guess yourself. Is that a G or a 6? Is that a 1 or an I?
Meanwhile, the rush order is sitting in the queue. The print shop is idle. And you're in a freezer arguing with a firmware bug.
The Part Nobody Talks About
Here's the thing that stayed with me after I finally got some distance from it: we were all just robotically following steps. None of us really understood the risk. We weren't shredding papers and isolating servers because we'd thought through the threat model. We were doing it because an auditor said so and the policy said so, and that was the end of it.
That disconnect is where most key management programs fall apart. Not because people are careless — most of them aren't. But because when you don't understand why something matters, you find ways to make it easier on yourself. You copy a key part into Notepad so you don't mistype it. You leave a session open while you go to lunch because you'll be back in twenty minutes. You write the combination to the safe in your notes app because you can never remember it.
Every one of those shortcuts is a crack in the wall. And the auditors always find them. Stale keys on servers. Keys that traveled through systems they weren't supposed to touch. Logbooks with missing entries. Every quarter, findings. Every finding, a remediation clock. Miss the deadline, end up on a probationary vendor list — public, searchable, embarrassing.
I watched it happen over and over, not because people were lazy or incompetent, but because they were managing something they didn't fully understand, on top of jobs they were already stretched thin doing. Key management wasn't anyone's main role. It was the thing you also had to do.
What I Know Now
Looking back on it, I can see clearly what I couldn't see from the inside: the human element isn't a variable in key management. It's the whole problem.
The encryption itself was fine. It did exactly what it was supposed to do. But everything built around it — the ceremonies, the logbooks, the manual entry, the shipping logistics, the key rotation schedules, the audit trails — all of it required people to do the right thing, every time, under pressure, without fully understanding why it mattered.
They usually did. Until they didn't.
The goal, if you're building something new, shouldn't be better ceremonies or stricter policies. There should be fewer humans in the loop. Ideally: none.
That's a longer conversation. But it starts here.
