My favorite are the systems where you can only issue one token, so that you can't do a zero downtime rotation by creating new one, making it active in your system, and only then removing the old one.
In some cases this makes rotation a big event to be avoided because costs are higher than gains.
I am still surprised that Keycloak makes this so hard. They finally added support for n=2 but it’s still walled off behind a “this is experimental, use at your own risk” warning, and it’s something that literally every OIDC client needs to do if you have any kind of compliance requirements.
Okay but now how do you recommend I hook up my Sentry instance to create tickets in Jira, now that Jira has deprecated long-lived keys and I have to refresh my token every 6 weeks or whatever. It needs long-lived access. Whether that comes in the form of a OAuth refresh token or a key is not particularly interesting or important, IMO.
OIDC with JWT doesnt need any long lived tokens. For example, I can safely grant gitlab the ability to push a container to ECR just using a short-lived token that gitlab itself issues. So the answer might be to ask your sentry/jira support rep to fast track supporting OIDC JWTs.
I disagree, I think increasing manual toil (having to log into Sentry every 6 months to put in a new Jira token) increases fatigue substantially for, in this case, next-to-no security benefit (Sentry never actually has any less access to Jira than it does in the long-lived token case, and any attacker who happens to compromise them is going to be gone well before six months is up anyway).
Instead, the right approach in this case is to worry less about the length of the token and more about making sure the token is properly scoped. If Sentry is only used for creating issues, then it should have write-only access, maybe with optional limited access to the tickets it creates to fetch status updates. That would make it significantly less valuable to attackers, without increasing manual toil at all, but I don't know any SaaS provider (except fly, of course) that supports such fine-grained tokens as this. Moving from a 10 year token to a 6 month token doesn't really move the needle for most services.
But then you just move the security issue elsewhere with more to secure. Now we have to think about securing the automation system, too.
This is the same argument I routinely have with client id/secret and username/password for SMTP. We're not really solving any major problem here, we're just pretending it's more secure because we're calling it a secret instead of a password.
Secrets tend to be randomly-generated tokens, chosen by the server, whereas passwords tend to be chosen by humans, easier to guess, and reused across different services and vendors.
> If you assume that someone is constantly trying to guess a key or password, the likelihood that they guess correctly grows over time.
If they can brute force the password or key, the rotation will, at best, force them to do it multiple times. You'll see more improvement from just adding another couple of characters to the length.
Fair enough, but that doesn't protect you in case of a leak. If you're going to solve for the leak anyway, is it worth it to solve for brute force in isolation? You can always add another couple of characters. At which point do you stop?
Even still, I prefer the simplicity of API keys. The mental overhead is low, and having to explain the concept to customers is zero. Rotating keys is not great but the tedium of it is preferred over the labyrinthian shenanigans of getting whatever the hot new security style of the week setup is.
Agreed! Been working on infra for an early-stage company recently and it's been awesome using OIDC and IRSA (or WIF if you're on google) for as many things as possible. Basically, there are no permanent keys for anything.
Slightly annoying to have to wrap some clis in scripts that generate the short-lived token, but it feels really magical to have services securely calling each other without any explicit keys or password to even store in our vault.
Lots of cool benefits --- for instance, we ran the compromised Trivy github action a few weeks ago, but our Github Actions had 0 keys for it to leak! Also really great that I don't have to worry about rotating shared credentials on short notice if an engineer on my team decides to leave the company.
After the Vercel hosting compromise and having to rotate a ton of keys recently, we are definitely implementing automated rotation of short lived keys. That was super painful.
The long-lived credentials life inside a stripped down machine. Cron/lego/Ansible handles the renewal. The machines on the edge can't renew their keys themselves.
What about dynamic credentials. Why can't we deploy HSM(hardware security module), they are so much more affordable now. We then deploy fido2 keys, have our long lived keys in there and have HSM serve as dynamic credentials server.
Something I don't understand is the absolute phobia of service accounts. There are things that need to happen regardless of who is doing it. Emails need to get sent every day with reports, for example.
Forcing these workflows into the nonsense security theater of "we can't have service accounts" is stupid and unproductive. So every time we fire or lay off the person whose name is on the automation, we need to rotate the keys? What is the benefit here?
If you are screaming "managed identity" here, I have a bridge to sell you because clearly even Microsoft has not been able to figure out or implement managed identities for internal workloads... Well not as of 2022, at least.
Service accounts are great! I just wish instead of having a password which gets shared around via 1password, there were a clear permission list ("this is a service account.. "real" users X, Y, X can login as it")
Seems like it's just Microsoft that cannot figure it out. AWS had roles forever, fully supported from web console or CLI. But when I request Azure service account, I am handed username and password.
Totally, but my service accounts own the api keys. But keys are still annoying to rotate. You know what’s not annoying to rotate? Short-lived tokens with very limited scope that get assigned more on demand
I’m sorry to be pedantic, that’s not exactly true. I agree in the sense that extracting hw based keys is next to impossible, but if your machine is compromised, there isn’t much stopping malware from using your hw based key (assuming 1. Left plugged in, 2. Unlocked with either ssh-agent or gpg-agent, and 3. You don’t have touch to auth turned on). Reduced risk? Absolutely. No risk? Absolutely not.
And if you want to be even more pedantic, shell access with a touch based key just means the attacker has to wait for you to auth, which makes touch based systems largely a waste of effort on the defenders part.
> shell access with a touch based key just means the attacker has to wait for you to auth
And if you want to be EVEN more pedantic, on most touch-based keys, you have to touch within 10–15 seconds otherwise it times out.
So it is not a waste of effort at all. First the need to touch at all eliminates a large chunk of attacks. Second the need to touch within 10–15 seconds eliminates a whole bunch more.
There would have to be some heavy-duty alignment of ducks going on to get past a touch requirement.
Even more if the target has touch AND PIN enabled.
The touch based key I use only responds once per touch. If someone compromises the machine it's plugged into, the action I expected to complete won't complete. This means the compromise becomes immediately visible.
Part of the threat model for an Engineering team is that people come and go. They move teams which have different levels of access. They leave the organization, in most cases, on good terms. I want to set up infrastructure where I don't need to remember that your SSH pubkey is baked into production configuration after you leave the company.
There are several options for setting up per-connection keys that are dispensed to users through the company SSO. That setup means you don't need to maintain separate infrastructure for (de-)provisioning SSH keys.
This is completely solved by SSH certificates. You still have the same private key in the hardware, but instead of using the public key directly, you issue temporary (~1 hour) SSH key certificates. I even automated it using an SSH proxy.
The target machines then just need to put the CA cert in the authorized_keys files.
It didn't for me, and I got the starry space feel, but I noticed the repeating patterns.
Perhaps some movement is needed? I do recall some relatively similar cases saved, if interested:
1. Moving forward in space (JavaScript/JS): https://codepen.io/the_artwork/pen/zYEdxyo
2. Rotating in space (JS): https://codepen.io/the_artwork/pen/NWMRYJP
3. Rotating in space (CSS+JS): https://codepen.io/the_artwork/pen/PoeNyyy
Where possible I prefer to implement signed policy objects. Then I can constrain access based on source IP and other request parameters. You can also easily implement an expiration date if you feel any particular application requires it, but some simple constraints may be useful enough that you might skip this in the majority of server to server applications.
This not only provides security but provides some resistance to bugs in your code which either call services incorrectly or call the incorrect methods of a service. I've avoided accidental data deletions and other painful events because my policy document did not allow this action. It turns these bugs into failures at the security perimeter.
I've used this concept in a few user applications as well. Typically those documents will always have expiration dates and I'll provide a "license" API which allows a single authenticated client request to retrieve an appropriate policy document. This is particularly nice when you want to implement a service across different providers or want to avoid downstream authentication overhead.
You dont usually want keys at all. At least in the sense of copy this key from system A and paste it in this other place system B. Usually CI. You want some continual method of authentication and authorization.
Some magnificent systems have APP_KEY/APP_SECRET that is also used for cookie and database encryption. A frequent rotation of this is... inadequate... in systems with high traffic, to say the least, and hence I am sorry, but I do not believe it's the "usual" desire. As always, it depends on the context and transaction scope.