Archive for January, 2025

Usable end-to-end encryption for cloud services

Saturday, January 11th, 2025

In the last year I’ve gotten a lot of questions from people who are interested in privacy and end-to-end encryption about which online services are safe to use. This has led to reading a lot of documents about the design of these services, from iCloud to 1Password to Cryptpad and others. I’ve seen a few different designs at this point for how to handle keys and logins for services where you want to hold data that is encrypted by the user and you don’t hold the key. Of course one could encrypt their files with PGP/GPG and upload them to Dropbox or Google Drive, making any online storage an encrypted service, but managing GPG keys is basically impossible, which these services need to solve.

I’ve found it interesting to learn about the latest thinking in how to make this key management usable for regular people, which often means reducing it down to deriving the key from one password that is used for both login and encryption. This would seem to create a big risk that the service provider has to know the user’s password, or password-equivalent material like a hash, at least some of the time like during login. What’s cool is that designs now exist that mostly or entirely avoid this. I’m going to describe a few I’ve seen that do (and don’t) accomplish this, and in particular the one I like the best, Firefox Sync via Mozilla Accounts, which I think has a nice design that could be adopted by others.

I’m not really a software engineer, and definitely not a cryptographer, so I may not be up to date on the latest recognized best practices in how to design applications like this. There may be a known gold-standard architecture that I haven’t heard of that everyone should be following. For example, for transport and messaging protocols, the advice is always “don’t roll your own”: use TLS, SSH, or Signal, and just wrap whatever you’re doing in that. But I think this is a fairly new thing that people are trying to do on this scale; prior to 2013, end-to-end encryption of stored user data was not a high priority for mass-market cloud services. You kind of do have to roll your own because there isn’t a standard that widely varying services can follow. My impression is that people have spent the last 10 years figuring this out, and I’m pleased to find that some of them seem to be getting there. Maybe some day we will reach a consensus on which of these is right and it can be packaged up into a nice library, but until then here’s what I’ve seen lately.

The wrong way: pinky swears

I have seen some services where the stored data is encrypted with a key derived from the user’s password, either on the server or locally, and the service provider doesn’t or can’t really make any promises about how they handle that password. This is more or less what Backblaze offers. Their default is that they use your login password to derive an encryption key on their server, which they re-calculate every time you login to add more files or download old files. They offer an option to instead provide your own encryption password, which is again used to derive the key on their server for encryption/decryption, and which they say they don’t store after the key derivation is done each time; this is slightly different than using the login password, since they do have to store a hash for handling logins, but can discard the encryption password each time they’re done using it. Unfortunately you’re left to trust them here. Hopefully they actually don’t store it. Doing the encryption on their server is not good for the paranoid. End-to-end encryption means the key never leaves your device and the provider can never learn it. I appreciate that they offer either of these methods, which is more than some backup providers, but this is not end-to-end encryption.

The annoying way: two passwords

You could improve on Backblaze’s design by doing the encryption locally instead of their server, so that the second password is never sent to them. This would be better for privacy, if not necessarily the efficiency of their data storage (presumably they do some kind of optimization in their data centers that requires doing the encryption there). It might not be better for the average user, who has enough trouble managing one password because most of them still don’t use password managers. Usability is a serious concern. If the second password is lost or forgotten, and everything actually works as it should, then so is the data. Maybe you can get by with a design like this for services targeted toward moderately technical people, like Backblaze or Proton, but this is risky for the general public.

In fact this is what Proton used to do. They had one password for logging in, managing your account, and obtaining session tokens for uploading and downloading content. Then they used a separate password to encrypt/decrypt the mail or files on the client, either a native app or the browser based version (let’s not get into the issue of dynamically downloaded Javascript web app cryptography). This probably worked for their original user base, the hacker con crowd. But they eventually got rid of this design because it was problematic for regular people. You really want the user to only have to remember (or store, in their password manager) one password to do everything, which it turns out is actually possible!

Proton: SRP

In 2016, Proton changed their authentication process to remove the need for a second password. The encryption key used on the client is now derived from the login password, and the the login password is never sent to the server. They use SRP for authentication. This involves some math based on modular exponentiation that derives a shared key initially based on the password, which the user never sends to the server directly.

I have to handwave about that math because I don’t really understand it. If it’s been a while since you studied crypto, number theory, discrete math, all of that, the protocol may look a little complicated. I can tell that it looks like some public key stuff, but that’s about it. It’s apparently proven to be equivalent to Diffie-Hellman for some purposes, and I will accept that.

Cryptographers say that this protocol works in theory. I don’t doubt it. Proton has audit reports from a third party who believes that they have implemented the crypto correctly. It’s cool that this should all work as described according to these experts. I really like that you don’t have to send your password to Proton and can still use it to derive encryption keys locally. One thing I don’t totally love is that this protocol is somewhat complicated. Cryptography always is, there’s no way that a random person off the street with no training in it will ever have any idea what we’re talking about here. But even with some basic background, I struggle with this one. That doesn’t mean it’s bad or doesn’t work, but I would love it even more if I could make more sense of it, if it were simpler. Which is not obviously possible! It wasn’t obvious to me in the first place that you could do this at all.

Cryptpad

Cryptpad is cool in that it’s basically real-time collaborative Google Docs or Drive, shared between users, but the server has no idea what the content of any document, arbitrary file, or even user account is. It’s also pretty complicated, which again is going to be somewhat unavoidable because what it’s trying to do is complicated. They have taken a very different approach to authentication. The username and password are hashed by the client to generate a random value that identifies an encrypted blob on the server, and an encryption key for that blob. The blob contains the key for the rest of the user’s files, among other things. So basically, “logging in” means knowing the ID of your blob and being able to decrypt it. My cursory reading is that anyone could request your blob if an attacker somehow knew which random ID was yours (maybe they compromised the server and saw a connection from your IP, or compelled the operator to log this), but without your password they couldn’t decrypt it, so there’s not much harm in handing it out to anyone who asks.

All of this is pretty transparent to the user. A lot is happening in the browser. It looks like you’re logging in to a normal service, but really the client is requesting a random blob ID based on your username and password, which won’t exist (or won’t decrypt correctly) if you provide an invalid combination. Once you appear to have a logged in session, what you have is a key in the browser’s memory that can decrypt the various file and directory blob IDs that the client requests from the server. Brute forcing your username and password combination would provide both the account blob ID and the decryption key for it, which Cryptpad intends to prevent by using scrypt for the password hashing. If someone ever figured out how to efficiently crack scrypt with special hardware or cryptanalysis or whatever, this might not work as well, but it’s intended to be resistant to hardware at least.

There’s more to Cryptpad than just this login process, because there are team sharing features, signatures for document edits, and all kinds of stuff. Usage can get a little confusing when you want to change shared access. The developers maintain a list of known weaknesses that could be improved over time. But it works pretty well, and I can at least understand some of the conceptual model of how you can do all of this based only on a single password that is never sent to the server. This system is probably not going to be replicated exactly anywhere else, though. It’s very specific to how Cryptpad stores documents and files.

1Password

1Password used to just encrypt your password database with a key derived from your main password, and let you sync that file via Dropbox, or manually, or directly between the phone and desktop over local wifi. Now they have moved to a subscription model that requires using an account with their cloud service. You still only need one password for the online account and the database, but they try to hide that password from themselves. And they have also added an extra key behind the scenes, so there’s actually more than just the password, but they have tried to make it pretty usable.

They use SRP for authentication like Proton. They raise the concern that obtaining the server’s data that it uses to verify the client during the SRP handshake could lead to the password being cracked. AFAIK this is not something that Proton tries to defend against in their usage of SRP. Again, I’m not a cryptographer, but my reading of SRP is that in order to learn the password from the server’s verifier, you would not only have to brute force the hash of the password but also break the discrete log of the verifier; but I was only ever shaky on this type of math to begin with, so maybe I’m wrong. In any case, 1Password adds another step to their encryption to try to defend against the case where an attacker is able to brute force the password from the server’s SRP verifier.

In addition to the password, you need another random key to decrypt the (wrapped key for the) password database. This is generated randomly on your device when you first set it up. This is one of those recovery key strings you have to print out, in case your device is lost or damaged. When adding a new device, you have to scan a QR code that contains this key. So basically there is a little sleight of hand here: you do need to manage a separate key for the encryption, but they try to make it easier for you by passing it around between your devices with a QR code, and storing it locally so you don’t have to enter it very often. The entire system does not in fact reduce down to the single password, there is an element of key management. You do need to save the piece of paper or else you’re screwed if all your devices are lost. This is still fairly usable, I think, because this amount of key management doesn’t rise anywhere near the level of PGP. You usually would have multiple devices with copies of the key. It’s interesting that they went to this length, because SRP on its own seems like it should be a pretty good way to hide the password from the server. But I’m not mad they added another offline key; better safe than sorry, if you can actually make it work for normal users. It’s a bit like 2FA: something you know (the password), and something you have (the device, or piece of paper).

1Password has all kinds of other stuff going on, like sharing between users and even to non-members with a URL, and a web interface, which I have to say introduces an awful lot of complexity that they might have screwed up when rolling all of this out in one big new version. But today I’m really only interested in the intuition behind the authentication and key management for the simplest version.

One final note: from the password and the random secret, they derive 2 keys: one used to decrypt the database key, and one used for authentication with SRP (the x in g^x % N). To keep the results distinct from each other, they use a different salt for each. This is one approach to key stretching, and there are others. Key stretching is the main concept that I was not thinking about before I understood how you could use one password in multiple ways. More on this with Firefox later.

iCloud (and Google?!)

Most people don’t encrypt their iCloud storage, for photos, notes, device backup, etc. You have to enable it. But once you do, the remote data is encrypted with a key protected by your device’s passcode. I say “protected by”, not “encrypted with”, because it’s a little surprising how they do this. Your passcode is likely not very long, making it subject to brute force. It couldn’t derive a good key-wrapping key on its own. And not only do you have to worry about someone brute forcing the key for your files, you also have to hand over the wrapped key to Apple so they can synchronize it between your devices. Seems sketchy, right?

What they do with the key is interesting. They store it in a hardware security module in their data centers. The HSM will only spit it back out if provided with the device passcode. Crucially, it will prevent more than 10 wrong guesses, the way the security modules in the A* chips in iPhones do. And we know those work pretty well. So, you have to trust Apple’s hardware a little bit here, but they have gone to some lengths to prevent a brute force of your weak password.

As I was writing this, I learned that Google actually does almost the same thing if you opt-in on Android. This didn’t seem to get a lot of press when they released it, at least not that I saw. They use their own custom HSMs in their data centers to control access to your key based on the passcode, with limited guesses enforced in hardware. Sound pretty similar.

You’ll be forgiven if you don’t trust either Apple or Google, but the approach is interesting. Manual key management is hard, so store the keys for the user, but try to tie your own hands as much as possible using hardware guarantees.

Firefox Sync

Firefox Sync is an end-to-end encrypted sync method that shares your bookmarks, history, saved passwords, etc across devices. I’ve never used a feature like this across any browser because historically this would have meant sharing all that info with the cloud provider, who stores it in their data center where a lot of other people (attackers, police) might be able to get access to it. By default, Google stores Chrome sync data unencrypted along with a lot of other stuff like your search history, which they use for targeting ads. But Mozilla seems to have done a reasonable job adding encryption so only your own devices can access it. And crucially, like all these systems, they have made it usable so that you only have to remember or store one password. Assuming the code works correctly as designed, you can log in to your Mozilla account with the same password used for encryption without Mozilla directly learning that password.

After reviewing all these systems again, the method used by Firefox is less surprising. It uses a basic concept that is common to some of these, key stretching. And they have done it in a fairly simple way, without basing their authentication system on SRP. Login works more or less like any online system, you send them your secret over TLS, they hash this on the server, and compare it with the store value in their database. But thanks to some straightforward manipulation by the client, you don’t send the actual password, you send a hash of it that can’t easily be used to derive the encryption key. A single operation of hashing your password locally and using the result for authentication and encryption would be vulnerable to this, but they apply cryptography intelligently to avoid that. Here’s the overview from Mozilla’s blog:

HKDF is a standardized application of HMAC, basically a well-known procedure to follow to use a hash function to derive a key from several pieces of data.

The steps are as follows:

  1. Derive a high-entropy value from the password using PBKDF2
  2. Use HKDF to hash the value from PBDKF2 along with static value #1 to produce the encryption key
  3. Use HKDF to hash the value from PBKDF2 along with static value #2 to produce the authentication token
  4. Send the token to the server to authenticate
  5. Get back the encrypted user data, decrypt it with the key from (2)

What I think is interesting here is that it doesn’t seem like the server should be able to do very well at deriving the encryption key from the authentication token. Both the token and the key are derived from the high-entropy PBKDF2 output, and thanks to different salts in steps (2) and (3) end up totally uncorrelated with each other (assuming a good hash function in the HKDF step). The salts don’t need to be secret, just different from each other. They could be unique per user, or baked in the code for all users, and the encryption key and token will still have nothing to do with each other. This is the key stretching step. One password, multiple keys. It’s not really an exotic concept, probably pretty vanilla to people who do more applied cryptography than I do.

The biggest weakness is that the authentication token sent to the server is more or less password-equivalent, assuming a well-resourced attacker. The input to HKDF is high entropy, but that value is derived from something that might not be. The two keys do ultimately have something to do with each other. If you know the password, you can derive the token to authenticate, and also decrypt the stored data the server has. The token is crackable for a weak password. SRP adds the difficulty of discrete log, and we don’t have that here. The service operator learns the raw token every time you log in, so they could save it if they wanted to or were forced to. Mozilla hashes the token with scrypt to save it in their authentication database, which is intended to make offline brute force harder, but you still have to send the raw token to them frequently.

If your password is subject to brute force because it’s short or reused on another site, this would compromise your stored data. Mozilla acknowledges this. If you are worried about privacy and encryption, you need to be using strong/unique passwords anyway. Some people won’t, but it’s not easy to protect them from themselves in that case. Most things based on passwords need to use good ones if you can’t rely on special hardware like Apple or Google do. I think this is a reasonable tradeoff for a use case like this. If any government comes knocking, Mozilla has given themselves enough room in this design to say “we really don’t know the password, and if it’s a good one, we really can’t decrypt the data for you even if you make us save the token”.

I like that I can understand the basic mechanism here. If I were to implement something like this, I still might introduce all kinds of bugs in practice that would leak bits of information, or expose the entire cleartext data on the client somehow. Bugs in encryption often involve something far from the cryptography itself. But still, the outline of how it works is something that I would be comfortable proposing that people use, especially if they weren’t going to use encryption at all before. It looks much easier to implement correctly than SRP. The design seems generally applicable to any cloud service where you want to store encrypted data on behalf of the user and don’t need to use it for the app’s functionality. Cloud backup, for example, seems like it should use this. If the user has to log in anyway, they know their password, so why not encrypt the data for them and hide it from yourself? SRP might be a better choice if you think you can do it correctly, but Mozilla’s scheme is probably easier to add to more apps by teams that don’t have deep crypto experience. If the choice is no encryption, or something relatively straightforward that’s pretty good, that’s a better choice.

It’s cool that more services are starting to offer better privacy for user data. At this point we have multiple end-to-end encrypted cloud applications for things besides messages. There isn’t really a standard protocol or library they can all adopt for their disparate use cases, so there’s a lot of parallel reinvention going on. They’re all coming up with custom systems. Some of that is unavoidable since they are doing such different tasks. But I think nailing down some best practices or reference architecture could provide future applications with a good starting point as encryption becomes more widespread. Mozilla has distilled the file-storage problem down into a relatively simple architecture that I think can be useful elsewhere. I hope future software engineers find it as understandable as I have and are able to build even more privacy-preserving services like this.