I wrote a local file encryption tool, around the same time Filippo was doing `age`, and used the AD on Chapoly to authenticate the chunk offset into the file. (The only thing interesting my tool did was that it could pull keys from AWS KMS).
So one use for AD is to authenticate headers; another is contextual binding.
If it helps (because 'stavros asked across the thread why bother having AD at all rather than just including it in the ciphertext), authenticated data can include data that doesn't even appear in the message, but rather is derived from the context at the time the message is encrypted and decrypted. A message only meant to be decrypted on a particular host (or whatever), for instance, could include the host in its AD, but never record that in the actual bits of the message.
It's important to use a carefully designed AEAD mode rather than assembling it yourself out of parts. If you try to combine a block cipher mode and message authenticator together, you might screw it up in a really funny way: https://soatok.blog/2021/07/30/canonicalization-attacks-agai...
Sanketh's talk at Real World Crypto 2024 about Next-Generation AEADs is also worth a watch for anyone that, for whatever weird reason, feels at all motivated to invent a new wheel here: https://www.youtube.com/watch?v=7GBzKytVjH4
If you squint at the example usage in the tests, it's basically the API that the blogpost describes.
https://github.com/peterldowns/symcrypt/blob/main/symcrypt_t...
As an aside, I'm always curious to understand why the encryption people say "never roll your own crypto" but then also ship confusing APIs without clear usage examples. For instance, check out the golang chacha20poly1305 docs:
Your `symcrypt` interface lands in a pretty weird place? AEADs in Go export "Seal" and "Unseal" --- with deliberately different names than crypto/cipher/Block's "Encrypt" and "Decrypt", because they're doing something different. The "Owner" thing in your package is kind of odd too.
You're exposing an interface over Go's AEAD primitives, but not letting users actually provide authenticated data. I don't much care except the whole point of this post is why that matters.
For me, personally, I'm going to side with tptacek - he has a track record that I have seen over at least a decade if not two.
I don't know the other bloke but this is a bit of a worry: "I'm not a cryptographer".
> Your `symcrypt` interface lands in a pretty weird place? AEADs in Go export "Seal" and "Unseal" --- with deliberately different names than crypto/cipher/Block's "Encrypt" and "Decrypt", because they're doing something different.
What should I use? I'd be extremely happy to do the Right Thing. I linked symcrypt and posted here because I am hoping someone can point me to it.
> You're exposing an interface over Go's AEAD primitives, but not letting users actually provide authenticated data.
I really don't understand what you mean by "not letting users actually provide authenticated data". Here, in this test, I show how if you encrypt some secret for one user (the associated data is the Owner), you can only decrypt it if you provide the same associated data (the same Owner). https://github.com/peterldowns/symcrypt/blob/c220f7767fa6c1a...
What you should do is just take the examples from cipher#AEAD, but where they do:
block, err := aes.NewCipher(key)
if err != nil {
panic(err.Error())
}
aesgcm, err := cipher.NewGCM(block)
if err != nil {
panic(err.Error())
}
chapoly, err := chacha20poly1305.NewX(key)
if err != nil {
panic(err.Error())
}
Your "Owner" looks like what cryptographers would call a "domain separation constant". Domain separation is good! It's another application of authenticated data, too. But not the only one.
The Go standard library's AEAD "Seal" and "Unseal" is a better interface than what you've got now.
Another source of inspiration (and something I use in production) is the Tink family of cryptographic libraries by Google [1]. Their Go implementation [2] is not without its warts, but it's very difficult to run into any of those security bugs that exist around cryptography. Where the Go documentation lacks, there are some examples in the developer docs that help fill some of the gaps [3] [4].
The documentation isn't 100% complete, but I find it more discoverable than the standard library because while the standard library requires you to read both `crypto/cipher` and `crypto/aes` or `golang.org/x/crypto/chacha20poly1305` depending on what kind of cipher you want, Tink organizes it by use cases [5] and generally groups together all the things you need to do cryptographic operations under the use-case-named interfaces in the `tink` package [6], with the corresponding key generation templates located under the top-level packages of the same name [7].
[1]: https://developers.google.com/tink
[2]: https://github.com/tink-crypto/tink-go/
[3]: https://developers.google.com/tink/key-management-overview#g...
[4]: https://developers.google.com/tink/encrypt-data#go
[5]: https://developers.google.com/tink/choose-primitive
[6]: https://pkg.go.dev/github.com/tink-crypto/tink-go/v2/tink#AE...
[7]: https://pkg.go.dev/github.com/tink-crypto/tink-go/v2@v2.4.0/...
https://news.ycombinator.com/item?id=43827342
Honestly, the classic "message routing" example most things give for AEAD is not very useful. Context binding is a much better primer for intuition.
Basically, I'm not sure why `encrypt(key, nonce, (data, associated data))` (ie adding the AD to your ciphertext, with the encryption framework being unaware of it) is that different from `encrypt(key, nonce, data, associated data)` (ie the AD being a first-class citizen).
EDIT: I saw your other message, and this makes it click for me:
> authenticated data can include data that doesn't even appear in the message, but rather is derived from the context at the time the message is encrypted and decrypted
So the AD can be an additional envelope-level thing at encryption/decryption time, that helps a lot, thanks!
Instead, just take the chunked large-file encryption use case I gave in that comment. The chunk offset isn't recorded anywhere in the ciphertext. It's derived contextually while you decrypt the file. The AD ensures that the decryption will explode if you try to cut and paste chunks of the file into different positions.
You can find an implementation of AES-CTR-HMAC (at a high level where AES-CTR and HMAC are both given) here: https://github.com/tink-crypto/tink-go/blob/main/aead/aesctr...
Ah, that's the key bit of perspective. Just talking about "context" is so abstract. That's a case where you don't even need to transmit the AD, right? Do you ever have cases where the AD is a mix of transmitted and locally/"contextually" derived data?
You could water down the example a bit to make it work:
1. Assume there's some other authentication mechanism for client-server communication, e.g. TLS.
2. The client sends the user ID unencrypted (within TLS) so the server can route, but encrypts the message contents so the server can't read it.
3. The final recipient can validate the message and the user ID.
This saves the client from having to send the user ID twice, once in the ciphertext and once in the clear.
But another more interesting use case is when you don't even send the associated data: https://news.ycombinator.com/item?id=43827342
Even if I gain access to the database, if the keys are managed securely, I can't read another user's data (or even really my own). I have to go through the authorization logic of the application that will decrypt it on my behalf.
However, if I can create a row in the database with my ID and another user's data, I can then convince the server I am authorized to view that cell, and it will happily decrypt it on my behalf, assuming something like AES-CTR or some other stream cipher without authentication.
Authenticated encryption like AES-CTR-HMAC solves that problem, because now the application will see that I am authorized to view that cell (because it sees the user ID matches mine) and it will decrypt it for me (using that user ID as the associated data), but the decryption will fail because the associated data does not match, leaving me unable to exfiltrate the data that I convinced the server belonged to me, and probably setting off some kind of alarm because that sort of decryption should never fail unless things have been tampered with.
I'm not overly fond of the example and I find it confusing as well. I think the example may be a bit confusing because the term "authentication" is overloaded between application-level authentication and cryptographic authentication, i.e., "if the chat protocol authenticates the user ID" sounds like it is talking about the user logging into the server securely. The user is authenticated by having a secret negotiated with the server. In the next bullet, they talk about "authenticating" the associated data, referring to it in the cryptographic context, but they don't indicate why that would be a problem because in their example, the malicious actor still doesn't have the key. The article handwaves it as "the attacker might be creative."
If they had the key, but not the associated data, you'd still be in a relatively bad situation, because the associated data is not secret. It doesn't serve as a second key because it is not high enough entropy and is ideally zero entropy conditional on already having all information from the originating context.
That guarantee is not provided with unauthenticated stream ciphers. For example, some stream ciphers work by essentially using a deterministic but unpredictable PRNG seeded with the key and IV to generate a bitstream, and then XOR the plaintext with that bitstream to generate the ciphertext.
With such a stream cipher, If Eve knows that the data format is, e.g., the an 8-byte unsigned integer user ID followed by the rest of the payload, Eve can take the first 8 bytes of the ciphertext and XOR it with Bob's user ID (public information) and her own user ID to corrupt the message in such a way that the ID in the resulting cleartext would contain her user ID instead of Bob's, and thus pass the validation that it seems like you are proposing.
Let C[] be the cipher text, K[] be the key stream, B be Bob's ID, and E be Eve's ID:
C[:8] = B ^ K[:8]
C[:8] ^ B = K[:8]
C' = C[:8] ^ B ^ E ++ C[8:]
C' would decrypt, validate as "belonging" to Eve, and contain Bob's data.
If you are using an AEAD cipher mode, then you always have AD, but sometimes that AD might be the empty string. In that case, the advantage to using contextual AD as opposed to using the empty string as AD and then doing additional verification on the decrypted object is that it prevents some kinds of timing attacks, because cryptographic libraries will often implement AEAD constructions to fail in constant time, where as your scheme will take longer if post-decryption validation of contextual data encoded in the plaintext fails compared to if decryption fails.
OK fair.
> and I would consider whatever "secret key" you pass into HMAC along with the ciphertext to be the "associated data"
The secret key isn't associated data. You take your base HKDF key and expand new crypto for an authenticated cipher from the offset as info (+ maybe other parameters like file name). That key is then used to decrypt. If you squint I guess you could call that AD but it's functionally a very different role.
> because cryptographic libraries will often implement AEAD constructions to fail in constant time, where as your scheme will take longer if post-decryption validation of contextual data encoded in the plaintext fails compared to if decryption fails.
I think you've misunderstood what I said. As I repeat above, the AEAD key is derived from the offset. There's no post-decryption validation of contextual data because the plaintext is empty. HKDF derivation is constant time and authenticated decryption is constant time. Once decrypted you have a valid block at that location. There's nothing extra left to validate (or perhaps the decrypted contents, but that's irrelevant for cryptographic purposes).
My broader point is that I have yet to encounter a use-case for a non-empty AD string.
As you said, same effect: the scheme you have described is not an alternative to AEAD. It is an example of AEAD. You're still using the offset as associated data, you just happen to be composing your AEAD scheme out of another AEAD scheme into which you pass an empty string as associated data.
Other than differences in the limit to the number of messages you can encrypt before nonce exhaustion or the number of bits of secrecy or authentication strength provided, the external interface and use case of your system perfectly matches that of AES-GCM or other popular AEAD constructions.
Ahh, but that’s not a trivial part of the design and why this is strictly better than using a single AES-GCM key with AD. And also it’s more generic across whatever type of key you choose to derive.
Somehow I assumed that the server was able to authenticate the receiver id, but as you correctly point out, that would require knowing the encryption key. I'll have to think about a fix for the example.
If Alice saves some data to her account, but Eve manages to access the database, Eve can change the database state to convince the application to retrieve Alice's data for her (by cloning it into a row with her own user ID). However, when the application attempts to decrypt that data, it will fail because of the AEAD. This ensures that both the database and some service with access to the encryption key (or the encryption key itself) would have to be compromised in order for Eve to exfiltrate her illicit copy of Alice's data.
I finally updated the example to a new one, though it's still message-based (it fits the rest of the article better). If I had come across your example earlier, I might have stayed away from a message-based formulation of the problem at all... Better luck next time I guess :)
An AEAD can also be made de novo. Such as AEGIS[1], which performs encryption and authentication in one pass (much like the sponges, but much more performant).
A naive implementation of the AEAD feature list could trivially allow you to guess the AD for a ciphertext if the AD validation is checked too early in the process.
We can imagine, e.g. in the context of e-mail, if the DKIM header signature were combined a PGP-encrypted body as one operation. I'm ducking under the table now, though.
just the right length and pacing to get me to the end and the point across