Separately encrypt etcd backups

[MichaelEischer]

How to categorize this topic?

/area backup
/kind enhancement

/label medium

What is the topic about?:
etcd backups created for shoots are stored in the same bucket for all shoots of a seed (GEP-0002). In case of a control plane compromise, a shoot would be able to read at least all backups for a seed. The most critical data, namely the secrets, are encrypted, however, it would be preferable to completely prevent exposure to data belonging to other shoots.

A possible approach for in-depth hardening would be to separately encrypt each backup in etcd-backup-restore using a shoot-specific encryption key. Then the compromise of a single control plane would only allow access to other encrypted backups. (Related to gardener/etcd-backup-restore#83 )

Points to consider:

• It should be possible to rotate the encryption key. This either requires a way to reencrypt all backups or to keep the old encryption keys until all old backups using that key have expired (beware of hibernated shoots!).
• The encryption key must properly be transferred on shoot control plane migration

[gardener-prow]

@MichaelEischer: The label(s) `/label medium

cannot be applied. These labels are supported:tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, ipcei/oidc, ipcei/workload-identity, teamsize/small, teamsize/medium, teamsize/large, fast-track, crypto, skip-review. Is this label configured under labels -> additional_labelsorlabels -> restricted_labelsinplugin.yaml`?

Details

In response to this:

How to categorize this topic?

/area backup
/kind enhancement

/label medium

What is the topic about?:
etcd backups created for shoots are stored in the same bucket for all shoots of a seed (GEP-0002). In case of a control plane compromise, a shoot would be able to read at least all backups for a seed. The most critical data, namely the secrets, are encrypted, however, it would be preferable to completely prevent exposure to data belonging to other shoots.

A possible approach for in-depth hardening would be to separately encrypt each backup in etcd-backup-restore using a shoot-specific encryption key. Then the compromise of a single control plane would only allow access to other encrypted backups. (Related to gardener/etcd-backup-restore#83 )

Points to consider:

• It should be possible to rotate the encryption key. This either requires a way to reencrypt all backups or to keep the old encryption keys until all old backups using that key have expired (beware of hibernated shoots!).
• The encryption key must properly be transferred on shoot control plane migration

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[MichaelEischer]

/label teamsize/medium

[Gerrit91]

We started a small prototype implementation for this. Here are some pointers:

• https://github.com/gardener/gardener/compare/master...metal-stack:gardener:etcd-backup-encryption?expand=1
• https://github.com/gardener/etcd-druid/compare/master...Gerrit91:etcd-druid:etcd-backup-encryption?expand=1
• https://github.com/RadaBDimitrova/etcd-backup-restore/tree/add-backup-encryption

TODOs / Ideas:

• Generate new secret with encryption key in the shoot reconcile
  • Persist in shoot state
• Deploy etcd-backup-restore with this encryption key from etcd-druid
  • Add configuration file similar to kube-apiserver encryption at rest EncryptionConfig
  • Add flag and volume mount for etcd-backup-restore
• Implement encryption package in etcd-backup-restore (for now only AESGCM)
• Support key rotation
  • Store an encrypted keyring file that contains the encryption keys along with the backups
    • This contains the keys that were used at certain points in time and allows restoring backups at any point in time
  • This keyring file is always encrypted with the latest key and gets updated on startup
• Wire up keyring and encryption package into the etcd-backup-restore backup / restore flows
• Add validations, probably also add a field in the shoot spec to set the encryption provider type
• More extensive testing
• Hide behind feature gate
• Support more provider types
• Support changing the provider type