volumes wont be deleted since update to docker 23

[Emporea]

Since the update to docker 23 unused volumes will not be deleted anymore with docker volume prune nor docker system prune --volumes.

The answer is always Total reclaimed space: 0B.
When I delete the volumes manually I even get these warning when running docker-compose.yml
Error response from daemon: open /var/lib/docker/volumes/docker_volume1/_data: no such file or directory

Whats happening?

[jcmcken]

Also seeing this issue. We had to revert to 20.10 because it was filling up disks with no way to recover and causing outages.

[jcmcken]

Also seeing this issue. We had to revert to 20.10 because it was filling up disks with no way to recover and causing outages.

It seems like it's this change in 23 (mentioned in the 23 changelog). When I run this all=true filter on docker volume prune it works. But docker system prune does not accept this filter, so now seems broken

Not clear why this default needed to change

[cpuguy83]

Yes this sounds like it is related to the mentioned change.
The default change allows us to:

1. Differentiate an anonymous volume from a named one
2. Make docker volume prune safer to execute. Given that named volumes typically are named so they can be easily referenced

It does mean that upgrading causes volumes which were created prior to 23.0.0 to not be considered for pruning except for when specifying --all.
However "anonymous" volumes created after 23.0.0 will be considered for pruning... and of course again --all gives the old behavior.

Also if your client uses the older API version it will also get the old behavior.

[Emporea]

Do you suggest to delete everything and recreate every volume ( obv. after backing up to restore after they have been recreated) to get rid of old obsolete configs?

[cpuguy83]

@Emporea docker volume prune --all should give the old behavior. I understand docker system prune doesn't support this yet (not intentionally).

Do you suggest to delete everything and recreate every volume

No, I don't think that should be necessary, it would likely be simpler to use docker volume prune --filter all=1 in addition to docker system prune until your older volumes are no longer in use.

You can also use DOCKER_API_VERSION=1.40 docker system prune --volumes to get the old behavior.

[Emporea]

root@server ~ [125]# docker volume prune --all
unknown flag: --all
See 'docker volume prune --help'.

root@server ~ [125]# docker --version
Docker version 23.0.0, build e92dd87

[cpuguy83]

Sorry I gave the wrong command in the first line there, docker volume prune --filter all=1

[Emporea]

Thank you. this works and helps me for now.

[jcmcken]

The default change allows us to:

1. Differentiate an anonymous volume from a named one

2. Make `docker volume prune` safer to execute. Given that named volumes typically are named so they can be easily referenced

I guess that's fine reasoning, but the execution of this change was very strange for multiple reasons:

• The CLI help output doesn't explain anything about these distinctions. (Still says "Remove all unused local volumes", nothing about anonymous vs named)
• The documentation does not explain anything about these distinctions. (Same as above). We had to look in a specific change log to find the information.
• As mentioned (and acknowledged by you), this change was not actually propagated to other parts of the CLI.
• The previous default behavior is not a flag or something obvious to re-enable, but something non-obvious (set env vars? set a particular filter?). Seems like when you change something like this, "principal of least surprise" should apply. It seems like it would've been more conservative to at least use deprecation warnings over multiple releases, or not change it at all and add the 'anonymous' distinction as an opt-in flag.
• The behavior simply isn't obvious. I don't care about anonymous volumes vs named volumes, for one. A novice Docker user is certainly not going to care or understand why the prune command isn't deleting their volume. I think trying to save the user from themselves using non-explicit behavior is awfully confusing. I think at the very least, the previous behavior either needs to be explicitly mentioned in the help output or provided as a standalone flag. That way, if a Docker user sees a volume isn't getting cleaned up, they can run -h, and hopefully notice mention of named vs anonymous volumes.

[neersighted]

The CLI help output doesn't explain anything about these distinctions. (Still says "Remove all unused local volumes", nothing about anonymous vs named)

Yeah, the help needs to change to say 'anonymous.'

The documentation does not explain anything about these distinctions. (Same as above). We had to look in a specific change log to find the information.

Same as the help output, PRs welcome.

Seems like when you change something like this, "principal of least surprise" should apply.
I don't care about anonymous volumes vs named volumes, for one.

I'd like to point out you're a tiny minority there -- for the vast majority of users, "Docker deleted my data after I ran system prune -a" has been a sharp edge for years. Most users expect prune to 'clean up garbage,' not 'clean up the things I wanted Docker to keep.'

As mentioned (and acknowledged by you), this change was not actually propagated to other parts of the CLI.

The only part where this possibly needs to propagate is docker system prune -a and we're still not sure what the better behavior is.

That way, if a Docker user sees a volume isn't getting cleaned up, they can run -h, and hopefully notice mention of named vs anonymous volumes.

Agreed, there should be an example of using the all filter in the help text.

---

Please keep in mind that this has been a persistent pain for educators, commercial support, and undercaffinated experts for years. People are here in this thread because they find the behavior change surprising, and yeah, it looks like review missed the docs updates needed (and this is because of the historical incorrect split of client/server logic we are still cleaning up) -- however, please keep in mind that this thread represents the minority of users who find this behavior unexpected or problematic.

We certainly can improve here, and there are a lot of valid points, but the happy path for the majority of users is to stop pruning named volumes by default.

[neersighted]

Also, as an aside: the behavior here changed in the daemon (and is dependent on API version) -- an older CLI against a current daemon will see the old behavior, and the new CLI against an older daemon will also see the old behavior.

So as we look at improving the docs & help output, we need to figure out how to explain that difference coherently.

[thaJeztah]

This issue is not directly related to Docker Desktop for Linux; probably would've been best in the https://github.com/moby/moby issue tracker, but I can't move it there because that's in a different org.

Let me move this to the docker/cli repository, where another thread was started in #4015