docker ps --format='{{json .}}' is extremely slow due to Size field (Fedora 33)

[buvoli]

• This is a bug report
• This is a feature request
• I searched existing issues before opening this one

Expected behavior

I am running Docker 20.10.1 on Fedora 33. The command docker ps --format='{{json .}}' should run in a reasonable amount of time.

Actual behavior

The command runs extremely slowly (30+ seconds) if there are multiple large containers. I was not experiencing this slowdown in an older version of Docker running on Fedora 30. The cause appears to be the field "Size".

Steps to reproduce the behavior

Here is an example using a larger Jupyter datascience image:

$ docker pull jupyter/datascience-notebook
$ ID=$(docker create jupyter/datascience-notebook)
$ docker start $ID
$ time docker ps --format='{{json .}}'
{"Command":"\"tini -g -- start-no…\"","CreatedAt":"2021-01-04 17:12:34 -0800 PST","ID":"c540990392f9","Image":"jupyter/datascience-notebook","Labels":"maintainer=Jupyter Project \u003cjupyter@googlegroups.com\u003e","LocalVolumes":"0","Mounts":"","Names":"admiring_ptolemy","Networks":"bridge","Ports":"8888/tcp","RunningFor":"About a minute ago","Size":"970B (virtual 4GB)","State":"running","Status":"Up 50 seconds"}

real	0m6.409s
user	0m0.035s
sys	0m0.044s

Here is evidence that the Size field is causing the problems:

$ time docker ps --format='{{json .Size}}'
"970B (virtual 4GB)"

real	0m6.600s
user	0m0.045s
sys	0m0.036s

while other fields run fast. E.g:

$ time docker ps --format='{{json .Command}}'
"\"tini -g -- start-no…\""

real	0m0.063s
user	0m0.033s
sys	0m0.042s

Output of docker version:

Client: Docker Engine - Community
 Version:           20.10.1
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        831ebea
 Built:             Tue Dec 15 04:35:49 2020
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.1
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       f001486
  Built:            Tue Dec 15 04:33:13 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.0-docker)

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 156
 Server Version: 20.10.1
 Storage Driver: btrfs
  Build Version: Btrfs v5.9 
  Library Version: 102
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.9.16-200.fc33.x86_64
 Operating System: Fedora 33 (Workstation Edition)
 OSType: linux
 Architecture: x86_64
 CPUs: 56
 Total Memory: 125.8GiB
 Name: host
 ID: PFSA:YXI2:G2EW:OHLI:5ET5:V5OJ:F7HJ:D46O:GIR3:RPD2:J7VR:Y3QL
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: user
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.)

[thaJeztah]

Hmm.. good point; perhaps we should make the JSON output also use the same default, and only fill in the "size" if the --size flag is set;

-s, --size            Display total file sizes

ISTR, it looks at the template to automatically decide wether or not it should fetch the size; in case of {{ json . }} it therefore determines that the Size field is needed, so it requests that data 🤔

[thaJeztah]

@cpuguy83 @tiborvass @silvin-lubecki WDYT?

If we change this, at least it should be mentioned clearly in the release notes, as users may depend on the existing behavior

[rpigab]

@thaJeztah I'm experiencing the same problem.

If you don't want to break existing behaviour, an option to exclude computing certain fields could be added, but IMHO it'd be fine either way.

[laurazard]

I made docker/cli#4067, which makes it so that explicitly setting --size=false overrides the automatic size request detection from the template on the CLI.

We should look at ways of improving the perfomance of the the size calculation, but for now this provides a workaround for this issue, as it's now possible to run docker ps --format='{{json .}}' --size=false which should not incur the performance penalty associated with the size calculation.

[bwateratmsft]

Is passing that --size=false flag to older versions of the Docker CLI going to cause problems?

[thaJeztah]

Don't think that would be an issue; older versions of the CLI would ignore that case, and continue using the "automatic detection" (in case of {{ json .}} that would mean; still go the "slow path" and request the Size)