Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -199,7 +199,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [0.9.5] - 2020-07-14

### Fixed
- Fixed a bug which caused zombie `docker events ...` processes to accumulate over time.
- Fixed a bug which caused zombie `docker system events ...` processes to accumulate over time.

## [0.9.4] - 2020-02-12

Expand Down Expand Up @@ -257,7 +257,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Docuum now automatically restarts itself when an error occurs.

### Fixed
- Docuum now cleans up the `docker events` child process when an error occurs.
- Docuum now cleans up the `docker system events` child process when an error occurs.

## [0.3.0] - 2020-01-06

Expand Down
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,5 @@ RUN apk add --no-cache docker-cli
COPY --from=build /usr/local/bin/docuum /usr/local/bin/docuum

# Set the entrypoint to Docuum. Note that Docuum is not intended to be run as
# an init process, so be sure to pass `--init` to `docker run`.
# an init process, so be sure to pass `--init` to `docker container run`.
ENTRYPOINT ["/usr/local/bin/docuum"]
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Docuum is used by Netflix (on its production Kubernetes nodes) and Airbnb (on it

## How it works

[Docker doesn't record when an image was last used.](https://github.com/moby/moby/issues/4237) To work around this, Docuum listens for notifications via `docker events` to learn when images are used. It maintains a small piece of state in a local data directory (see [this](https://docs.rs/dirs/3.0.2/dirs/fn.data_local_dir.html) for details about where this directory is on various platforms). That persisted state allows you to freely restart Docuum (or the whole machine) without losing the image usage timestamp data.
[Docker doesn't record when an image was last used.](https://github.com/moby/moby/issues/4237) To work around this, Docuum listens for notifications via `docker system events` to learn when images are used. It maintains a small piece of state in a local data directory (see [this](https://docs.rs/dirs/3.0.2/dirs/fn.data_local_dir.html) for details about where this directory is on various platforms). That persisted state allows you to freely restart Docuum (or the whole machine) without losing the image usage timestamp data.

When Docuum first starts and subsequently whenever a new Docker event comes in, LRU eviction is performed until the total disk usage due to Docker images is below the given threshold. This design has a few advantages over evicting images based on a fixed [time to live](https://en.wikipedia.org/wiki/Time_to_live) (TTL), which is what various other tools in the Docker ecosystem do:

Expand Down Expand Up @@ -129,7 +129,7 @@ You can run that command with `--force` to update an existing installation.
If you prefer not to install Docuum on your system and you're running macOS or Linux (AArch64 or x86-64), you can run it in a container:

```sh
docker run \
docker container run \
--init \
--rm \
--tty \
Expand All @@ -142,7 +142,7 @@ docker run \
If you're on a Windows system configured to run Linux containers, use this command:

```powershell
docker run `
docker container run `
--init `
--rm `
--tty `
Expand Down
16 changes: 8 additions & 8 deletions integration-test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ set -euxo pipefail

# Wait for the Docker daemon to start up.
echo 'Waiting for Docker to start…'
while ! docker ps > /dev/null 2>&1; do
while ! docker container ls > /dev/null 2>&1; do
sleep 1
done

Expand All @@ -32,39 +32,39 @@ wait_for_docuum

# This image uses ~5.5 MB.
echo "Using an image we don't want to delete…"
docker run --rm alpine@sha256:f27cad9117495d32d067133afff942cb2dc745dfe9163e949f6bfe8a6a245339 \
docker container run --rm alpine@sha256:f27cad9117495d32d067133afff942cb2dc745dfe9163e949f6bfe8a6a245339 \
true
docker tag alpine@sha256:f27cad9117495d32d067133afff942cb2dc745dfe9163e949f6bfe8a6a245339 \
docker image tag alpine@sha256:f27cad9117495d32d067133afff942cb2dc745dfe9163e949f6bfe8a6a245339 \
alpine:keep

wait_for_docuum

# This image also uses ~5.5 MB.
echo 'Using another image…'
docker run --rm alpine@sha256:2039be0c5ec6ce8566809626a252c930216a92109c043f282504accb5ee3c0c6 true
docker container run --rm alpine@sha256:2039be0c5ec6ce8566809626a252c930216a92109c043f282504accb5ee3c0c6 true

wait_for_docuum

# This image also uses ~5.5 MB. For some reason, this pushes us over the 20 MB
# threshold, even though we've only downloaded ~5.5 MB * 3 = ~16.5 MB.
echo 'Using another image…'
docker run --rm alpine@sha256:4d889c14e7d5a73929ab00be2ef8ff22437e7cbc545931e52554a7b00e123d8b true
docker container run --rm alpine@sha256:4d889c14e7d5a73929ab00be2ef8ff22437e7cbc545931e52554a7b00e123d8b true

wait_for_docuum

# Assert that the image protected by the `--keep` flag is still present.
echo 'Checking that the protected image is still present…'
docker inspect alpine@sha256:f27cad9117495d32d067133afff942cb2dc745dfe9163e949f6bfe8a6a245339 > \
docker image inspect alpine@sha256:f27cad9117495d32d067133afff942cb2dc745dfe9163e949f6bfe8a6a245339 > \
/dev/null 2>&1

# Assert that the last image is still present.
echo 'Checking that the last image is still present…'
docker inspect alpine@sha256:4d889c14e7d5a73929ab00be2ef8ff22437e7cbc545931e52554a7b00e123d8b > \
docker image inspect alpine@sha256:4d889c14e7d5a73929ab00be2ef8ff22437e7cbc545931e52554a7b00e123d8b > \
/dev/null 2>&1

# Assert that the first non-protected image was deleted.
echo 'Checking that the first non-protected image was deleted…'
if docker inspect alpine@sha256:2039be0c5ec6ce8566809626a252c930216a92109c043f282504accb5ee3c0c6 \
if docker image inspect alpine@sha256:2039be0c5ec6ce8566809626a252c930216a92109c043f282504accb5ee3c0c6 \
> /dev/null 2>&1
then
echo "The image wasn't deleted."
Expand Down
12 changes: 6 additions & 6 deletions src/run.rs
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ const CONTAINER_STATUSES: [&str; 7] = [
CONTAINER_STATUS_REMOVING,
];

// A Docker event (a line of output from `docker events --format '{{json .}}'`)
// A Docker event (a line of output from `docker system events --format '{{json .}}'`)
#[derive(Deserialize, Serialize, Debug)]
struct Event {
#[serde(rename = "Type")]
Expand Down Expand Up @@ -339,7 +339,7 @@ fn image_ids_in_use() -> io::Result<HashSet<String>> {
fn docker_root_dir() -> io::Result<PathBuf> {
// Query Docker for it.
let output = Command::new("docker")
.args(["info", "--format", "{{.DockerRootDir}}"])
.args(["system", "info", "--format", "{{.DockerRootDir}}"])
.stderr(Stdio::inherit())
.output()?;

Expand Down Expand Up @@ -843,9 +843,9 @@ pub fn run(
state::save(state)?;
*first_run = false;

// Spawn `docker events --format '{{json .}}'`.
// Spawn `docker system events --format '{{json .}}'`.
let mut child = Command::new("docker")
.args(["events", "--format", "{{json .}}"])
.args(["system", "events", "--format", "{{json .}}"])
.stdout(Stdio::piped()) // [tag:stdout]
.spawn()?;

Expand Down Expand Up @@ -935,10 +935,10 @@ pub fn run(
debug!("Going back to sleep\u{2026}");
}

// The `for` loop above will only terminate if something happened to `docker events`.
// The `for` loop above will only terminate if something happened to `docker system events`.
Err(io::Error::other(format!(
"{} terminated.",
"docker events".code_str(),
"docker system events".code_str(),
)))
}

Expand Down
12 changes: 6 additions & 6 deletions toast.yml
Original file line number Diff line number Diff line change
Expand Up @@ -276,22 +276,22 @@ tasks:
# This will be called when the task finishes, regardless of whether it succeeded.
function cleanup {
# Delete the container created below to run the integration tests.
docker rm --force dind
docker container rm --force dind
}
trap cleanup EXIT

# Run the integration test suite [tag:integration_test_step]. We use a Docker-in-Docker
# environment to run the suite because we don't want to inadvertently delete images from the
# host machine.
docker run \
docker container run \
--privileged \
--name dind \
--detach \
docker:dind
docker exec dind apk add bash
docker cp "artifacts/docuum-x86_64-unknown-linux-musl" dind:/
docker cp integration-test.sh dind:/
docker exec dind ./integration-test.sh
docker container exec dind apk add bash
docker container cp "artifacts/docuum-x86_64-unknown-linux-musl" dind:/
docker container cp integration-test.sh dind:/
docker container exec dind ./integration-test.sh

publish:
description: Publish the crate to crates.io.
Expand Down