Allow for shared implementations

[UnlimitedCookies]

Due to the high similarity of both the solver and generator it seems reasonable to create a shared directory at the generator and solver directory level. Due to the Dockerfile being inside the generator and solver directory, it does not allow for COPYing the shared directory into the newly built container (resulting in a "COPY failed: forbidden path outside the build context: ../shared ()").

I'm currently unsure what might be a good solution.

[ImogenBits]

What you're asking for cannot be done directly. Due to the way that Docker works it requires every file used in the build to be contained in a single top level folder called the build context. When using Docker directly you could set the build context to be the entire project folder containing generator, solver, and shared, but that cannot be done within Algobattle and also cannot implemented due to the way it is implemented. You also cannot use symlinks to get the folder structure you want to work.

However, there is a way to work with your use case, even though it has some downsides. Let's say you start with a folder structure like this:

Pairsum
 ├─ generator/
 │  ├─ unique_data.json
 │  └─ Dockerfile
 ├─ results/
 ├─ solver/
 │  ├─ unique_data.json
 │  └─ Dockerfile
 ├─ shared/
 │  └─ shared_data.json
 ├─ .gitignore
 ├─ algobattle.toml
 ├─ description.md
 └─ problem.py

What we can do is just move everything inside all of generator, solver, and shared into a single folder, giving everything a unique name and possibly moving things that aren't the dockerfiles into subdirectories:

Pairsum
 ├─ everything/
 │  ├─ generator.dockerfile
 │  ├─ generator/
 │  │  └─ unique_data.json
 │  ├─ solver.dockerfile
 │  ├─ solver/
 │  │  └─ unique_data.json
 │  └─ shared_data.json
 ├─ results/
 ├─ .gitignore
 ├─ algobattle.toml
 ├─ description.md
 └─ problem.py

And then edit the algobattle.toml to reflect the new paths of our dockerfiles (in particular, the path to the files itself, not just their parent folder):

# other stuff above this
[teams."Red Pandas"]
generator = "everything/generator.dockerfile"
solver = "everything/solver.dockerfile"

Of course, you will then also want to adapt your dockerfiles to look for everything in the new paths.

There are two important things to note here. The first is that the dockerfiles must be in the folder that contains everything used in the builds and it must have the .dockerfile extension. That is, they cannot be placed in the everything/generator or everything/solver subdirectories. And while they can be named something like my very fancy name.dockerfile, it must always carry that extension or it will not be recognised by Algobattle.

The second is that when you package an Algobattle project and upload it to the server, it will always bundle the entire build context together. It isn't possible to determine that some folders might not be needed for the generator or solver builds, so you would end up duplicating this data. If these are only a couple of light code files that isn't an actual issue, but if you accidentally leave things like build artifacts or precompiled binaries (which is bad practice in the first place) in the folders before packaging, you will create rather large files unnecessarily.

[UnlimitedCookies]

Thank you for the elaborate suggestion for how to make a shared lib work.
Btw it should be documented that only Dockerfile and some_name.dockerfile syntax is allowed.

The problem is that algobattle package programs then ends up emitting an empty zip file, because the program_root is the unix path of that Dockerfile (TeamInfo: generator=PosixPath('../generator.dockerfile') solver=PosixPath('../solver.dockerfile')).
The responsible snippet is here

algobattle/algobattle/cli.py

Lines 636 to 643 in 75c9302

	def _package_program(name: str, info: TeamInfo, role: Role) -> None:
	with console.status(f"Packaging {name}'s {role}"), ZipFile(out / f"{name} {role}.prog", "w") as zipfile:
	program_root: Path = getattr(info, role)
	for file in program_root.rglob("*"):
	if file.is_dir():
	continue
	zipfile.write(file, file.relative_to(program_root))
	console.print(f"[success]Packaged {name}'s {role}")

[UnlimitedCookies]

Technically, the information what files to include lies in the Dockerfile, but I don't know if it's a good idea to parse that.
Before I dove into the source I expected the prog-file to just be a docker image, but that does not seem to be the case.

[ImogenBits]

The problem is that algobattle package programs then ends up emitting an empty zip file, because the program_root is the unix path of that Dockerfile (TeamInfo: generator=PosixPath('../generator.dockerfile') solver=PosixPath('../solver.dockerfile')).

That's a bug we should fix, thanks for the report.

Technically, the information what files to include lies in the Dockerfile, but I don't know if it's a good idea to parse that.
Before I dove into the source I expected the prog-file to just be a docker image, but that does not seem to be the case.

Yeah, it would make most sense if they were docker images, but that comes with some problems. The docker image doesn't just contain the code and any compilation results, but also the whole OS and language compiler/runtime. If you just grab the images as-is then you'll get files that typically are over a gigabyte large, compared to a few kilobytes of just the code. In a perfect world we'd be able to create a .prog file that contains exactly those parts of the docker image that depend on the code and save only those.

That is somewhat doable, but fairly complex to actually implement. There are a couple different options, but all either rely on very intricated manipulation of internal docker structures, or exposing fairly esoteric docker specific concepts to the students in Algobattle courses. The most basic would be to build the docker image locally and then save it to a .tar file. This will contain the individual layers and some metadata needed to piece them together. In particular, the first layer will be the one corresponding to the FROM ... statement. The .tar file should contain enough metadata to find the exact image in the docker hub registry, so we can remove the layer data from the file. If you're clever enough and also parse the dockerfile you can probably also manage to identify a decent chunk of layers that just install something from the web, like RUN apt install -y clang-15 cmake ninja-build, but that is going to be fairly difficult. You'd then probably also need to save some of your own metadata so that Algobattle can then grab the removed layers from the web when building and restore the entire image properly.

While that is theoretically possible, it's not super feasible and frankly does not have a lot of relevant upsides to the current solution of just zipping up the whole folder. The only real advantage would be that you can't have weird issues where your build environment looks different on the server for some reason, but that isn't a big concern anyways. This also wouldn't help with your goal of having stuff outside the build context included in the builds.