Running inside Docker doesn't take care of persistence

[francescoaracu]

I'm implementing a crawler using BS (you can find the repo here).

Even if I set purge_on_start to False inside a Configuration object, after pausing the crawler via Ctrl-C in the attached Docker container's shell, the crawler doesn't seem to take care of the persistence status set in the RequestList instance.

The volume mounted in my docker-compose.yml is then overwritten directly when running the container again with docker start.

[Mantisus]

Hi, @francescoaracu :)

Could you try setting up the configuration via the service_locator and let us know if that solves the issue?

from crawlee import service_locator

config = service_locator.get_configuration()
config.available_memory_ratio = 0.7
config.purge_on_start = False

[francescoaracu]

It seems to be working now; it starts first with the "first-level" requests from the queue and then it continues with the other "zero-level" links from the CSV.