How can I re-queue a URL?

[matecsaj]

Crawlee optimizes requests by suppressing redundant URLs and giving up on a URL after reaching a configurable retry limit. This is great, but I encountered an edge case.

Situation: You're extracting data from a page and realize that it wasn’t downloaded properly.

Ideal Goal: Re-queue the URL while decrementing its retry counter.

Alternative: Add the URL back to the queue with a fresh retry counter.

How can I achieve either of these?

[B4nan]

You can enqueue it again with a different unique_key, that's what we use for deduplication (and it defaults to the URL itself).

[vdusek]

Ideal Goal: Re-queue the URL while decrementing its retry counter.

I am afraid this is not possible.

Add the URL back to the queue with a fresh retry counter.

You can use the always_enqueue=True:

request = Request.from_url('https://crawlee.dev/', always_enqueue=True)

which will generate "unique" unique_key under the hood. And add the request to the request queue.