-
Notifications
You must be signed in to change notification settings - Fork 612
Closed
Labels
bugSomething isn't working.Something isn't working.
Description
Hi, it seems like user_data will drop dict keys that have null values under them. See the example
from crawlee.router import Router
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
from crawlee import Request
import asyncio
async def main():
# define links
target_url = "https://example.com/" # target site
# define router
router = Router[BeautifulSoupCrawlingContext]()
@router.handler("MAIN")
async def main_handler(context : BeautifulSoupCrawlingContext) -> None:
loaded_user_data = dict(context.request.user_data)
print("loaded user_data:", loaded_user_data)
return
# then try to do a request with crawlee with the same proxy
crawler = BeautifulSoupCrawler(request_handler=router)
# run it
await crawler.run( [Request.from_url(target_url, label='MAIN', user_data={"a":1, "b":"2", "c":None, "d":False})] )
if __name__ == '__main__':
asyncio.run(main())
"c" key got dropped as I load it into a dictionary in the handler.
Metadata
Metadata
Assignees
Labels
bugSomething isn't working.Something isn't working.