Skip to content

user_data drops null values that are passed in #1706

@DoctorEvil92

Description

@DoctorEvil92

Hi, it seems like user_data will drop dict keys that have null values under them. See the example

from crawlee.router import Router
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
from crawlee import Request
import asyncio


async def main():
    # define links
    target_url = "https://example.com/" # target site

    # define router
    router = Router[BeautifulSoupCrawlingContext]()
    @router.handler("MAIN")
    async def main_handler(context : BeautifulSoupCrawlingContext) -> None:
        loaded_user_data = dict(context.request.user_data)
        print("loaded user_data:", loaded_user_data)
        return
    
    # then try to do a request with crawlee with the same proxy
    crawler = BeautifulSoupCrawler(request_handler=router)

    # run it
    await crawler.run( [Request.from_url(target_url, label='MAIN', user_data={"a":1, "b":"2", "c":None, "d":False})] )

    
    

if __name__ == '__main__':
    asyncio.run(main())

Image

"c" key got dropped as I load it into a dictionary in the handler.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions