fix(playwright): filter unsupported context options in persistent browser by sushant-mutnale · Pull Request #1796 · apify/crawlee-python

sushant-mutnale · 2026-03-16T03:46:26Z

This PR fixes issue #1784, where PlaywrightCrawler would crash when passing context options (like storage_state) that are unsupported by Playwright's launch_persistent_context method.

Changes:

Implemented dynamic argument filtering in PlaywrightPersistentBrowser.new_context using inspect. signature.
Added a warning log to guide users when options are filtered out, suggesting the use of incognito pages as an alternative.
Added a unit test in

tests/unit/browsers/test_playwright_browser.py
to verify the fix and prevent regressions.
Fixes #1784

…wser This addresses issue apify#1784 by dynamically filtering options passed to launch_persistent_context and providing a warning log for ignored options like storage_state.

Pijukatel

Hello, thanks for the PR. Please see my comments; maybe we can use this approach on a different level.

Pijukatel · 2026-03-16T08:51:50Z

pyproject.toml

    "scraping",
 ]
 dependencies = [
+    "apify-fingerprint-datapoints>=0.11.0",


We have all these added dependencies in the optional dependencies group playwright. So please remove them from here.

Pijukatel · 2026-03-16T09:50:55Z

src/crawlee/browsers/_playwright_browser.py

            user_data_dir = tempfile.mkdtemp(prefix=self._TMP_DIR_PREFIX)
            self._temp_dir = Path(user_data_dir)

+        launch_persistent_context_sig = inspect.signature(self._browser_type.launch_persistent_context)


This is a reasonable approach, but it has some drawbacks. If user has just typo ( in otherwise valid argument name), it will just show warning in log. Same for using some completely nonsensical argument. That should raise an error and not just log a warning.

For example, this should raise (typo in headles):

persist_browser = PlaywrightPersistentBrowser( playwright.chromium, browser_launch_options={'headles': True} )

Maybe this approach could be adopted one lever higher (not in PlaywrightPersistentBrowser - which always just calls launch_persistent_context), but in PlaywrightBrowserController - that is the class that decides about calling launch_persistent_context or new_context, but feeds them the same arguments.

It should properly raise exceptions for bad arguments, but it could just log a warning as per your suggestion for arguments at least valid in the other method. It would have to get 3 sets of arguments to be able to do such a distinction. Something like:

... launch_persistent_context_sig = set(inspect.signature(BrowserType.launch_persistent_context).parameters) new_context_sig = set(inspect.signature(Browser.new_context).parameters) persistent_unique_options = launch_persistent_context_sig - new_context_sig new_context_unique_options = new_context_sig - launch_persistent_context_sig common_options = launch_persistent_context_sig & new_context_sig ...

And then raise an exception or just log based on the selected mode.

…owserController Moving the validation logic from the browser instance to its controller as suggested by the reviewer. This improves user experience by raising TypeError for typos and nonsensical arguments while still providing helpful warnings for valid but incompatible cross-mode options like storage_state in persistent contexts. Also fixed dependency management in pyproject.toml.

sushant-mutnale · 2026-03-18T06:08:47Z

Hello! Thank you for the detailed feedback. I've refactored the validation logic into

PlaywrightBrowserController using the suggested three-set approach with cached signatures. I also moved the dependencies back to the optional group in
pyproject.toml.

New unit tests cover both the warning and error scenarios. Ready for another look!

Ran ruff formatter to fix CI lint error.

vdusek · 2026-03-19T09:07:09Z

pyproject.toml

+    "browserforge>=1.2.4",
    "cachetools>=5.5.0",
    "colorama>=0.4.0",
    "impit>=0.8.0",
    "more-itertools>=10.2.0",
+    "playwright>=1.58.0",


browserforge and playwright should not be part of core dependencies

vdusek · 2026-03-19T09:07:24Z

pyproject.toml

    "playwright>=1.27.0",
    "scikit-learn>=1.6.0",
-    "apify_fingerprint_datapoints>=0.0.3",
+    "apify_fingerprint_datapoints>=0.11.0",


vdusek · 2026-03-19T09:12:25Z

src/crawlee/browsers/_playwright_browser_controller.py

+_launch_persistent_context_params = set(inspect.signature(PlaywrightBrowserType.launch_persistent_context).parameters)
+_new_context_params = set(inspect.signature(Browser.new_context).parameters)


Is it necessary to run these at the import time of the module?

Removed browserforge and playwright from core dependencies in pyproject.toml as they belong in optional dependencies. Refactored Playwright signature cache in _playwright_browser_controller.py to load lazily via lru_cache rather than at module import time, preventing overhead when Playwright is not used.

Pijukatel

Thanks for the changes, and apologies for the delayed review. Just a few small comments, and I think it will be ready.

Pijukatel · 2026-03-23T09:00:00Z

src/crawlee/browsers/_playwright_browser_controller.py

+        filtered_options = {}
+        for key, value in browser_new_context_options.items():
+            if self._use_incognito_pages:
+                # Incognito mode (new_context)
+                if key in params_cache['common'] or key in params_cache['incognito_unique']:
+                    filtered_options[key] = value
+                elif key in params_cache['persistent_unique']:
+                    logger.warning(
+                        f'Option "{key}" is only supported in persistent context mode '
+                        '(use_incognito_pages=False) and will be ignored.'
+                    )
+                else:
+                    raise TypeError(f'"{key}" is not a valid Playwright context option.')
+            elif key in params_cache['common'] or key in params_cache['persistent_unique']:
+                # Persistent mode (launch_persistent_context)
+                filtered_options[key] = value
+            elif key in params_cache['incognito_unique']:
+                logger.warning(
+                    f'Option "{key}" is only supported in incognito context mode '
+                    '(use_incognito_pages=True) and will be ignored.'
+                )
+            else:
+                raise TypeError(f'"{key}" is not a valid Playwright context option.')


Could you please extract to a standalone private method with docstring explaining it.
filtered_options = self._filter_new_context_options(options=browser_new_context_options)

Pijukatel · 2026-03-23T09:05:31Z

src/crawlee/browsers/_playwright_browser_controller.py

+                        '(use_incognito_pages=False) and will be ignored.'
+                    )
+                else:
+                    raise TypeError(f'"{key}" is not a valid Playwright context option.')


I do not think we need to raise here; it is better for the Playwright code to raise, so that anyone can see the code where the arguments are defined.

It will be sufficient to filter out the arguments valid in the other case and warn for those, while letting the completely wrong arguments go through, and let them fail in Playwright. So it can be simplified to something like

if self._use_incognito_pages and key in params_cache['persistent_unique']: logger.warning( f'Option "{key}" is only supported in persistent context mode ' '(use_incognito_pages=False) and will be ignored.' ) elif not self._use_incognito_pages and key in params_cache['incognito_unique']: logger.warning( f'Option "{key}" is only supported in incognito context mode ' '(use_incognito_pages=True) and will be ignored.' ) else: filtered_options[key] = value

…ints

fix(playwright): filter unsupported context options in persistent bro…

3ce9bb3

…wser This addresses issue apify#1784 by dynamically filtering options passed to launch_persistent_context and providing a warning log for ignored options like storage_state.

janbuchar requested a review from Pijukatel March 16, 2026 09:11

Pijukatel requested changes Mar 16, 2026

View reviewed changes

style: fix formatting in test_playwright_controller_validation.py

57da910

Ran ruff formatter to fix CI lint error.

vdusek requested changes Mar 19, 2026

View reviewed changes

Pijukatel reviewed Mar 23, 2026

View reviewed changes

sushant-mutnale added 2 commits March 24, 2026 08:49

chore: revert unintentional version bumps of apify_fingerprint_datapo…

2021f04

…ints

chore: sync uv.lock after removing accidental dependencies

cd84846

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(playwright): filter unsupported context options in persistent browser#1796

fix(playwright): filter unsupported context options in persistent browser#1796
sushant-mutnale wants to merge 6 commits intoapify:masterfrom
sushant-mutnale:fix/playwright-context-options

sushant-mutnale commented Mar 16, 2026

Uh oh!

Pijukatel left a comment

Uh oh!

Pijukatel Mar 16, 2026

Uh oh!

Pijukatel Mar 16, 2026

Uh oh!

sushant-mutnale commented Mar 18, 2026

Uh oh!

vdusek Mar 19, 2026

Uh oh!

vdusek Mar 19, 2026

Uh oh!

vdusek Mar 19, 2026

Uh oh!

Pijukatel left a comment

Uh oh!

Pijukatel Mar 23, 2026

Uh oh!

Pijukatel Mar 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

		_launch_persistent_context_params = set(inspect.signature(PlaywrightBrowserType.launch_persistent_context).parameters)
		_new_context_params = set(inspect.signature(Browser.new_context).parameters)

Conversation

sushant-mutnale commented Mar 16, 2026

Uh oh!

Pijukatel left a comment

Choose a reason for hiding this comment

Uh oh!

Pijukatel Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

Pijukatel Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

sushant-mutnale commented Mar 18, 2026

Uh oh!

vdusek Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

vdusek Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

vdusek Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

Pijukatel left a comment

Choose a reason for hiding this comment

Uh oh!

Pijukatel Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Pijukatel Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants