Website opening hangs

[bbrtj]

Hello,
thank you for this awesome distribution!

I'm having a problem with $webkit->open method - it sometimes hangs the execution of the program, tends to do this for the same urls, but that may be just a coincidence since my program always opens urls in the same order, and I see in code that this function waits for pending data on the page it's leaving. However, this happened to me even when I opened an url that only contained simple minified CSS.

The program then uses 100% CPU and doesn't seem to ever return to my caller function - waited ~10 mins. I can currently pretty deterministically reproduce this problem, but it happens after a login wall of a closed webpage so can't share any link you can follow. Could you assist me in debugging or at the very least give some insights on what could have gone wrong?

My pipeline is somewhat like this:

• open url, wait for the page to load
• wait for layout element to become visible
• wait additional 3 sec (a lot of JS lazyloading)
• run some javascript on the page (mostly get js-generated content) and analyze it
• move to the next page, repeating the process

My current best guess is that the page I'm leaving is loading a lot of stuff with javascript lazyloaders, which takes a lot of time and more for every batch. I could try to stop all the scripts on that page and just leave it, but I can't see a way in module's API to do this.

Any help will be appreciated, thank you!

[bbrtj]

Some info about my configuration:

• OpenSUSE leap 15.2 OS
• perl 5.32.0 installed with perlbrew
• required libraries installed from zypper repositories

[bbrtj]

Got some new info on this: ran WWW::Webkit2 without xvfb argument and was able to see that the page loaded just fine. For some reason the library still thinks that the page is not loaded. It only happens in a specific combination of page loads it seems, but they do not have to be consecutive. Once I click on something else and that page fully loads, the program continues to execute.
I'm diving into details to try to find the reason for this behavior.

[niner]

A reduced test case that reproduces the issue at least sometimes would be extremely useful. WebKit does behave strangely, sometimes ignoring navigation requests entirely when there are still requests on the way. And sometimes waiting for such requests is what causes us to hang in the first place. It's a difficult balance with a number of edge cases we have already discovered (like redirects and about:blank).

[bbrtj]

A test case will be very hard to collect because I'm not allowed to share fetched html (company's intranet system) and I would need to find the exact source of the problem to assemble something custom.

I was able to work around the problem by using ->view->load_uri + ->process_page_load instead of ->open. Seems like open hangs on the very last instruction ->process_page_load, but when I added it myself it runs just fine. ->open function contains two calls to ->process_page_load, and if I reproduce this in my script it also hangs just like ->open.

So, the first call to ->process_page_load makes the second call to it hang - because I can clearly see a page changing in webkit window. Hopefully that's useful enough so that you can narrow down and find the bug more easily. Let me know if I can check anything else

[bbrtj]

After thousands of pages loaded, my approach from the previous comment works great in 99.9% of the cases. I only had two pages where the program hung, had to keep the window visible so that I can navigate somewhere else and continue with the execution. No idea why it happened, these pages had nothing on them visible that would be that different from all the other pages.

When it hangs, refreshing and navigating to the same page does nothing. Only going to a different url puts my program back in the driver's seat