Replies: 3 comments 42 replies
-
|
You have Alternatively, if you have a way to reproduce the crash, you could start frankenphp through gdb and get into a debugging session that way. |
Beta Was this translation helpful? Give feedback.
-
|
I got a new core dump from FrankenPHP running in a Docker container as the debug build. I think it's more usable now, but still missing debug symbols in libphp, right? |
Beta Was this translation helpful? Give feedback.
-
|
Only by compiling php yourself I think. https://github.com/php/frankenphp/blob/main/dev.Dockerfile for pointers. The current stack trace is already a lot more informative. It seems to hint at an extension doing something incorrectly. Or a php library, though I think that should only be possible with worker mode. |
Beta Was this translation helpful? Give feedback.
-
|
Sorry, I'm stupid. libphp has to be compiled into FrankenPHP, not replaced, right? |
Beta Was this translation helpful? Give feedback.
-
|
No, libphp is compiled into a shared library, frankenphp is linked against it. Simply replacing it should be enough, though frankenphp won't show anything different on startup. It stores the version info and stuff from headers at build time, not from the actually loaded shared library at runtime. Confusing, I know. |
Beta Was this translation helpful? Give feedback.
-
|
OK, but replacing it doesn't allow the PHP extensions to load. So, if I understand correctly, I have to first build libphp with enable debug, then build the modules and FrankenPHP? Sorry, I'm a total noob when it comes to the FrankenPHP fundamentals. |
Beta Was this translation helpful? Give feedback.
-
|
Looking at the trace, the segfault seems to occur in a shutdown function. More explicitly, the shutdown function manipulates output buffers (something like I remember there was a similar issue with a third party extension once, don't know which one it was. Do maybe use blackfire or something similar? |
Beta Was this translation helpful? Give feedback.
-
|
That parameter was actually the first thing on my mental list to check, but "unfortunately" What's even "worse", this app doesn't use Doctrine, only HTTP requests to an API. So I've been digging into all the places where maybe curl handles could leak, but no success so far. The app was running with FrankenPHP 1.10, PHP 8.4 and Symfony 7.3 for months without problems (at least as far as I know) and after 1.11, 8.5 and Symfony 8 suddenly multiple OOMs per hour. |
Beta Was this translation helpful? Give feedback.
-
|
Oh, and regarding Sentry: I'm clearing the scope because that's the first thing that came to my mind. |
Beta Was this translation helpful? Give feedback.
-
|
Have you tried removing Sentry entirely for a test? I'm not using it anymore and haven't had another crash. Although hard to tell, because I had them so rarely before too. |
Beta Was this translation helpful? Give feedback.
-
That's the last point on my list. I disabled Sentry's Monolog integration and tracing, no success so far. Next week I'll try a few more things and then remove Sentry entirely. Got sidetracked by trying to get Blackfire running (it doesn't and crashes FrankenPHP on each terminate, so currently I cannot recommend it at all). Just for reference the configs. sentry.yamlparameters:
env(SENTRY_DSN): ''
env(SENTRY_ENVIRONMENT): 'development'
when@prod:
sentry:
dsn: '%env(SENTRY_DSN)%'
options:
environment: '%env(SENTRY_ENVIRONMENT)%'
send_default_pii: true
enable_logs: true
sample_rate: 1
traces_sample_rate: 0
ignore_exceptions:
- 'Symfony\Component\HttpKernel\Exception\BadRequestHttpException' # 400
- 'Symfony\Component\HttpKernel\Exception\UnauthorizedHttpException' # 401
- 'Symfony\Component\HttpKernel\Exception\AccessDeniedHttpException' # 403
- 'Symfony\Component\HttpKernel\Exception\NotFoundHttpException' # 404
- 'Symfony\Component\HttpKernel\Exception\MethodNotAllowedHttpException' # 405
- 'Symfony\Component\HttpKernel\Exception\NotAcceptableHttpException' # 406
- 'Symfony\Component\HttpKernel\Exception\GoneHttpException' # 410
- 'Symfony\Component\HttpKernel\Exception\LockedHttpException' # 423
- 'Symfony\Component\HttpKernel\Exception\ServiceUnavailableHttpException' # 503
services:
Sentry\SentryBundle\Monolog\LogsHandler:
arguments:
$level: !php/const Monolog\Logger::ERRORmonolog.yamlmonolog:
channels:
- deprecation # Deprecations are logged in the dedicated "deprecation" channel when it exists
when@prod:
monolog:
handlers:
main:
type: fingers_crossed
action_level: error
handler: stream
excluded_http_codes: [400, 403, 404, 405, 406, 410, 503]
channels: ["!deprecation"]
buffer_size: 50 # How many messages should be saved? Prevent memory leaks
stream:
type: stream
path: php://stderr
formatter: monolog.formatter.json
console:
type: console
process_psr_3_messages: false
channels: ["!event", "!doctrine", "!deprecation"]
sentry_logs:
type: service
id: Sentry\SentryBundle\Monolog\LogsHandler
deprecation:
type: 'null'
channels: [deprecation] |
Beta Was this translation helpful? Give feedback.
-
|
Ah that's a shame, I'm using Tideways now which works perfectly well, but it doesn't have a memory breakdown and doesn't have all the monitoring features of Sentry. Much easier to integrate across multiple projects without code changes, though. |
Beta Was this translation helpful? Give feedback.
-
|
Not directly related to the threads above, but maybe food for thought: is there anything that could be done on the FrankenPHP side to help with debugging memory leaks in code? Searching for it I found quite a few people running into the same problem. A bigger code base and FrankenPHP in worker mode with no "easy" way to profile memory allocations for longer running processes. Blackfire, Sentry, XDebug, no tool seems to be able to help. All "solutions" so far I've come across were "I poked around at the code base until I found the leak". The DX feature of my dreams would be "Hey, here's your memory allocation per class / property / global. This is the $property in this class.php that consumed 100MBs of memory before kernel (worker?) shutdown." Can something like that even be done? |
Beta Was this translation helpful? Give feedback.
-
|
It probably can, but that sounds much outside the scope of the project and better suited for a profiler. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah I think that would indeed be possible. I wonder if it might even be possible as a PHP library via reflection. Just gather all globals and statics after a request and check if any arrays/strings are growing out of bounds. Might be more complicated than it sounds. |
Beta Was this translation helpful? Give feedback.
-
You know what? That didn't even occur to me so far. I'll dig deeper into whether PHP gives you a way to reliably do that next week.
I get that that's outside of FrankenPHP, but being the first actual product with a worker concept that's worth using from my point of view (also being a big Caddy fan), it would be a great companion project. |
Beta Was this translation helpful? Give feedback.
-
|
Thinking harder about it I don't think it can be done in PHP. If I'm somehow able to track instances (e.g. through the autoloader) I keep them in memory and artificially blow it up. I'll try to find out what a tick handler could do, but I'm not too optimistic. |
Beta Was this translation helpful? Give feedback.
-
|
I don't think this can be done in php at all, it would need to be a zend_extension hooking into all allocations and vm calls (and gc) to get a somewhat useful picture. I thought this is what blackfire did, but it's disappointing that they still haven't fixed ZTS compatibility. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Recently (starting around a deploy on January 2nd) I've been getting crashes in our Docker container without any indication on why and where they happen. I cannot reproduce them locally and they happen infrequently (sometimes multiple times in a few hours, sometimes after days).
After removing core limits, I get huge core dumps of 1.4GB (PHP memory limit 256MB). The only log line that comes at least a little bit close is
docker-97da20bc7166cdb1414cde3710ee7089b886fca5966096203743b17cc6d01936.scope: Consumed 6h 20min 6.235s CPU time, 1.7G memory peak, 44M read from disk, 1.9G written to disk.from the docker unit, where typically a container would write only a few MBs in days.I compiled FrankenPHP with the same Caddy modules our live container is using and debugging symbols enabled:
Here's what gdb has to say:
If it was PHP code, I would expect it to fail and then just log like any normal "out of memory error". I suspect it's a PHP extension, but except for updating FrankenPHP to 1.11 there were no other updates corresponding to the deploy date.
After running for two days now I can't see anything suspicious in the metrics either:
The worker crashes correspond to a strange
Maximum execution time of 35 seconds exceeded in /application/vendor/symfony/http-foundation/Response.phpwe've been getting for a very long time now. There are no scripts that could somehow run in this timeout at all, so recently I've been thinking these are client related issues with HTTP/3 (client goes away before being able to send early hints, e.g.), but that's another issue entirely.frankenphp build-info
Beta Was this translation helpful? Give feedback.
All reactions