pycparser 3.0 compatibiilty

[bkmgit]

Is there interest in making pycparser-ext work with pycparser 3.0 ?

From #99 (comment) it seems this will require some work, but unclear if it is desirable to have this done or if the long term aim is to remain on pycparser < 3

[inducer]

I don't use pycparserext myself for anything at the moment, so it's hard for me to justify sinking time into even reviewing such a thing, given the anticipated scale. As such, I think a pycparser 3.0-compatible pycparserext would require a bigger break from the existing setup. I would be happy for someone who actually cares to take over maintainership. Alternatively, the project can be forked, too, but I can do a bit of work to help and avoid that if it helps provide a smoother upgrade path for users.

Btw: I had no idea this was used in a Chapel-adjacent tool, neat! 👋 Hi @bradcray, nice to (virtually?) see you!

[inducer]

Oh boy. In view of the times in which we are living, I've just let an AI loose on the porting task. Everything I said above still stands (i.e. I don't know how eager I am to review what it makes beyond a cursory look), but it costs me next to nothing, and curiosity got the better of me. We'll see what happens.

[inducer]

Well, that makes life more complicated. There's now a test-passing, mostly-reasonable-looking port hanging out in #102. I don't really know what to do with that fact. 🙂

[inducer]

If nothing else: Could you guys test whether it works for your use case?

[bradcray]

Cool! (😅 ?) As a simple test, we should be able to run it over the tests in our regression suite that exercise c2chapel. I don't think we have a ton of tests there, but will have to refresh my memory (and also check to make sure they exercise GNU extensions, which is the main reason we went with pycparserext, IIRC). I'll also try to refresh my memory as to cases where we've used it manually in the past that might make for good manual tests. @bkmgit, do you have motivating cases of your own to run it on?

I went looking at the version 3.0 release notes on pyparser, in hopes that maybe a feature of 3.0 would be gcc extensions, but that does not seem to be the case. And they point to pycparserext as a technology to fill that gap, so more people may find their way here to either test or help out (?).

[bradcray]

I needed a study break, so gave this a quick spin on our test directory and it seems to be working correctly? The only failure I saw was a case where we intentionally give it a bad C header that's actually a Chapel file:

proc foo(x : int);

and the error message has changed, from:

Unable to parse file: invalidC.h:2:6: before: foo

to:

Unable to parse file: invalidC.h:2:1: Invalid function definition

which seems very acceptable (and I think is due to the pycparser 3.0 change rather than anything in #102.

There were more tests than I was remembering/seeing at first glance, so that's encouraging. But I'm still not sure to what extent they exercise GNU extensions or not…

[bradcray]

Oh, it also looks like we've run c2chapel in the past on

• netcdf.h
• hdf5_hl.h
• ISO_Fortran_binding.h
• FFTW (not sure of the header name offhand…)

So those could be some other headers to run through it to see how it's working.

[inducer]

Well, even absent any GNU extensions, not completely destroying the output is already a result. If you give the changes in #102 a read, the code ties into the guts of pycparser in... I'm looking for a suitable adjective here... "delicate" (?) ways, with gobs of (apparently) copy-pasted code. (To the extent that @eliben likely should be mentioned as a copyright holder on that file.) So the fact that it does anything at all is a small miracle. I wish there was a different way, but I admit that, in the face of pycparser's recursive-descent rewrite, I'm not sure how else I would do it if I was doing it myself. By comparison, the integration via PLY was quite a bit less yucky, but that's not our choice to make.

[bradcray]

I happened to have copies of hdf5_hl.h and ISO_Fortran_bindings.h on my system, so ran them through c2chapel and that seemed to go OK. The tool didn't choke, the generated codes generally looked reasonable, and they compiled with almost no issues (in one case, I needed to move a pair of typedefs higher in the file to make things legit). So that's all encouraging. The generated HDF5 file was 5000+ lines, so that's something.

I don't feel like I can weigh in on whether this is a good chance for pycparserext or not. And I don't know that it ought to be done simply for our sake, in that we can pin to older versions of the tools and just stick with them, since they've worked well so far. That said, if this were merged and considered the future of pycparserext, I think we'd use it in order to keep up with the latest.

[bkmgit]

Will examine later this week. Thanks for the responses.