Python 3 support 🙲 locking of dependencies

[cerna]

Changing the supported version of Python from 2 to 3 in Machinekit-HAL dependency (in machinekit/machinekit-hal#303) caused that EMCApplication Debian packages are installable, but the software is actuall unusable.

This is because EMCApplication has Machinekit-HAL as a dependency, but is comfortable with anything >0.4. In hindsight this is naïve as the development of Machinekit-HAL (and other dependencies) can go at paces different from EMCApplication, so (minimally when creating binary packages) the dependencies should be locked at known version.

In case with Machinekit-HAL, the version with which the EMCApplication was build with.

[jallwine]

Didn't LinuxCNC finish porting from Python 2 to 3? After installing EMCApplication on top of MachineKit-HAL, I can only import linuxcnc from Python 2.

[cerna]

@jallwine,
I don't think they fully finished it. But yes, the LinuxCNC should be able to run on Python3. The problem described here has multiple vectors:

1. I made mistake in presumption that binary packages created from this repository (Debian) will be able to run with any version of Machinekit-HAL. What I should have done is to pin the required package of Machinekit-HAL libraries in the EMCApplication binary package.

2. Debian packaging in LinuxCNC does not actually take Python 3 into account right now. It is still building packages based on Python 2. (Look for example on this line.) And there is no switch I can toggle to make it use Python 3. There is actually discussion how to make the switch, so I was waiting on this. (As everything I do here should be minimal.) But it looks like I will have to solve it and when it is decided in LinuxCNC upstream, revert it back/bend it to the decision.

[jallwine]

When I build LinuxCNC from source using this configure command, I do get a python3 version of the linuxcnc module:

./configure --with-realtime=uspace --with-python=python3 --with-boost-python=boost_python-py35

[cerna]

@jallwine,
yes, sorry, I was not precise. I am talking about Debian (and Ubuntu) packaging. You can of course build in-tree version with Python 3.

Looks like @rene-dev is busy working and actually has support for Python 3 based packages in LinuxCNC/linuxcnc#943.

[jallwine]

Gotcha, so following the instructions in the EMCApplication README I can build it from source to have Python 3 support?

Copied mostly from the README, but with the python flags:

git clone https://github.com/machinekit/emcapplication.git
cd emcapplication
sudo apt install build-essential fakeroot devscripts python
debian/configure machinekit-hal no-docs
mk-build-deps -irs sudo
cd src
./autogen.sh
./configure --with-hal=machinekit-hal --with-python=python3 --with-boost-python=boost_python-py35
make
sudo make install
cd ..
source ./scripts/rip-environment

[rene-dev]

linuxcnc master works with python3, but no python3 packages have been released, and it is currently not the default.
Main reason is that the only UI that works is axis.
my branch fixes the UIs, and packaging. so if you want UIs other than axis, use my branch.
is EMCApplication just linuxcnc with updated build system to accept MK HAL? if yes, how large is the diff?

[cerna]

@jallwine,

Gotcha, so following the instructions in the EMCApplication README I can build it from source to have Python 3 support?

Yes. (At least I have just tried it with update PR #4.)

Copied mostly from the README, but with the python flags:

Yup, this should work!

[cerna]

is EMCApplication just linuxcnc with updated build system to accept MK HAL? if yes, how large is the diff?

It is basically patched LinuxCNC@master branch to build and work against Machinekit-HAL repository (RuntimeAPI, HAL, modules etc.). Only the EMC application (+ at least one GUI) is expected to work.

The whole idea is to have a bunch of Featured Applications running against Machinekit-HAL and support that way the modularized approach. (Another one is the HAL ROS integration.)

my branch fixes the UIs, and packaging. so if you want UIs other than axis, use my branch.

Thanks, but the idea behind EMCApplication is to have minimal patching and LinuxCNC@master tracking. So I will have to wait until it is merged to master.

[rene-dev]

Is there any reason the updated build system can't be merged into linuxcnc master?

[zultron]

Is there any reason the updated build system can't be merged into linuxcnc master?

It would be fantastic if the LCNC project could merge the changes for building EMC against an external HAL. When I wrote them, I deliberately made them as minimal as possible, both to make them easier to merge into updated LCNC, and also in hopes they would be easier to review for LCNC inclusion.

@cerna knows best the latest status, but once the EMCApplication is building smoothly in CI and we've worked out any remaining issues, I'd consider the changes ready for inclusion upstream.

[cerna]

Is there any reason the updated build system can't be merged into linuxcnc master?

Because I didn't think (@zultron probably too) that it would fly. I still don't. This is basically packaging parts of the LinuxCNC for use in other programs, it could be a precursor to splitting the LinuxCNC into HAL, CNC and GUIs parts. And I just cannot see the community going for it.

Or you mean the Python 3 packaging for Debian? Well, I wouldn't be too much considerate to the non-Python 3 supporting parts. One GUI is good enough for me. And now you, heavyweight in LinuxCNC development community, are doing it.

The status is that now the CI part must be done. Given that the Machinekit wants to use Machinekit-CI repository for most CI related tasks, I need to abstract away the differences between all repositories to some useable and not-too-stupid level.

Then there are the tests, which not all pass because of environment/executable mismatch.

I haven't really spent any time on this recently and considered it lower priority. (It was working). So, so far I would consider good plan to implement machinekit-hal package locking and rebuild all images with Python2. It won't help @jallwine or others needing newer versions of HAL, but at least it will be useable.

[jallwine]

@rene-dev how much more is there to do to get your changes into LinuxCNC master? It seems like the move to Python 3 is actively progressing and rolling back to Python 2 isn't the best plan.

I would love to see LinuxCNC split up into HAL, CNC and GUIs. Encapsulation is the name of the game in software development. It's more maintainable, understandable and flexible. We don't use any of the GUIs, so its frustrating that they seem to be the main thing holding back progress. Python 2 has been EOL for almost a year now. I understand open source projects can lag behind due to the nature of volunteers driving development, and it can be difficult to get buy-in on large changes due to the limited resources to implement them, but if LinuxCNC was split up into more manageable pieces, it seems like it would be better for long term development. @cerna, @zultron what do you think the main resistance to splitting up would be?

[zultron]

@zultron what do you think the main resistance to splitting up would be?

Of course, this question should be asked in the LCNC project, but I do remember the idea being broached in the past.

I recall seeing only tepid support. Speculating, perhaps a lack of enthusiasm stems from not understanding the benefits, or a low benefit to effort ratio. (FWIW, I'm obviously in favor of splitting!)

The only concrete reason opposing a split that I recall concerned a fear that users submitting issues would be confused about which repo to file the issue in, for example the UI, the EMC or the HAL repo. IMO, if that's actually a significant problem, there are ways to mitigate it.