Collision with protobuf runtime and provided proto files after pip installation - Detected incompatible Protobuf Gencode/Runtime versions

[psch0rr]

Hey there,

I was facing a strange protobuf versioning issue during the installation of the kuksa-client.

I was using pip install kuksa-client to install the current version 0.5.1. So far so good, it was resolving all the needed dependencies to make the kuksa-client runnable.
The protobuf version is defined by the grpcio-tools dependency, which requires anything above 5.26.1, in our case it was 5.27.3, so no upgrade performed during the installation. (see below)

sudo pip install kuksa-client
Collecting kuksa-client

...
 Using cached kuksa_client-0.5.1-py3-none-any.whl (82 kB)Requirement already satisfied: protobuf<6.0dev,>=5.26.1 in 
/usr/local/lib/python3.10/dist-packages (from grpcio-tools==1.68.0->kuksa-client) (5.27.3)
...

Installing collected packages: kuksa-client
Successfully installed kuksa-client-0.5.1

Ok still good, the issue started as soon as I was starting my application:

It raises a runtime and generation version error: google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime.
It seems that the shipped protofiles from you are generated using the 5.28.1 version. So they are not compatible, or at least raise this error, if you do a normal installation using pip.

Feb 11 14:22:44 fgvt-ivs-11-au bash[1441142]:   File "/usr/local/lib/python3.10/dist-packages/kuksa/val/v1/types_pb2.py", line 12, in <module>
Feb 11 14:22:44 fgvt-ivs-11-au bash[1441142]:     _runtime_version.ValidateProtobufRuntimeVersion(
Feb 11 14:22:44 fgvt-ivs-11-au bash[1441142]:   File "/usr/local/lib/python3.10/dist-packages/google/protobuf/runtime_version.py", line 86, in ValidateProtobufRuntimeVersion
Feb 11 14:22:44 fgvt-ivs-11-au bash[1441142]:     raise VersionError(
Feb 11 14:22:44 fgvt-ivs-11-au bash[1441142]: google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading kuksa/val/v1/types.proto: gencode 5.28.1 runtime 5.27.3. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

Solution

Solution was to manually upgrade protobuf to 5.29.6, what is the latest compatible version of Protobuf supporting the grpcio-tools==1.68.0 what is mandatory for the kuksa-client.
The current protobuf-6.33.5 only works with grpcio-tools-1.78.0, no longer with the grpcio-tools-1.68.0

Question

Is is possible to add a protobuf version to the pip dependencies equal or above the proto gen version to ensure that the installation is working again ootb?

Thanks a lot and best regards
Philipp

[lukasmittag]

Hey @psch0rr, that sounds good! Do you want to create the fix? Else I will take it over :)

[erikbosch]

I know that protobuf/grpcio-versioning has caused problems in the past as well. If hardcoding protobuf version in pyproject.toml we should possible also include it in setup.cfg, and update the instructions at https://github.com/eclipse-kuksa/kuksa-python-sdk/wiki/Release-Process#update-dependencies on how to keep them in sync going forward.

The requirements.txt file was just updated, see https://github.com/eclipse-kuksa/kuksa-python-sdk/commits/main/kuksa-client/requirements.txt

[psch0rr]

Hey, thanks for this quick reply!

@erikbosch you are right in the requirement.txt the protobuf version was always fine, and it would have worked if this version mentioned in the requirement.txt would have been installed during the pip installation.

If I got you right, if we add the protobuf version to the pyproject.toml it would resolve the dependency during the pip installation, right?

But as long as we fix the grpcio-tools version to 1.68.0 pyproject.toml and this is the only part that needs the protobuf in a version above 5.26.1 and below 6.x.x. the hardcoded protobuf version would not result in a issue.
As long as we generate the protofiles with the same version, or am I wrong.

In addition grpcio-tools==1.68.0 only works until protobuf version 5.29.5, so that would be then the limiting factor, as long as we need grpcio-tools in that specified version.

Best regards
Philipp

[psch0rr]

@erikbosch I created a pull request adding the protobuf dependency to the pyproject.toml and setup.cfg as you proposed, I hope I met your idea.

In terms of updating the instructions:
https://github.com/eclipse-kuksa/kuksa-python-sdk/wiki/Release-Process#update-dependencies
Maybe we can extend the existing section like this:

It is also important to check that the protobuf gencode version not exceeds the protobuf runtime version in the pyproject.toml and setup.cfg to avoid Protobuf Gencode/Runtime errors.