Build of 0.3.3 fails with `TypeError: '>' not supported between instances of 'NoneType' and 'float'`

[dkasak]

Downloading https://github.com/trailofbits/polyfile/archive/refs/tags/v0.3.3.tar.gz and building yields an error.

❯ python setup.py build

Traceback (most recent call last):
  File "/home/dkasak/code/dkasak/packages/aur_packages/polyfile/src/polyfile-0.3.3/setup.py", line 47, in <module>
    if not MANIFEST_PATH.exists() or newest_definition > MANIFEST_PATH.stat().st_mtime:
TypeError: '>' not supported between instances of 'NoneType' and 'float'

Looking at setup.py, the cause is here

# see if any of the files are out of date and need to be recompiled
newest_definition: Optional[float] = None
for definition in KAITAI_FORMAT_LIBRARY.glob("**/*.ksy"):
    mtime = definition.stat().st_mtime
    if newest_definition is None or newest_definition < mtime:
        newest_definition = mtime

It turns out that the KAITAI_FORMAT_LIBRARY: Path = POLYFILE_DIR / "kaitai_struct_formats" dir is empty so the above loop never gets executed and newest_definition remains None:

❯ ls -l kaitai_struct_formats
total 0

[hbrodin]

Hi dkasak,
Thanks for reaching out! I verified myself that I cannot download the v0.3.3.tar.gz and build according to your description. The error I see is different, it indicates that I am not able to fetch git submodules (one of them being kaitai_struct_formats). I believe this is the root cause of the problem.

One option is to clone the repo for this release using the following command

git clone --branch v0.3.3 https://github.com/trailofbits/polyfile.git

When I run

python3 setup.py build

it works. Hope this helps.

[pbottine]

I believe this issue can be closed as it has been resolved in v0.5.0 and later versions.

Root Cause: The original issue in v0.3.3 occurred because the setup.py script attempted to determine if Kaitai parser files needed recompilation by comparing modification times. However, when installing from a tarball (rather than cloning the repository), the kaitai_struct_formats git submodule was not present, causing the comparison logic to fail with a TypeError when comparing None to a float.

Resolution: In commit 0a44b31 (November 14, 2022), the Kaitai rebuilding logic was refactored into a separate compile_kaitai_parsers.py script. This new implementation includes robust handling for missing submodules:

1. Lines 18-20 automatically initialize and update git submodules if they are missing:

if not (KAITAI_FORMAT_LIBRARY / "README.md").exists():
    subprocess.check_call(["git", "submodule", "init"], cwd=str(POLYFILE_DIR))
    subprocess.check_call(["git", "submodule", "update"], cwd=str(POLYFILE_DIR))

2. Lines 58-63 properly handle the case where no Kaitai definition files exist, using proper datetime objects instead of floats and correctly initializing the newest_definition variable.

Current Status: The issue has been fixed for over 3 years (since November 2022) and is resolved in all currently supported versions. Users experiencing this issue should upgrade to v0.5.0 or later.

Workaround for Legacy Versions: For users who must use v0.3.3, the workaround mentioned by @hbrodin remains valid:
git clone --branch v0.3.3 --recurse-submodules https://github.com/trailofbits/polyfile.git

As such, I recommend closing the issue.

cc: @smoelius