perf: graphistry wheel size (625 kB) — GFQL engine files are large, investigate modularization

[lmeyerov]

Observation

Installing graphistry 0.53.16 from PyPI downloads a 625 kB wheel (2.75 MB uncompressed).

What's NOT the problem

Tests: not in wheel. graphistry/tests/ has no top-level __init__.py so find_packages() skips it entirely. Confirmed by inspecting the actual built wheel (212 files, 0 test files).

Deps are already clean. Confirmed in Docker (python:3.12-slim):

Use case	Required deps
import graphistry + basic GFQL	pandas, numpy, requests, pyarrow, typing_extensions, packaging
Cypher string GFQL	+ lark (already lazy — only imported on first parse)
squarify, palettable, scipy, sklearn, igraph, cugraph	not imported unless explicitly used

squarify is being eliminated separately.

What IS the problem

GFQL/Cypher engine files are large and may be bloated — grown organically without modularization passes:

File	Uncompressed	Lines
graphistry/compute/gfql/cypher/lowering.py	294 KB	8,212
graphistry/compute/gfql/row/pipeline.py	181 KB	3,976
graphistry/PlotterBase.py	150 KB	3,768
graphistry/feature_utils.py	112 KB	3,097
graphistry/pygraphistry.py	101 KB	2,653
graphistry/compute/gfql/cypher/parser.py	84 KB	1,949
graphistry/compute/gfql/temporal_text.py	74 KB	2,073
graphistry/compute/ast.py	66 KB	1,701
graphistry/compute/gfql_unified.py	58 KB	1,548
graphistry/compute/chain.py	50 KB	1,227

GFQL-specific files alone (lowering, pipeline, parser, temporal_text, ast, gfql_unified, chain) = ~808 KB uncompressed, ~29% of total wheel.

Scope

1. Audit lowering.py (8,212 lines) and pipeline.py (3,976 lines) for dead code, duplication, or extractable sub-modules
2. Same for temporal_text.py and ast.py
3. Modularization into focused sub-files improves maintainability regardless of wheel impact
4. Separately: PlotterBase.py and feature_utils.py are non-GFQL large files worth a similar pass

Not urgent — 625 kB downloads in ~18ms on a typical connection. Worth doing as engineering hygiene when touching these files for other reasons.

[lmeyerov]

Investigation findings

The wheel is clean — no tests, no data files, no non-code assets. The 625 kB is entirely real application code.

Top contributors

File	Size	% of wheel
graphistry/compute/gfql/cypher/lowering.py	294 KB	~47%
graphistry/compute/gfql/row/pipeline.py	181 KB	~29%
graphistry/PlotterBase.py	150 KB	~24%
graphistry/feature_utils.py	112 KB	~18%
graphistry/pygraphistry.py	101 KB	~16%

Why it's large

The compute/GFQL/Cypher layer (graphistry/compute/) accounts for ~55% of the wheel (1.51 MB of source, compresses to ~340 KB in the wheel). This is mostly the Cypher lowering, row pipeline, parser, and AST — all real query engine code added over the past year.

No easy wins

• Tests are already excluded (find_packages() excludes graphistry/tests/)
• No bundled data files or grammar files
• No duplicate or generated assets

Reduction options

1. Split lowering.py (8,212 lines) into sub-modules — good engineering hygiene, marginal wheel benefit since Python compresses well
2. Lazy imports — defer heavy optional deps (feature_utils, umap, etc.) so the import-time surface is smaller (doesn't affect wheel size but improves cold-start)
3. Accept the size — 625 kB for a full graph query engine + visualization API is reasonable; Colab installs at 35 MB/s so it's ~18ms download

Updating target: this is probably not worth a major refactor purely for wheel size. Closing as "won't fix" unless there's a measured install-time regression or a specific embedding context where it matters.

[lmeyerov]

Minimal-install investigation: can basic GFQL work with only pandas?

Answer: No — requests and pyarrow are hard module-level blockers today.

Empirically confirmed (Docker python:3.12-slim)

Deps installed	import graphistry	basic gfql([n(),e(),n()])
pandas only	❌ ModuleNotFoundError: requests	—
+ requests	❌ ModuleNotFoundError: pyarrow	—
+ requests + pyarrow	✅	✅

Exact blockers

1. graphistry/pygraphistry.py line 12: import ... requests ... (module-level)
2. graphistry/arrow_uploader.py line 3: import io, pyarrow as pa, requests, sys (module-level, imported unconditionally from pygraphistry.py)
3. graphistry/PlotterBase.py line 9: import ... pyarrow as pa ... (module-level)

What's already lazy ✅

• lark — already deferred inside _lark_imports() in cypher/parser.py
• All GFQL compute internals (chain.py, gfql_unified.py, ast.py) — clean, no heavy deps

Fix path to pandas-only GFQL

Three targeted changes needed:

1. Move requests import in pygraphistry.py inside functions that use it (auth, upload, HTTP calls)
2. Move pyarrow import in arrow_uploader.py inside the ArrowUploader/ArrowFileUploader methods that need it
3. Move pyarrow import in PlotterBase.py inside .plot() / upload-path methods

None of these require pyarrow at class-definition time — it's only needed when actually serializing/uploading data to the Graphistry server.

requests is similarly only needed for .register() / .plot() / upload paths, not for local GFQL computation.

Filed as scope for this issue.

[lmeyerov]

Confirmed minimal dependency matrix (Docker python:3.12-slim clean room)

Use case	Required deps	Status
import graphistry	pandas, numpy, requests, pyarrow, typing_extensions, packaging	✅ clean
basic GFQL gfql([n(), e(), n()])	same	✅ clean
predicate GFQL gfql([n({'val': 1}), ...])	same	✅ clean
multihop GFQL gfql([n(), e(hops=2), n()])	same	✅ clean
Cypher GFQL gfql('MATCH ...')	+ lark	✅ clean (lark already lazy)

squarify, palettable, lark, scipy, sklearn, igraph, cugraph — none are imported at graphistry import time or during basic/Cypher GFQL. All properly lazy or optional.

numpy is a transitive dep of pandas so not separately called out.

Summary

The dependency story is already good:

• pandas + requests + pyarrow = everything you need for GFQL (local compute, no upload)
• + lark = Cypher string queries
• Everything else (squarify, palettable, ML deps) stays unimported until explicitly used

No lazy-import work needed for the GFQL use case. The squarify elimination (already in progress) will further clean up the install footprint.

Closing scope to: wheel size is fine, import hygiene is fine. Leaving open only to track squarify removal.

[lmeyerov]

#1058 audit closeout status:

• Opened draft PR chore(gfql): remove unused cypher lowering helper #1500 with a narrow safe cleanup: removed unused private Cypher lowering helper _expr_aggregate_match_aliases and updated CHANGELOG.md.
• Filed broad follow-up modularization issues instead of making a risky mega-PR:
  • GFQL/Cypher: extract reentry helper ownership out of lowering.py #1497 reentry helper ownership out of lowering.py
  • GFQL temporal_text.py: split temporal parsing/folding domains #1498 split temporal_text.py
  • GFQL row pipeline: split hot-path call families into focused modules #1499 split row pipeline call families
• Review-skill pass converged: 2 waves, no findings.
• Validation:
  • ruff/typecheck passed on touched lowering file
  • focused local CPU lowering slice passed: 212 passed, 859 deselected
  • local cuDF slice was blocked only by no local CUDA device; DGX RAPIDS 25.02 + 26.02 explicit cuDF node IDs passed
  • GitHub CI green, including TCK and ReadTheDocs

Leaving #1058 open for coordinator/owner decision because the PR intentionally uses Refs #1058 and broader modularization is now split out.

[lmeyerov]

#1058 closeout:

• Merged PR chore(gfql): remove unused cypher lowering helper #1500: chore(gfql): remove unused cypher lowering helper #1500
• Merge commit: 6a9905e5847250672d5fd69da535d5778e49e730
• Scope stayed narrow: removed unused private Cypher lowering helper _expr_aggregate_match_aliases and updated CHANGELOG.md.
• Review skill converged with no findings.
• Validation passed:
  • focused local ruff/typecheck on graphistry/compute/gfql/cypher/lowering.py
  • focused local CPU Cypher lowering tests after final rebase: 213 passed, 864 deselected
  • prior DGX RAPIDS 25.02 + 26.02 explicit cuDF slice passed
  • GitHub CI green on final rebased head, including TCK and ReadTheDocs
• TCK note: a stale matching tck-gfql branch caused an earlier CI failure; redundant TCK PR fix(igraph test installer): xenial could not use the ppa #136 was closed and the stale branch was deleted so pygraphistry CI uses updated tck-gfql main.

Closing #1058 as the audit closeout. Broader modularization remains split into focused follow-ups:

• GFQL/Cypher: extract reentry helper ownership out of lowering.py #1497 reentry helper ownership out of lowering.py
• GFQL temporal_text.py: split temporal parsing/folding domains #1498 split temporal_text.py
• GFQL row pipeline: split hot-path call families into focused modules #1499 split row pipeline call families

[lmeyerov]

#1058 addendum: split-out row-pipeline follow-up #1499 is complete.

• Merged PR Split GFQL row frame operations #1507: Split GFQL row frame operations #1507
• Merge commit: dfb59912dd622c7fa65aeb36039a4dcc50165d9d
• Scope: behavior-preserving extraction of row-frame hot-path operations from row/pipeline.py to row/frame_ops.py.
• Validation: local focused row/GFQL tests, ruff/typecheck/surface guard, DGX RAPIDS 25.02 + 26.02 cuDF slice, review skill convergence, and green GitHub CI.

No new GFQL/Cypher semantics were added; this was a mechanical modularization lane.

[lmeyerov]

#1058 addendum: split-out reentry helper ownership follow-up #1497 is complete.

• Merged PR Extract reentry helper ownership from lowering #1506: Extract reentry helper ownership from lowering #1506
• Merge commit: f5ef17b465777ebef2a5f002bafdf1191785fdd0
• Scope: behavior-preserving extraction of bounded-reentry lowering helper ownership from cypher/lowering.py to cypher/reentry/lowering_support.py.
• Validation: focused local reentry/lowering tests, ruff/typecheck/surface guard, review skill convergence, DGX RAPIDS 25.02 + 26.02 full GFQL evidence on the pre-final-rebase code head, and green GitHub CI on final merged head.

No runtime/compiler semantics were intentionally changed; this was a modularization cleanup lane for the GFQL/Cypher file-size/ownership train.