Updating DLCLive!-GUI to pyQt6 as well as DLC3/pytorch models #34

arturoptophys · 2025-12-05T15:27:04Z

This updates the GUI to pyQt6.

New features:

Hardware encoding of videos to offload to GPU (nvenc)
New processor selection and on the fly change.
remote control of the video/ processor - recording over processor socket.
visualization of a user definable bounding box

…modern-python-and-pyqt6 Add Basler and GenTL camera backends for modular capture

…era-functionality

…camera-functionality Rework layout and camera handling controls

…' into artur/test_update

…r integration - Implemented `get_device_count` method in `GenTLCameraBackend` to retrieve the number of GenTL devices detected. - Added `max_devices` configuration option in `CameraSettings` to limit device probing. - Introduced `BoundingBoxSettings` for bounding box visualization, integrated into the main GUI. - Enhanced `DLCLiveProcessor` to accept a processor instance during configuration. - Updated GUI to support processor selection and auto-recording based on processor commands. - Refactored camera properties handling and removed deprecated advanced properties editor. - Improved error handling and logging for processor connections and recording states.

…e display

Copilot

Pull request overview

This PR modernizes the DeepLabCut-live-GUI by migrating from Tkinter to PyQt6 and adding support for DLC3/PyTorch models. The update includes hardware-accelerated video encoding (NVENC), a new processor selection system with on-the-fly changes, remote control capabilities via processor sockets, and user-definable bounding box visualization.

Key Changes:

Complete UI rewrite from Tkinter to PyQt6
Added support for PyTorch-based DLC models
Implemented hardware-accelerated video recording with NVENC
Added processor plugin system with remote control capabilities
Introduced multiple camera backends (OpenCV, GenTL, Aravis, Basler)

Reviewed changes

Copilot reviewed 48 out of 49 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
setup.py	Updated version to 2.0, Python requirement to >=3.10, replaced dependencies for PyQt6
pyproject.toml	Added modern Python packaging configuration with comprehensive test setup
docs/user_guide.md	New comprehensive user guide for camera setup, DLC configuration, and recording
docs/timestamp_format.md	Documents JSON timestamp format for frame synchronization
docs/features.md	Detailed feature documentation including camera backends and processor system
docs/camera_support.md	Camera backend comparison and installation guides
docs/aravis_backend.md	Aravis backend documentation for GenICam cameras
docs/README.md	Documentation index with navigation by use case and topic
dlclivegui/video_recorder.py	New video recording module using vidgear with hardware acceleration
dlclivegui/gui.py	Complete PyQt6-based GUI implementation replacing Tkinter
dlclivegui/dlc_processor.py	DLCLive integration with threading and performance profiling
dlclivegui/config.py	Configuration management with dataclasses
dlclivegui/cameras/opencv_backend.py	OpenCV camera backend implementation
dlclivegui/cameras/gentl_backend.py	GenTL/Harvesters camera backend
dlclivegui/cameras/factory.py	Camera backend factory and detection
dlclivegui/cameras/base.py	Abstract camera backend base class
dlclivegui/processors/processor_utils.py	Processor plugin loading utilities
dlclivegui/processors/dlc_processor_socket.py	Socket-based processor with remote control
dlclivegui/processors/PLUGIN_SYSTEM.md	Processor plugin system documentation

Comments suppressed due to low confidence (1)

dlclivegui/processors/processor_utils.py:1

The docstring 'Returns:' section contains what appears to be an absolute file path rather than a description of the return value. This should describe the returned dictionary structure instead.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-05T16:44:45Z

dlclivegui/gui.py

+from dlclivegui.processors.processor_utils import instantiate_from_scan, scan_processor_folder
+from dlclivegui.video_recorder import RecorderStats, VideoRecorder
+
+os.environ["CUDA_VISIBLE_DEVICES"] = "1"


Hardcoded environment variable and absolute path should not be in production code. These appear to be development-specific settings that should be removed or made configurable through the configuration system.

Suggested change

os.environ["CUDA_VISIBLE_DEVICES"] = "1"

# If you need to restrict CUDA devices, set CUDA_VISIBLE_DEVICES externally or via configuration.

Copilot · 2025-12-05T16:44:46Z

dlclivegui/gui.py

+
+logging.basicConfig(level=logging.INFO)
+
+PATH2MODELS = "C:\\Users\\User\\Repos\\DeepLabCut-live-GUI\\dlc_training\\dlclive"


Hardcoded environment variable and absolute path should not be in production code. These appear to be development-specific settings that should be removed or made configurable through the configuration system.

Suggested change

PATH2MODELS = "C:\\Users\\User\\Repos\\DeepLabCut-live-GUI\\dlc_training\\dlclive"

# Use environment variable DLC_PATH2MODELS if set, otherwise default to a relative path

PATH2MODELS = os.environ.get("DLC_PATH2MODELS", str(Path(__file__).parent.parent / "dlc_training" / "dlclive"))

@arturoptophys see copilot suggestions; please let me know if you would like assistance!

dlclivegui/gui.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

MMathisLab · 2025-12-05T16:52:25Z

dlclivegui/cameras/base.py

+        return True
+
+    def stop(self) -> None:
+        """Request a graceful stop."""


… options to config

deruyter92

Amazing improvements, @arturoptophys!
I've tested it with python 3.10, 3.11 and 3.12. Only ran with pytorch models. Only a few specific comments/questions. Overall great improvement of the user experience and well-written code!

deruyter92 · 2025-12-11T06:55:36Z

dlclivegui/camera_controller.py

+    def stop(self, wait: bool = False, *, preserve_pending: bool = False) -> None:
+        if not self.is_running():
+            if not preserve_pending:
+                self._pending_settings = None
+            return
+        assert self._worker is not None
+        assert self._thread is not None
+        if not preserve_pending:
+            self._pending_settings = None
+        QMetaObject.invokeMethod(
+            self._worker,
+            "stop",
+            Qt.ConnectionType.QueuedConnection,
+        )
+        self._worker.stop()
+        self._thread.quit()
+        if wait:
+            self._thread.wait()


Is there a reason why self._worker.stop() is called twice? Otherwise only keep the QueuedConnection.

deruyter92 · 2025-12-11T07:32:49Z

dlclivegui/cameras/opencv_backend.py

+        # Try grab first - this is non-blocking and helps detect connection issues faster
+        grabbed = self._capture.grab()
+        if not grabbed:
+            # Check if camera is still opened - if not, it's a serious error
+            if not self._capture.isOpened():
+                raise RuntimeError("OpenCV camera connection lost")
+            # Otherwise treat as temporary frame read failure (timeout-like)
+            raise TimeoutError("Failed to grab frame from OpenCV camera (temporary)")


great error handling!

deruyter92 · 2025-12-11T07:38:15Z

dlclivegui/cameras/opencv_backend.py

+        # Set FPS if specified
+        if self.settings.fps:
+            self._capture.set(cv2.CAP_PROP_FPS, float(self.settings.fps))
+
+        # Set any additional properties from the properties dict
+        for prop, value in self.settings.properties.items():
+            if prop in ("api", "resolution"):
+                continue
+            try:
+                prop_id = int(prop)
+            except (TypeError, ValueError):
+                continue
+            self._capture.set(prop_id, float(value))
+
+        # Update actual FPS from camera
+        actual_fps = self._capture.get(cv2.CAP_PROP_FPS)
+        if actual_fps:
+            self.settings.fps = float(actual_fps)


Maybe useful to add logging here. E.g. if a property fails to set. Or when the actual_fps turns out to be different from self.settings.fps.

added logging if parameters dont go through or are still set to unexpected values

deruyter92 · 2025-12-11T07:45:57Z

dlclivegui/dlc_processor.py

+try:  # pragma: no cover - optional dependency
+    from dlclive import DLCLive  # type: ignore
+except Exception:  # pragma: no cover - handled gracefully
+    DLCLive = None  # type: ignore[assignment]
+


I think it would be definitely useful to log a warning here that DLCLive is not imported.

Also: printing the exception telling why DLCLive was not imported helps users debug cases where they unsucessfully tried to install DLCLive (e.g. DLCLive is installed, but one of the dependencies missing, resulting in a failed import).

added logging of the exception. would this be enough to pinpoint the issue?

deruyter92 · 2025-12-11T11:20:15Z

dlclivegui/dlc_processor.py

+    def _start_worker(self, init_frame: np.ndarray, init_timestamp: float) -> None:
+        if self._worker_thread is not None and self._worker_thread.is_alive():
+            return
+
+        self._queue = queue.Queue(maxsize=1)


What is the reasoning behind maxsize=1?

To only predict newest frames, as we want to use most up-to-date infos we dont need to care about older frames and just drop them

deruyter92 · 2025-12-11T11:33:43Z

docs/user_guide.md

+
+## DLCLive Configuration
+
+### Prerequisites
+
+1. Exported DLCLive model (see DLC documentation)
+2. DeepLabCut-live installed (`pip install deeplabcut-live`)
+3. Camera preview running
+


include link for step 1:
see DeepLabCut documentation

step 2:
I think deeplabcut-live (with pytorch) should be a dependency that is automatically installed, do you agree?

deruyter92 · 2025-12-11T11:37:58Z

docs/user_guide.md

+1. Click **Browse** next to "Model directory"
+2. Navigate to your exported DLCLive model folder
+3. Select the folder containing:
+   - `pose_cfg.yaml`
+   - Model weights (`.pb`, `.pth`, etc.)


I was thinking, would be nice to have an option here to directly download superanimal models from the modelzoo.. What do you think? If you want I can make a follow-up PR for this.

Sounds great! Would need to run export models to dlclive format as well ! Also as many of superanimal models have detector we also need to fix this DeepLabCut/DeepLabCut-live#137

deruyter92 · 2025-12-11T11:54:32Z

pyproject.toml

+
+dependencies = [
+    "deeplabcut-live",
+    "PyQt6",
+    "numpy",
+    "opencv-python",
+    "vidgear[core]",
+    "matplotlib",
+]


There are many branches of deeplabcut-live and it has recently seen quite some updates. Since not everything is backward compatible, I think it will be important to pin a specific version here.

https://github.com/DeepLabCut/DeepLabCut-live/tree/maxim/dlclive3 this is the branch of dlclive I worked with

deruyter92 · 2025-12-11T11:57:14Z

pyproject.toml

+
+[project.optional-dependencies]
+basler = ["pypylon"]
+gentl = ["harvesters"]
+all = ["pypylon", "harvesters"]


Wouldn't it be useful to take over the optional torch and tensorflow dependencies of deeplabcut-live?

(These packages are exclusively used in the deeplabcut-live backend, and the backend specifies which torch/tensorflow work well).

i.e. to summarize my suggestions for pyproject.toml:

default dependency deeplabcut-live with pytorch

optional dependency deeplabcut-live with tensorflow

pin the required version of deeplabcut-live to branch dlclive3

dependencies = [ "deeplabcut-live[pytorch]", "PyQt6", "numpy", "opencv-python", "vidgear[core]", "matplotlib", ] [project.optional-dependencies] tensorflow = ["deeplabcut-live[tf]"] basler = ["pypylon"] ... [tool.uv.sources] deeplabcut-live = { git = "https://github.com/DeepLabCut/DeepLabCut-live.git", branch = "dlclive3" }

deruyter92 · 2025-12-11T17:41:29Z

setup.py

Shall we empty this setup.py, since everything is in pyproject.toml? Just leave something like this for backward compatibility:

from setuptools import setup # All configuration is now in pyproject.toml # This file is kept for backward compatibility with tools that don't support pyproject.toml setup()

deruyter92 · 2025-12-16T14:22:08Z

dlclivegui/dlc_processor.py

+
+            init_start = time.perf_counter()
+            options = {
+                "model_path": self._settings.model_path,
+                "model_type": self._settings.model_type,
+                "processor": self._processor,
+                "dynamic": list(self._settings.dynamic),
+                "resize": self._settings.resize,
+                "precision": self._settings.precision,
+            }
+            # Add device if specified in settings
+            if self._settings.device is not None:
+                options["device"] = self._settings.device
+            self._dlc = DLCLive(**options)


Note: options is missing a parameter single_animal. In the current DeepLabCut-live version 3 from maxim, this boolean parameter defaults to True. So even when a multi-animal model is running, DeepLabCut live is configured to treat it as a single-animal setup. The consequence is that for multiple pose outputs only the first one is selected.

We might choose to accept this behavior as convenient for now, while we think about the required adjustments for the multi-animal case. Meanwhile I think it would be good to include the argument single_animal: True here, to make it explicit here that you are launching DeepLabCut-live in single animal mode.

i have added single_animal=True as explicit parameter into the configs. once we add the option to actually deal with multi-animal predictions we can expose this parameter to be configurable.

- Update configuration settings to include support for single-animal models in DLCProcessorSettings. - Improve user guide with a link to DLC documentation for model export.

…le_animal in DLCLiveProcessor settings

arturoptophys and others added 28 commits October 21, 2025 11:22

Add modular camera backends with Basler and GenTL support

cc55285

Merge pull request #1 from arturoptophys/codex/update-repository-for-…

09abd98

…modern-python-and-pyqt6 Add Basler and GenTL camera backends for modular capture

Rearrange UI and improve camera controls

893b28a

modifyed gentl_backend

0b0ac33

Merge branch 'artur/test_update' into codex/update-gui-to-enhance-cam…

b13a417

…era-functionality

Improve camera control flow and recording alignment

8110085

Merge pull request #2 from arturoptophys/codex/update-gui-to-enhance-…

8e728e7

…camera-functionality Rework layout and camera handling controls

Improve camera stop flow and parameter propagation

efda9d3

Merge branch 'codex/update-gui-to-enhance-camera-functionality-xq83kx…

13bff62

…' into artur/test_update

fixed video recordings (roughly)

f39c4c4

updates

7e7e3b6

Added processors

6402566

update processors

c81e897

fixing sizes

c7ee2f1

modified the processor to be controllable

7823829

formatting and precommit workflow

f1ab8b0

documentations

39be1b2

update dlc_processor, fix filtering

ddcbc41

setting frame resolution

e2ff610

updated to look for model files

44de723

dropped tensorflow support via GUI

a5e70a6

more profiling of the processes

229a839

Add visualization settings and update camera backend for improved pos…

448bdc5

…e display

small bug fixes

246452f

black formatting

103399c

some docs

bcd89cc

updated docs

7b70c24

MMathisLab requested a review from Copilot December 5, 2025 16:35

MMathisLab assigned AlexEMG Dec 5, 2025

MMathisLab assigned deruyter92 Dec 5, 2025

Copilot started reviewing on behalf of MMathisLab December 5, 2025 16:36 View session

Copilot finished reviewing on behalf of MMathisLab December 5, 2025 16:37

Copilot AI reviewed Dec 5, 2025

View reviewed changes

Update dlclivegui/gui.py

94ccc7b

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

MMathisLab reviewed Dec 5, 2025

View reviewed changes

dlclivegui/cameras/base.py

return True

def stop(self) -> None:

"""Request a graceful stop."""

Copy link

Member

MMathisLab Dec 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:)

MMathisLab and others added 4 commits December 5, 2025 18:15

Rename DLC Processor to DeepLabCut Processor

c5659d7

Moved path2models as well as optional device and further DLCProcessor…

89cc84a

… options to config

remove path from docstring

edf6b80

fix exposure settings in gentl backend

37ca392

deruyter92 self-requested a review December 11, 2025 06:35

deruyter92 reviewed Dec 12, 2025

View reviewed changes

deruyter92 reviewed Dec 16, 2025

View reviewed changes

arturoptophys added 2 commits December 18, 2025 14:18

Add logging for camera configuration and error handling in backends

05ff126

- Update configuration settings to include support for single-animal models in DLCProcessorSettings. - Improve user guide with a link to DLC documentation for model export.

Set default device to 'auto' in DLCProcessorSettings and include sing…

5634059

…le_animal in DLCLiveProcessor settings

	os.environ["CUDA_VISIBLE_DEVICES"] = "1"
	# If you need to restrict CUDA devices, set CUDA_VISIBLE_DEVICES externally or via configuration.


		logging.basicConfig(level=logging.INFO)

		PATH2MODELS = "C:\\Users\\User\\Repos\\DeepLabCut-live-GUI\\dlc_training\\dlclive"

	PATH2MODELS = "C:\\Users\\User\\Repos\\DeepLabCut-live-GUI\\dlc_training\\dlclive"
	# Use environment variable DLC_PATH2MODELS if set, otherwise default to a relative path
	PATH2MODELS = os.environ.get("DLC_PATH2MODELS", str(Path(__file__).parent.parent / "dlc_training" / "dlclive"))

Updating DLCLive!-GUI to pyQt6 as well as DLC3/pytorch models #34

Are you sure you want to change the base?

Updating DLCLive!-GUI to pyQt6 as well as DLC3/pytorch models #34

Uh oh!

Conversation

arturoptophys commented Dec 5, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deruyter92 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants