Gk dev (ready for review!)

[MMathisLab]

starting PR for review; would be good to get this package in a sharable state soon! :)

[gkane26]

Alright, should be ready for review! Quick summary of notable changes:

• Added **kwargs to DLCLive.get_pose. You can now pass keyword arguments to processor objects -- this can be any data you want during a task so the processor can act accordingly. For example, you can pass record=True/False to let the processor know that the session has started so you don't start optogenetic stimulation before you actually start on the task!
• Created a new dlclive.analyze set of tools. dlclive.analyze does the following: i) benchmark inference times (dlclive.bench functionality has been moved here, but still exists just in case we need it), ii) display keypoints to visually inspect accuracy of exported models while using different resize parameters, and iii) analyze and create labeled videos using exported models - this operates just as deeplabcut.analyze_videos and deeplabcut.create_labeled_video, but you can run it with exported models and the exact resize parameters, etc that you want to run live experiments on (and it lets you use these functions without a full deeplabcut install). All of the dlclive.analyze features can be run on multiple videos in a single call.
• dlclive.analyze also comes with command line interface to all of these features.
• readme is updated to include instructions for using dlclive.analyze

I would like to get rid of/deprecate dlclive.bench in favor of dlclive.analyze. dlclive.analyze will perform the benchmarking exactly the same way bench did (including saving files in the same manner), but I think that using a dlclive.analyze_videos function will make it more consistent with deeplabcut and more accurately represents its full functionality now. I'm also happy to update analyze with the changes Jonny made to bench -- should definitely include those :)

[MMathisLab]

I would like to reserve analyze_videos for dlc, as it is a core function. I would be in favor of dlclive.benchmarking ?

[gkane26]

happy to change the name from analyze_videos, but it just feel weird naming the function that allows people to display keypoints/run inference and save labeled videos 'benchmarking'. If that's not a concern, I can change it though

[MMathisLab]

I agree it requires some discussion; my thoughts currently would be that the creating videos is a great thing to add (i.e. this is lighter weight than DLC and gives it some core inference functionality :) -- reminds me though, how it pcutoff set here?-- but I think it's confusing to have the same fxn name that requires diff inputs, i.e. path to model, vs. path to config, etc etc ... so I think it's good to highlight this as a option, but the main utility should be to use as a testscript of sorts, i.e. can you import the package and get "as advertised" speed?

Happy for others/your thoughts though :)

[gkane26]

the pcutoff here is set by the display_lik parameter -- I can change that to pcutoff throughout the package. I agree mostly that this will primarily be for testing, but analyzing videos may be really important for some experiments -- for the izzy experiment, I'm using it to get DLC keypoints for all frames, from the exact models used for live inference, to compare to tracking obtained live (which will drop frames, and is altered by using a prediction algorithm). For that reason, some form of analyze_videos could end up being a workhorse function. But if the main issue is just the name, the more I think about it, it seems less important -- as long its advertised it'll be hard to miss :)

[MMathisLab]

not sure if you already addressed these :)

[MMathisLab]

resolved

following as written in the README.md:

mwmathis@Mackenzies-MacBook-Pro DeepLabCut-live % pythonw
Python 3.7.6 (default, Jan  8 2020, 13:42:34) 
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dlclive
>>> dlclive.analyze_videos('/Users/mwmathis/Downloads/DLC_preycap_resnet_50_iteration-6_shuffle-1', '/Users/mwmathis/Desktop/preycap-mwm-2019-09-30/video-benchmark/J463b_top_cricket1_112119_2.avi', resize=0.5, display=True, pcutoff=0.5, display_radius=4, cmap='bmy')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mwmathis/Documents/DeepLabCut-live/dlclive/analyze.py", line 390, in analyze_videos
    pixels = [None for r in resize]
TypeError: 'float' object is not iterable
>>> dlclive.analyze_videos('/Users/mwmathis/Downloads/DLC_preycap_resnet_50_iteration-6_shuffle-1', '/Users/mwmathis/Desktop/preycap-mwm-2019-09-30/video-benchmark/J463b_top_cricket1_112119_2.avi', display=True, pcutoff=0.5, display_radius=4, cmap='bmy')

Run 1 / 1

[MMathisLab]

plotted points are not matching what is known from DLC analyze_videos: @AlexEMG this is the newly exported model, fyi.

same video:

[MMathisLab]

haha ...

[MMathisLab]

I think it's inverted, and rotated 90 degrees? is that possible?

[MMathisLab]

I also confirmed it's not just this model, i.e. with the pupil_vclose from the model zoo it's also incorrectly plotted:

[gkane26]

Are these models exported with TFGPUinference? There's something wrong with the way TFGPU inferefence models are being loaded by DLCLive, but I haven't been able to figure it out (there's seems to be an offset like you see in the pupil network). Exporting a numpy inference model should solve this -- need to make this clear in the readme

[MMathisLab]

All model zoos are exported this way, with the defaults from model_export, so this needs to work...

[MMathisLab]

It’s also not shifted, I confirmed it’s rotated and flipped.

[gkane26]

agreed, but I think I need another set of eyes on this... I've spent quite a bit of time on it and can't figure out why. I assume the issue is loading the network because things are working in Bonsai, right?

[MMathisLab]

seems bench.py is now obsolete, is this correct?

[MMathisLab]

@gkane26 - Gary if you can resolve issues in bench.py PR can be merged now :)

[gkane26]

I'm going to implement some changes from the new bench.py into analyze.py, quickly test, then should be good be good! At that point, bench will be deprecated... should we just delete it?

[gkane26]

I accepted pretty much all of the changes to bench.py from the master branch and manually incorporated the important changes from bench.py into benchmark.py (which was analyze.py). We should be able to get rid of bench.py now -- I was reluctant to keep it because this is the exact format that we were reading/plotting all of the current benchmark data from, but not it's changed substantially enough that we'd need to update plotting code anyways

[MMathisLab]

yeah I think dropping bench.py might be good; but let's merge this for now :)

[MMathisLab]

@gkane26 can you summarize basecamp discussion on plotting issue; also it was included in this PR no?

[MMathisLab]

Namely it was a plotting bug