CDJ2000nxs2 + DJM900nxs2 Wireshark trace files incl handshakes, glitches, crashes and beat link trigger operation

[Joren2087]

Hi all

Due to the success of STC on CDJ3000's last week (cfr #8 ), I last minute tried to run STC on a new show with older Pionneer hardware..

I got to use STC v1.9.9 on a show using 3x CDJ2000NXS2 + DJM900NXS2 gear. I was aware that STC is potentially unstable on this platform (cfr #6 ) and I unfortunatly experienced this first hand with numerous glitches, hanging cdj's and crashing STC. To preserve professional integrity, I had to abandon STC for beat-link-trigger (BLT) which worked flawlessly. However, I did manage to collect a lot of wireshark tracefiles to dig through when I had some time alone with the gear.

Be warned: it is a lot of tracefiles and probably contain a lot of noise, but I tried to name and seperate as much as possible. I also recorded some handshakes between the CDJs and beat-link-trigger (which worked flawlessly with accurate waveform, track metadata and beat/timecode information) without STC involved, to get a sense of what a proper handshake should look like.

Setup

• CDJ2000nxs2 firmware version: v1.87
• DJM900nxs2 firmware version: v2.07
• STC version: v1.9.9 (unless mentioned otherwise)

DHCP / MAC config:

Device	IPv4	MAC
ChamSys MQ50	2.0.0.2	x
STC Laptop	2.0.0.3	x
Player 1	2.0.0.9	c8:3d:fc:03:31:d2
Player 2	2.0.0.8	c8:3d:fc:03:53:2a
Player 3	2.0.0.7	c8:3d:fc:03:5f:30
DJM	2.0.0.10	c8:3d:fc:03:54:0c

It is possible there is some ArtNet noise in some tracefiles (coming from/to the mq50), but it is completely unrelated to STC and/or the crashes. Player 3 was also continuously autoplaying tracks as ambient music, so there might be some noise from that too. In early tracefiles player 1 is the USB source, but mostly it's player 2 that has the USB plugged in as music source.

Wireshark trace files: https://drive.google.com/drive/folders/1FlXRdtrigYOCjFDeNKqq-TU9aH76GgaT?usp=sharing

The issues

• I had STC to crash once
• Time information was very glitchy, jumping all over the place in STC but mostly increasing with random intervals. For reference, BLT had perfect tracking of time and tempo, so it's likely an STC issue rather than hardware limitation.
• With STC on, players often were unable to connect with others either throwing "Link Communication Error" or just hanging indefinetly.
• IF we got past the intial linking (eg when STC was not on during cdj boot, but booting stc afterwards), often a new booted player managed to get a link with others and STC, but waveform & audio would never load on the CDJ (just metadata) and any other operation after that would just hang the CDJ (the "waiting" loop, as shown here: CDJ-2000nxs2 metadata issue #6 (comment) ).
• Once I had a player go into emergency loop, probably due to the above issues

Notes on each tracefile

It was quite hectic to gather information and take notes. However, the more time I got to spend with the gear, the more I was able to eliminate noise, so the most recent captures are probably the most useful. As a troubleshooting step, I also switched to STC v1.9.5 and STC v1.9.8 but both had the exact same issues. I know one (or 2) of the tracefiles is with STC v1.9.5, but I can't remember which one so I'm afraid you'll have to guess based on STC behaviour (sorry for that!).

This is what I remember for each trace file:

Handshakes

• djm900nx2_handshake_no_stc - This is a link handshake between the DJM and other players without BLT or STC involved. I simply turned on the DJM and waited till it had a link connection.
• player1_proper_handshake_no_stc - Same as previous. No BLT or STC involved, simply turing on player 1 and recording until it has a link.
• djm_handshake_v1.9.9 - The DJM never showed up in STC - This is a recording between STC v1.9.9 and the DJM after STC or the DJM was turned back on. (I think STC was turned back on).

Beat-Link-Trigger Handshakes

Here no STC is involved, but it might be very valuable to see how BLT does handshakes (which then worked perfectly for the remaining 8h of DJ'ing). The "start" of each trace is BLT booting up (CDJs were already on and had a link between themselves)

• beat-link-trigger-handshake - If I recall correctly, this is a handshake between BLT and a player. Not really sure which one
• beat-link-trigger-player1-track-load - iirc, this is player 1 loading a track and BLT loading the waveform + track metadata. It probably also started playing, so beat information might be of value as well (as BLT had no jumps in time tracking / waveform tracking).
• beat-link-trigger-player1-non-rekordbox-track - It just happend the first DJ had tracks that were not in rekordbox, so I took a wireshark trace of that as well. It's not the cdj3000 as we discussed in our previous conversation but might be of great value as well. BLT was actually able to identify this and gave me a warning saying the track was a non-rekordbox track. Note that in this trace there is a fair bit of time between USB ingest and actually loading the track.

Crashes & Glitches

• player1_hang_once_first_track_loaded_never_waveform_on_cdj - Pretty much what it says. This was the biggest issue making STC unusable with nxs2 gear. The CDJ would hang after initial track metadata load (no waveform, no audio) and require a reboot.
• player1_link_error_unable_to_load_track_and_hangs - Similar / same issue as above iirc
• player1_emerg_loop - This was where player1 went into emergency loop (when it was already playing, iirc when the USB was still in player 1) - very likely related to a prodj link error causing it to bug out (even when not streaming audio/metadata over network).
• cdj2000nxs2_link_issue_v1.9.9 - One of the earlier and large tracefile (prob with artnet noise). It definetly has the same issue as described above.
• capture_cdj2000nxs2 - First tracefile, with artnet noise but all clean states. Iirc were CDJ+djm all on before I fired up STC v1.9.9. So initial link went fine, but then the CDJs started to behave very weird and glitchy, especially when loading tracks over prodj link. It also still has the old IPv4 addresses, so you'll have to look up the MAC addresses to see who is what player/djm.

Further investigation

Next week (fri-sat) I have a new gig scheduled for 2-3 days which will likely use the same cdj2000nxs2+djm900nxs2 setup (if not, its cdj3000+DJM A9 or similar). There is a fair chance I find time to play around with the gear and STC again, so it might be a good opportunity to gather information.
If there are any specific sequences / handshakes you would like to have a capture of, please let me know in advance such that I can be more strategic in testing & recording the gear.

I am a developer as well, and while I'm no where near familiar with JUCE (and know a bit of C++), I can imagine that having STC debug/logging information of some sort might be of great value. Therefore, if you were able to guide me into the right direction on how to set that up I might be able to debug STC in a live show with nxs2 gear and gather more information from that as well to report back here. Either way, I'll run the latest STC in the hope it is more stable.

If you're willing to, can you add me on Discord (joren2087) or similar for a quicker feedback loop than github issues?

I like STC very much and I would love to see it grow to its full potential hence I'm trying my best to help the (rather small) open source AV community where I can. I hope this report can help you too.

Thank you in advance !
If you have any questions, let me know!

Kind regards
Joren2087

[fiverecords]

Hi, thanks for the captures + the BLT side-by-side — that comparison was really valuable. Combining yours with the captures @trickofdjequip shared on issue #6 (different setup, overlapping symptoms) let me see the full picture: his case made one mechanism visible, yours made the other two, and the open-source NXS2 references (beat-link, python-prodj-link) confirmed both.

Three things stood out in your captures, all of which BLT happens to avoid:

1. STC opened 72 separate dbserver TCP sessions to the USB-holder NXS2 in ~120 s of player1_link_error_unable_to_load_track_and_hangs.pcapng — one fresh connection roughly every 1.7 s, for the same loaded track. Root cause: STC's connection pool was closing idle sockets after just 1 second. python-prodj-link (Fabian Lesniak's reference, explicitly tested against CDJ-2000NXS2 + DJM-900NXS2) keeps each socket for 30 s and resets the TTL on every use.

2. Zero NFS traffic to the players in any of your captures, even though STC ships the equivalent of beat-link's CrateDigger (NFSv2-based analysis file fetch). The existing NFS path was being suppressed because dbserver phase-2 queries were filling the cache flags with unusable-but-non-empty data, so the fallback that should have run never did. beat-link documents this exact failure mode: NXS2 dbserver waveform queries are unreliable, and CrateDigger over NFS is the canonical workaround.

3. STC was reading NFS files in 8 KB chunks. NFSv2 allows that, but an 8 KB UDP datagram crosses the 1500 B Ethernet MTU and gets sent as ~6 IP fragments. On a multi-NIC laptop like yours (two IPs broadcasting Art-Net at ~260 packets/s combined, plus three NXS2 plus the DJM-900NXS2 on the same switch) fragment reassembly is exactly the kind of thing that gets unreliable. python-prodj-link uses 1280 B reads specifically to avoid this.

Attached is STC v1.9.10-beta-Win.zip with five surgical changes.

STC v1.9.10-beta-Win.zip

The three relevant to you:

• Track analysis (waveform, beat grid, cues, song structure) now goes through NFS on NXS2, matching beat-link's CrateDigger approach.
• Connection idle timeout raised from 1 s to 30 s, matching python-prodj-link.
• NFS read chunk size lowered from 8192 B to 1280 B, also from python-prodj-link.

Honest caveat: I don't have NXS2 hardware to test against. The fixes are inferred from your captures + trickofdjequip's captures + a code-level audit of three open-source PRO DJ LINK implementations that work on this gear. The mechanism is well-grounded on the wire but I can't prove the fix end-to-end without you running the beta.

From the TCP fingerprints in your captures (TTL 128, window 65535, classic Windows option pattern) you're on Windows — attached you'll find a built 1.9.10-beta installer/exe.

If you have time:

• Load 2-3 tracks across the players. Do the waveforms appear on the CDJs and does audio play normally? That's the main test.
• If yes, a short Wireshark capture (a couple of minutes of normal use) would let me confirm two things: NFS traffic now visible (filter udp.port == 2049) and dbserver reconnects down by ~50x (filter tcp.dstport == 1051).

One thing this beta does NOT address: the DJM-900NXS2 intermittent discovery you also reported. I haven't touched DJM-facing code at all in this round — the last DJM model fully confirmed working in STC is a DJM-A9, so the 900NXS2 path is somewhat unknown territory. If that issue persists after this beta lands the CDJ side, we'll pick it up as a separate thread.

Thanks again — your BLT-vs-STC comparison was a big part of what made this debuggable.

[Joren2087]

Amazing!

I'll run the beta on Friday with the gear. I'll likely have some spare time with it so I'll try to get your captures and post them here.

Also well spotted: I'm on windows 11 indeed. My laptop indeed has a wireless NIC and and ethernet one (USB dongle), although only the Ethernet one is used on the local network.
I'm also running a DHCP server on my laptop to get the cdjs a different IP in the 2.x range rather than link local stuff.

Thanks for the quick reply and debug!