Compatibility Between DeepFilterNet Repos and JS Worker Implementation

[jozefchutka]

Hi @grazder , thank you for your great work on https://github.com/grazder/DeepFilterNet also using onnx_worker_sharedarraybuffer works like a charm.

I am trying to get my head around how to get the most recent version running in javascript. Is my understanding correct that:

1. Original https://github.com/Rikorose/DeepFilterNet is now ahead of yours https://github.com/grazder/DeepFilterNet (This branch is 29 commits ahead of, 166 commits behind Rikorose/DeepFilterNet:main.)
2. And your https://github.com/grazder/DeepFilterNet is also ahead of implementation in https://github.com/grazder/samejs/blob/master/deepfilternet3/onnx_worker_sharedarraybuffer/worker.js considering differences in inputNames (input_frame, states, atten_lim_db vs input_frame, erb_norm_state, band_unit_norm_state, ...)

If so, are there any plans to update the current worker.js implementation to work with the latest denoiser_model.onnx from DeepFilterNet3_torchDF_onnx.tar or are there any .js implementations you are aware of?

[grazder]

Hi! I don't have such plans right now. It seems like original repo has not have any major updates since my fork.

What about DeepFilterNet3_torchDF_onnx.tar - worker.js should work with our new checkpoint, but you should change input names, its shapes and initializations, yes

[jozefchutka]

Thanks for the quick response. I would like to handle updates on my side but I am being lost.

In the existing worker.js there are 3 inputs: input_frame, states, atten_lim_db... seems obvious how its being used and what each stand for.

Now in the DeepFilterNet3_torchDF_onnx.tar there are 13 inputs:
input_frame, erb_norm_state, band_unit_norm_state, analysis_mem, synthesis_mem, rolling_erb_buf, rolling_feat_spec_buf, rolling_c0_buf, rolling_spec_buf_x, rolling_spec_buf_y, enc_hidden, erb_dec_hidden, df_dec_hidden and I am having hard times understand how are these supposed to be used even following https://github.com/grazder/DeepFilterNet/blob/408414f2bdb9ed772580558d1c3ec26c5dc002d7/torchDF/torch_df_offline.py . Can you please share a bit knowledge here?

[grazder]

https://github.com/grazder/DeepFilterNet/blob/408414f2bdb9ed772580558d1c3ec26c5dc002d7/torchDF/torch_df_streaming_minimal.py#L1025

You can look at this class and repeat how the input handling is done but on JS. In constructor you can find all the initial values. And then you simply substitute the values ​​from the previous inference for each next launch.

[jozefchutka]

@grazder thanks for the heads up.

I managed to put together minimal working script using .onnx file from https://github.com/grazder/DeepFilterNet/blob/torchDF_main/models/DeepFilterNet3_torchDF_onnx.tar .

Can you please have a look and let me know if anything obvious is wrong/missing?

I have also noticed that atten_lim_db that existed in the previous .onnx model, is not available anymore. I used it as denoising "strength", is any of the current inputs available for such feature?

<script src="https://cdnjs.cloudflare.com/ajax/libs/onnxruntime-web/1.22.0/ort.webgpu.js"></script>
<script type="module">
const session = await ort.InferenceSession.create("./denoiser_model.onnx",
	{executionProviders: ["wasm"]});

// ffmpeg -i input.wav -filter_complex:a "[0:a]aformat=channel_layouts=mono,aresample=48000[aout]" -map "[aout]" -c:a pcm_f32le -f data input.pcm -y
const channelData = new Float32Array(await (await fetch("input.pcm")).arrayBuffer());

const hopSize = 512;
const fftSize = hopSize * 2;
const origLen = channelData.length;
const padSize = (hopSize - (origLen % hopSize)) % hopSize;
const padded = new Float32Array(origLen + fftSize + padSize);
padded.set(channelData);

const tensor = (shape, buffer) => new ort.Tensor("float32", buffer ?? new Float32Array(shape.reduce((a, b) => a * b)), shape);

const states = {
	erb_norm_state: tensor([32],
		Float32Array.from(Array.from({length:32}, (_, i) => -60.0 + (i * (-90.0 + 60.0)) / (32 - 1)))),
	band_unit_norm_state: tensor([1,96,1],
		Float32Array.from(Array.from({length:96}, (_, i) => 0.001 + (i * (0.0001 - 0.001)) / (96 - 1)))),
	analysis_mem: tensor([hopSize]),
	synthesis_mem: tensor([hopSize]),
	rolling_erb_buf: tensor([1,1,3,32]),
	rolling_feat_spec_buf: tensor([1,2,3,96]),
	rolling_c0_buf: tensor([1,64,5,96]),
	rolling_spec_buf_x: tensor([5,(hopSize+1),2]),
	rolling_spec_buf_y: tensor([7,(hopSize+1),2]),
	enc_hidden: tensor([1,1,256]),
	erb_dec_hidden: tensor([2,1,256]),
	df_dec_hidden: tensor([2,1,256])}

const outputFrames = [];
for(let i = 0; i < padded.length; i += hopSize) {
	const input_frame = tensor([hopSize], padded.subarray(i, i + hopSize));
	const result = await session.run({input_frame, ...states});
	outputFrames.push(new Blob([result.enhanced_audio_frame.data]));
	for(const key of Object.keys(states))
		states[key].data.set(result[`new_${key}`].data);
}

const d = fftSize - hopSize;
const output = new Blob(outputFrames).slice(d * 4, (d + origLen) * 4);
// ffmpeg -f f32le -ar 48000 -ac 1 -i output.pcm output.wav -y
</script>