added conditional import and buffer widget

[alnitak]

I have added a conditional import to make working with the web and other platforms easy.

Added also a buffer widget that lets you see the amount of memory and the total time while receiving the data.

I tried with both Chrome and Linux and I do not have problems with the huge string. Looking at the buffer, the audio is about 20.7 MB.

I see you are using fvm with Flutter v3.24. Maybe it is worth trying to upgrade it? I do not have any other clue regarding that at your end it doesn't complete all the audio! I don't think the flutter_soloud BufferStream has problems handling so much data, I have tested with a lot more and faster streams (locally).

Let me know your findings!

[callmephil]

Thanks for the PR that's a great addition!
tho 20.7 MB for a non-hd audio file sounds high. the network tab says it's 10.9MB; I guess there's a miss calculation somewhere.

My issue with the audio might be related to unstable/slow network PCM chunks are quite big actually... in comparison, an opus output will be around 1.1MB and AAC 1.4MB

Do you think the Soloud package will be able to support those for streaming in the future?

[alnitak]

You are right! Almost I forgot that the plugin only uses floats 32 format for the audio output. So in whatever format you set the StreamBuffef, it is then converted to 4 bytes floats. In this case, the 2 bytes of the s16le is converted to 4 bytes for float. Hence the buffer size is displayed correctly and also the downloaded size.

Do you think the Soloud package will be able to support those for streaming in the future?

I think yes for the Ogg and Wav uncompressed, I have to look closer for the others.

[callmephil]

You are right! Almost I forgot that the plugin only uses floats 32 format for the audio output. So in whatever format you set the StreamBuffef, it is then converted to 4 bytes floats. In this case, the 2 bytes of the s16le is converted to 4 bytes for float. Hence the buffer size is displayed correctly and also the downloaded size.

Do you think the Soloud package will be able to support those for streaming in the future?

I think yes for the Ogg and Wav uncompressed, I have to look closer for the others.

Okay, I see so we should divide the results by 2?

Regarding the codecs, Ogg isn't supported on iOS/Safari natively so the best second option for speed would be AAC

[alnitak]

Okay, I see so we should divide the results by 2?

In this case yes.

Regarding the codecs, Ogg isn't supported on iOS/Safari natively so the best second option for speed would be AAC

its support will be coded inside the flutter_soloud plugin, doesn't matter if it isn't supported by the OS.

[alnitak]

Hey @callmephil,

I have added Opus support. I have added Xiph Ogg and Opus libraries for this.
No Web support yet, I don't know how to compile the libraries yet.

I tested on Linux and Android.

I couldn't test on MacOS because I use KVM which doesn't support Metal, but it compiles and should work. Same for iOS, but I cannot compile because the simulator doesn't start!! :)

Anyway, the support for Opus/Ogg is in the opus branch of flutter_soloud within this PR. If you have any questions/issues, please write there!

I have also updated this PR to work with Opus.

[callmephil]

You are awesome man!

I tried it for iOS, but it doesn't find the library. I'll try to debug that and let you know.

Could not build the precompiled application for the device.
Error (Xcode): Undefined symbol: _ogg_page_serialno
Error (Xcode): Undefined symbol: _ogg_stream_clear
Error (Xcode): Undefined symbol: _ogg_stream_init
Error (Xcode): Undefined symbol: _ogg_stream_packetout
Error (Xcode): Undefined symbol: _ogg_stream_pagein
Error (Xcode): Undefined symbol: _ogg_sync_buffer
Error (Xcode): Undefined symbol: _ogg_sync_clear
Error (Xcode): Undefined symbol: _ogg_sync_init
Error (Xcode): Undefined symbol: _ogg_sync_pageout
Error (Xcode): Undefined symbol: _ogg_sync_wrote
Error (Xcode): Undefined symbol: _opus_decode_float
Error (Xcode): Undefined symbol: _opus_decoder_create
Error (Xcode): Undefined symbol: _opus_decoder_destroy
Error (Xcode): Undefined symbol: _opus_strerror
Error launching application on Philippe’s iPhone.

btw I think the simulator didn't start because it was missing the iOS config.

[callmephil]

On Android, it's working great I also noticed that we don't need to create bigger chunks so we can return the stream as is.

[alnitak]

On Android, it's working great I also noticed that we don't need to create bigger chunks so we can return the stream as is.

yes, that's ok for Opus, but for PCM a buffer is still needed

btw I think the simulator didn't start because it was missing the iOS config.

what is "iOS config"? (sorry I am not much familiar with Apple world)

Regard the error, in the xiph folder there is a script to compile iOS and Mac libs (opus and ogg) if you want to try to recompile them. But maybe that is not the problem. The ios/flutter_soloud.podspec could be the problem, don't know.

[callmephil]

So I couldn't get it to work on iOS, I've cloned your branch and re-built the pods on my Mac using sudo on build_iOS_OSX.sh I've also tried importing the compiled output directly in Xcode but it didn't work either.

If you have some time next week maybe we can have a pair session and see if we can get somewhere or perhaps I can provide you with access to a Mac in remote so you can play around whenever you feel like.

FYI for iOS config, it's the iOS folder generated by Flutter when you create a project on Mac similar to the Android and Linux folders.

[alnitak]

Sorry for the late reply, but you inspired me to try Hackintosh again. It took me days, but I finally did it!
I still have to set up the whole development environment, but I think I'll update you soon! And thank you very much for your offer!!

[callmephil]

Sorry for the late reply, but you inspired me to try Hackintosh again. It took me days, but I finally did it! I still have to set up the whole development environment, but I think I'll update you soon! And thank you very much for your offer!!

Thanks to you for your support and contributions! Let me know If I can help with anything :)

[alnitak]

Hey Phil, good news. I have modified the script to compile the libs. Now instead of making a "fat lib" for both device and simulator, it splits them and links according. I tried on iOS and MacOS and now I can tell that I tested :)
So, on iOS the opus and ogg libraries are embedded. On MacOS (and Linux) you should install them with maybe brew install libogg opus. If not installed it compiles anyway but it will throw if you want to use the opus streaming feature.

Please let me know of any problems!

[callmephil]

@alnitak I made a PR for the iOS config and I'll add an extension to be able to switch between custom chunking for pcm. If you want to pull from this branch, so we can stay on the same page.

#3

btw it works wonderfully, I need to do some benchmarks, add a cancelation mechanism and test is conditions where the internet is poor.

[callmephil]

-- Update --

1. I've implemented the cancel mechanism for both HTTP and stream.
2. I've moved the bytes increase from the HTTP requests to an extension and added a conditional for the web to use PCM instead (currently doesn't compile on opus branch)
3. I've done some code cleanup

[alnitak]

Great job!

If I understand well, you can now close the other PR without merging and we will continue here?

Ok, I am now aligned and I tried successfully on Linux. FYI I already started coding the opus support for the web, but since it is a bad beast, maybe it will take a while :)

Since I use BufferBar also elsewhere, may I ask what is not working well? I thought that could be useful to know how much data is added to the audio aside from the amount of RAW memory used. Could it be useful?

[callmephil]

Since I use BufferBar also elsewhere, may I ask what is not working well? I thought that could be useful to know how much data is added to the audio aside from the amount of RAW memory used. Could it be useful?

So far, for the audio player it seems stable! I'll do some benchmarks as soon as possible and start reviewing your Soloud branch for Opus. Perhaps I might be able to help.

Regarding the buffer bar I'm not sure what's not working but It seems that the data sounds off. I'll investigate that as well.

btw I've noticed OpenAI is counting the whole string despite requesting a stream. I'm trying to monitor the pricing and see if we can optimize the request somewhere...

[alnitak]

Hey @callmephil,

I've merged your opus_ios_config into this.

Finally I got opus working also on the Web! The only thing is that I hear clicks when fast adding data at maybe 10 MB/s. Don't know well why. Can you confirm this?

I didn't try with this project to save OpenAI counting but it should work fine.

Regarding flutter_soloud for the Web, it now needs another script to add to the index.html. I have also updated it.

Let me know!

[callmephil]

@alnitak I've just tested the branch and it seems to work great for Opus files.

PCM broke on my end, probably because of the changes I've made to the chunking mechanism, I'll look into it.

Make sure you disable the usePCM at line 41

bool get usePCM => false;

I'll keep trying different scenarios and let you know if there's anything wrong.

[alnitak]

I didn't notice problems using PCM, anyway I have added a buffering natively (hopefully correctly :) ). So I think there is no more need to use the chunking extension.