Thunder on multiple cores

[RijadAlisic]

I've noticed Thunder only uses one core of the processor to analyze data - is there a way to run it on multiple cores locally?

[jwittenbach]

@RijadAlisic Are you trying to using "local mode" or are you trying to run against Spark?

[freeman-lab]

Assuming it's local, you can technically run Spark on a single machine, and leverage multiple cores, but running Spark locally also incurs a lot of overhead compared to the generic local numpy use, so I'm not sure it'd be worth it.

Looking ahead, we want to make dask an optional backend engine for thunder, alongside the current spark and numpy backends, and I think that will give a large boost to local multi-core performance.