leave the scala-library out of the assembly

[michael72]

Hello!

Thanks again for your great spark plugin!
We had a few issues with our spark setup in the cluster and an assertion due to different scala versions being used in spark master (2.13.8) and driver application (2.13.17). This is not a usual or a general error though and it came from different pod versions being used.

However I found that the dataflint plugin (fat jar) also had additional scala library classes inside. I do not think that they are necessary though.
I changed the build.sbt myself - I am not sure how to actually build the fat jar for spark3 with scala_2.13. So I cannot test it. The resulting jar for scala_2.12 is ~1/3rd of the size of the original.

As an aside I would also recommend this great plugin to check for newer library versions since there are lot newer versions available (most of them could be changed - except probably aws libs):
https://github.com/rtimush/sbt-updates

Kind regards
Michael

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

Michael Schulte seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[michael72]

hum - so - the licensing signing doesn't seem to work / at the moment - although I authorized the application to do so 🤷‍♂️

but I hereby do aggree to the SAP Individual Contributor License Agreement, Version: 2026-01-07
Feel free to use my contribution

[menishmueli]

Already did it few days ago, will be added in the next version

[michael72]

@menishmueli OK - I only just stumbled on that setting and have been building assemblies for a couple of years now.
Anyway as it is dataflint is only usable for me in history server. Despite claiming otherwise spark-submit seems to build the classpath with the supplied packages first and then using the spark internal jars.
So it would be really good if you changed that! Thanks!

[menishmueli]

Yeah is a bit bizarre behavior, really appreciate you contributing to DataFlint
As for spark-submit - you can downgrade to earlier version for now (like 0.6.1) until we release the next version