Optional Language Detect

[gavishpoddar]

Implementing optional language detection and allow the creation of custom language detection libraries by users.

This work is currently in progress and this PR is created for the monitoring purpose by the maintainers to suggest progress.

This PR is currently not complete.

TODOS:

• Restructuring protect to pass function for language detection.
• Managing HTTP exception for fasttext model download.
• Updating setup.py for optional language detection.
• Mock test of parse, search_dates and DateDataParser.
• Unit tests for language detect functions.
• Functionality analysis of the implemented language detection.
• Testing optional language detection settings.
• Make apply_setting independent functions.
• Fasttext model download manager for models.
• Documentation optional language detection settings.
• Language Mapping (CLDR & ISO 639).
• Check working with other settings.
• Removing unsupported locale from language detecting
• Improving docs.
• Setting fasttext default language.
• Removing langauge_map.json and sinking to language_data.py
• langdetect set default DetectorFactory without changing global state.
• dateparser-download set default caching folder.
• Documenting dateparser-download.
• Changes for preventing breaking changes.
• detect_languages_func -> detect_languages_function

[lopuhin]

regarding adding dateparser/custom_lang_detect/data/lid.176.ftz - I'm not sure we should package this in dateparser even though it's not too big, but this is a minor thing which can be dealt with later. It's good to be able to download that automatically though.

[gavishpoddar]

HI @lopuhin, I am actually still developing and testing so I have not deleted it but will do that but in the end, I will clean codes and remove them.

[noviluni]

Hey @gavishpoddar good job! 💪

I left some comments. We will iterate over this code and probably the result will be quite different, but really good to see that you are getting results so quickly 🚀

Let me know if there's something you need any clarification on. 😄

[gavishpoddar]

Hi @lopuhin and @noviluni, I have made some changes can you suggest. I have also tried to cache the model without the global keyword instead used a class for the same.

I have started writing additional tests will try to complete them and cleaning up the code by Friday.

[noviluni]

Awesome job! ✔️ 🚀

Really happy to see this happening 🎉 😄

[lopuhin]

Thanks for the great PR @gavishpoddar , added some minor notes, I think it's basically ready to be merged.

[lopuhin]

Looks great, thanks @gavishpoddar 👍

[lopuhin]

@gavishpoddar congrats on the merge, great work 👍

@noviluni sorry that I didn't ask before merging, I hope a merge commit is fine, or would you prefer a squash and merge?

[gavishpoddar]

Thanks for the merge :)

[noviluni]

Hi @lopuhin, sorry for the late answer. I prefer to squash because in that way it is easier for me to build the changelog when releasing a new version, but no problem, it is fine :)

(Congrats Gavish! 💪 )