OAK-12089: Lucene 4->5 upgrade

[bhabegger]

Experimentation

The goal of this PR is not necessarily to be merged, but rather an attempt to see if we can leverage AI to help us improve the code base

Attempt to upgrade Lucene from 4.x to 5.y keeping existing index segments using 4.x codecs (oak or otherwise) readable. New segments are written using 5.x codec.

A new version of the oak specific codec is introduced.

This code base was mostly obtained by carefully piloting Augment step by step.

[reschke]

@bhabegger - could you please create a ticket and link your PR to that ticket?

[reschke]

Wow; this is ... hard ... to review.

Any chance to break this down into smaller pieces; right now this touches 300 files.

[bhabegger]

Wow; this is ... hard ... to review.

Any chance to break this down into smaller pieces; right now this touches 300 files.

Well most of them are the deletion of the embedded lucene version. As it's a migration that requires some adaptations, I don't really know how I could split it, but I'll have a look.

@bhabegger - could you please create a ticket and link your PR to that ticket?

For the moment it is just a trial draft. I created the PR to have the pipeline run and check if any thing fails. But if and when I feel it's worth it, I will indeed create a ticket. Unless you believe it is worthwhile in any case ?