HDDS-13311. Directory Deleting Service can use deleteRange for subDirectories and subFiles

[aryangupta1998]

What changes were proposed in this pull request?

DirectoryDeletingService should use rocksdb deleteRange instead of creating individual tombstones which can cause seek time issue. But in the presence of snapshots the deleteRange APi should stitch continuous key ranges together that are reclaimable and not issue a blind deleteRange which could lead to incorrect reclaimation of the entry and lead to unreference orphan blocks when the snapshots are deleted.

DeleteRange APIs on FileTable, DirectoryTable, KeyTable can be used by background garbage collection services and should never be used by user facing APIs like keyDelete as that can cause issues in snapshot correctness.

E.g.

Dir1/Key1(Reclaimable) Dir1/Key2(Reclaimable) Dir1/key3(Not Reclaimable) Dir1/Key4(Reclaimable) Dir1/Key5(Reclaimable)

Then DirectoryDeletingService should issue 2 delete range like
[Dir1/Key1..Dir1/Key2] (Both inclusive)
[Dir1/Key4..Dir1/Key5]
In terms of rocksdb deleteRange where the end key range is exclusive this would be equivalent to
[Dir1/Key1..Dir1/Key3) and [Dir1/Key4..lexicographicalHigherString(Dir1/)]

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13311

How was this patch tested?

Testes via UT.

[swamirishi]

thanks for the patch @aryangupta1998 Change this one fix I am reviewing other parts in the meanwhile

[adoroszlai]

Please wait for clean CI run in fork before opening PR.

[swamirishi]

@aryangupta1998 please add unit tests for KeyManagerImpl and DirectoryDeletingService changes

[swamirishi]

@aryangupta1998 Don't make it ready for review until you get a +1 from reviewers

[jojochuang]

@rnblough @ptlrs

[sumitagrawl]

@aryangupta1998 It seems not feasible to use delete range, as parallel new item can get added in middle within range, and can delete those unexpected entry.
cc: @swamirishi

[swamirishi]

Please add unit tests with mocking layer

[swamirishi]

Why a separate test class?

[swamirishi]

Please write the test in TestKeyManagerImpl for the function testGetPendingDeletionSubFiles & SubDirs in the class hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestKeyManagerImpl.java

[swamirishi]

And use mocks to precisely figure out all corner cases in the loop testing each and every if condition in the function gatherSubPathsWithIterator

[swamirishi]

Similarly write a test in directoryDeletingService which tests out rrequest submitted to ratis by mocking all layers KeyManagerImpl, ReclaimableFilter just testing out DirectoryDeletingService functionalities.