Support deduplication of various formats #5
Conversation
deduplication/args.py
Outdated
| ) | ||
| parser.add_argument( | ||
| "--skip-insertion", | ||
| help="If set, will skip inserting entries to index. THis is a MOCK ARG", |
There was a problem hiding this comment.
nit on capitalization here :)
| case _: | ||
| raise NotImplementedError(f"{p.suffixes[-2:]} is not a supported filetype") | ||
|
|
||
| def compute_minhash_for_anyfile(infile: str, output_dir: str, num_perm: int): |
There was a problem hiding this comment.
Since this covers all filetypes, perhaps we should delete compute_minhash_for_file below and refactor our minhasher to use this function instead
There was a problem hiding this comment.
I wasn't sure if this was used else where or upstream or if this was something you wrote. If you wrote this compute_minhash_for_file, I'll remove/refactor it.
There was a problem hiding this comment.
Yeah I think we can refactor this here. The only other thing to note is that the MinHasher class has another compute_minhash_for_file function that needs to be updated to reflect the new change - that MinHasher method is used elsewhere in workflows.py.
|
There are several changes from me that need cleanup here; I will address these as soon as possible. |
…ents * When `skip_insertion` is enabled, unique entries found are not inserted into the index. This enables deduplicating against an index without modifying it.
|
@123epsilon @robertu94 Is there anything that's pending here that's blocking? |
|
@yadudoc @robertu94 Not to my knowledge - I am partial to merging #7 first because its small, there's some overlap, and its mostly minor bugfixes |
3 participants
@123epsilon and @yadudoc here are the changes to support various formats we can review when we chat next week.