Specify or at least document the expectations for merging sets

[gouttegd]

The spec should say something about what to expect when two (or more) sets are merged together.

It does not necessarily have to mandate a given behaviour, especially since I don’t think there would be a single merging behaviour that would fit all use cases. But it should at least describe what the possible behaviours are, and possibly recommend one as the “most sensible” and suitable as a default behaviour.

Some of the questions that arise when merging sets together:

About the individual mappings

• Should duplicated mappings be dropped? If so, how to detect duplicated mappings?

• What to do if two mappings have the same record_id? (Note that this is different question from the previous one: two mappings may have the same record_id and yet be different – within a single set this is not supposed to happen, but this could very well happen when merging several sets.)

• What to do if mappings from one set have record_id and mappings from the other set don’t? Drop all record_id from all mappings? Automatically generate record_id for the mappings that don’t have one? Do nothing (which would result in an invalid set, since mixing mappings with a record_id and mappings without such a record is not allowed)?

About the set metadata

• Which metadata to use for the final, merged set?

Possible options:

• Use only the metadata from one of the merged sets (which one?).

• For any given slot, use the value from the first set if present, otherwise the value from the second set.

• For any given slot, only set the value in the resulting set if both input sets have the same value, otherwise do not set the value at all.

• For any multi-valued slot, if both input sets have a different list of values, merge the values together in the resulting set.

Existing behaviours

FWIW, the current behaviour of SSSOM-Java’s sssom-cli tool is:

• merging all mappings without any attempt at dropping duplicates or fixing record_id (though users have the possibility of doing that themselves using some SSSOM/T rules);
• for single-valued set metadata slots, using the value from the first set only, whatever that value is (including no value);
• for multi-valued set metadata slots, merging the values from all sets, unless the --no-metadata-merge option is used (in which case the behaviour is the same as for single-valued slots: only the values from the input set are used).

I have not looked yet at what sssom-py does.

[matentzn]

Should duplicated mappings be dropped? If so, how to detect duplicated mappings?

I would say mappings identical in the sense of #436 should be dropped, else no, no default dropping.

What to do if two mappings have the same record_id? (Note that this is different question from the previous one: two mappings may have the same record_id and yet be different – within a single set this is not supposed to happen, but this could very well happen when merging several sets.)

IMO if the mappings are not identical and the record id is the same, the merge process should fail?

What to do if mappings from one set have record_id and mappings from the other set don’t? Drop all record_id from all mappings? Automatically generate record_id for the mappings that don’t have one? Do nothing (which would result in an invalid set, since mixing mappings with a record_id and mappings without such a record is not allowed)?

Hmm. Maybe this:

1. Fail because merged set is invalid
2. Tools can choose to provide option for generating record_ids

? Not sure.

Which metadata to use for the final, merged set?

Difficult question. Minus concerns around condensation and propagation I would do it like this (this is NOT at all how sssom-py does it):

• By default, do not copy any metadata, generate a mapping_set_id and a unknown license
• Set mapping_set_source values with mapping_set_id values from merged sets
• Perhaps add a comment or boilerplate description that the mapping set is the consequence of merging A, B, C

Use only the metadata from one of the merged sets (which one?).

I would not include any mapping set specific metadata; however, tools can choose to implement an option like "--merge-metadata (first|all)" or some such.

For any given slot, use the value from the first set if present, otherwise the value from the second set.

Reasonable but I would not copy metadata by default probably? Maybe I am wrong.

For any given slot, only set the value in the resulting set if both input sets have the same value, otherwise do not set the value at all.

For any multi-valued slot, if both input sets have a different list of values, merge the values together in the resulting set.

[matentzn]

In any case just to preclude any comments on making things unnecessarily hard: I am fine with your default suggestions as well!

[gouttegd]

[…] Maybe I am wrong.

To be clear I think that for most of these questions there may not be “right” or “wrong” answers. You may want one particular behaviour for a given use case and another behaviour for another use case.

Best we can do, I believe, is to (i) document what the possible behaviours can be, and (ii) possibly suggest a “sensible” default behaviour (the behaviour that, ideally, most users would expect). After that it is left to implementations to decide what to do.

Should duplicated mappings be dropped? If so, how to detect duplicated mappings?

I would say mappings identical in the sense of #436 should be dropped, else no, no default dropping.

The standard hashing procedure proposed in #436 would be one way of checking for identical mappings, but it may be slightly overkill here. The point of that procedure is to be able to compare mappings accross implementations – this is not needed here: the merging is happening inside a single implementation. Internally, the implementation may have a faster and more efficient way of comparing two mappings, and there would be no reason to use it here. (Of course if the implementation does not have a faster method to check for identical mappings, then it’s perfectly fine to use the standard hashing procedure.)

I think what is important here is to clearly define what we mean by “identical mappings”: two mappings are identical if all their slots are identical, excluding the record_id and the mapping_cardinality slots. For multi-valued slots, they are considered identical if they contain the same values, regardless of their order.

Also, not dropping duplicated mappings should in my opinion always be an option. In almost all of my own use cases, I actually don’t care that there may be duplicated mappings.

What to do if mappings from one set have record_id and mappings from the other set don’t? Drop all record_id from all mappings? Automatically generate record_id for the mappings that don’t have one? Do nothing (which would result in an invalid set, since mixing mappings with a record_id and mappings without such a record is not allowed)?

Hmm. Maybe this:

1. Fail because merged set is invalid
2. Tools can choose to provide option for generating record_ids

? Not sure.

This is a typical question where I think all possible behaviours (drop all record_id slots; generate record_id slots for the records that don’t have one; do nothing; fail) can be equally useful depending on the use case.

Not sure what the “sensible” default should be, but I think I’ll err on the side of “by default, fail because merged set is invalid”. Simply because it doesn’t look like a good idea for an implementation to allow the creation of an invalid set by default.

Which metadata to use for the final, merged set?

Difficult question. Minus concerns around condensation and propagation

If the implementation supports condensation and propagation, there should be no concern. Propagation should always happen just after a set is read, before any merging operation can be attempted.

So if you are merging two sets A and B that may contain condensed slots: (1) you read A, (2) you propagate all condensed slots in A, (3) you read B, (4) you propagate all condensed slots in B, and finally (5) you merge A and B. By the time you reach (5), boths sets are already in their propagated form, so you will not lose any mapping metadata regardless of what you decide to do with the set metadata.

Of course if the implementation does not support propagation, then you very much risk losing information if the sets you have to merge happen to contain condensed slots. Nothing much we can do about that, apart maybe from warning users not to use condensed slots if they know their sets are going to be processed by an implementation that can’t propagate them. (It’s not for nothing that I wanted support for propagation to be mandatory instead of merely recommended.)

I would do it like this (this is NOT at all how sssom-py does it):

• By default, do not copy any metadata, generate a mapping_set_id and a unknown license
• Set mapping_set_source values with mapping_set_id values from merged sets
• Perhaps add a comment or boilerplate description that the mapping set is the consequence of merging A, B, C

Sounds good, except for the comment part, which I believe should not be part of the default.

FWIW, with SSSOM-Java’s sssom-cli I frequently use the following pattern:

sssom-cli dummy.sssom.tsv set1.sssom.tsv set2.sssom.tsv set3.sssom.tsv --no-metadata-merge -o merged.sssom.tsv

where setN.sssom.tsv are the sets that I want to merge, and dummy.sssom.tsv is an empty set that contains only a metadata block with the metdata that I want for the output set:

#mapping_set_id: https://example.org/my-merged-set.sssom.tsv
#license: https://creativecommons.org/licenses/by/4.0
#mapping_set_description: This is my merged set.
subject_id  predicate_id    object_id   mapping_justification

Since sssom-cli takes the metadata of the first set as reference, those metadata will end up becoming the metadata for the final merged set, ignoring all the metadata from the real sets that I am merging.

[matentzn]

To be clear I think that for most of these questions there may not be “right” or “wrong” answers.

Yes I think that way too

I think what is important here is to clearly define what we mean by “identical mappings”: two mappings are identical if all their slots are identical, excluding the record_id and the mapping_cardinality slots.

I guess we are having this discussion in #436 no need to re-hash here. :P

Since sssom-cli takes the metadata of the first set as reference

I think sssom-py does this as well but I still think its best to not by default copy over any metadata; there are of course use cases for copying over (especially if you want to use the merge command to merge new mappings into an existing mapping set) but mostly I think you don't want to blindly copy over the metadata. The licenses wont stack right; the creator will probably be different, as will any date related information (and many more).

Minus concerns around condensation and propagation

When I set "minus" I meant (sorry I should have been clearer) that I am assuming a canonical representation here (i.e. the minus means - "ignoring any concerns around propagation and condensation".

I think we are mostly on the same page. I would also like to promote now sssom-java as the reference implementation as its already way more complete then sssom py.

[gouttegd]

I guess we are having this discussion in #436 no need to re-hash here. :P

Agreed. In fact thinking back about this, I think the best way here is to simply say, “two mappings are considered identical if the standard SSSOM hashing procedure (as defined in ...) produces the same hash for them.”

This is unambiguous but still leaves room for implementation to have their own internal mapping comparison function independently of the hashing procedure if they so want for any reason (e.g. for efficiency reason).

I think sssom-py does this as well but I still think its best to not by default copy over any metadata […]

Noted, and that sounds reasonable indeed.

[gouttegd]

I would also like to promote now sssom-java as the reference implementation as its already way more complete then sssom py.

No objection as you know, but I hope you’ll understand that I’d rather stay out of any discussion about that (and I’d rather not be assigned to review any PR that implements that change).