Duplicated rows of TSV is unclear cause of TAG_EXPRESSION_REPEATED

[CodyCBakerPhD]

From bids-standard/bids-validator#326 (comment)

git clone https://github.com/bids-dandisets/000003

hedpy validate-bids 000003

Finds zero issues, but the BIDS validator returns a problem involving repeated HED tags, which actually occurs because of duplicated rows in a TSV. These appear as

/sub-YutaMouse57/ses-YutaMouse57-161010/ecephys/sub-YutaMouse57_ses-YutaMouse57-161010_events.tsv - ERROR: [TAG_EXPRESSION_REPEATED] Duplicate tags - "[Experimental-trial] in "Experimental-trial,Experimental-trial".  (For more information on this HED error, see https://hed-specification.readthedocs.io/en/latest/Appendix_B.html#tag-expression-repeated.)
/sub-YutaMouse42/ses-YutaMouse42-151102/ecephys/sub-YutaMouse42_ses-YutaMouse42-151102_events.tsv - ERROR: [TAG_EXPRESSION_REPEATED] Duplicate tags - "[Experimental-trial] in "Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial,Experimental-trial".  (For more information on this HED error, see https://hed-specification.readthedocs.io/en/latest/Appendix_B.html#tag-expression-repeated.)

inside the BIDS validator

I would also request that the error message be more informative to the cause - given that the problem was obvious after discovery of duplicated rows in the TSV

[VisLab]

@CodyCBakerPhD
Could you clarify what error the python validator is returning that is causing the problem? On the TAG_EXPRESSION_REPEATED both validators provide a details link to: https://www.hedtags.org/hed-specification/Appendix_B.html#tag-expression-repeated/.

There are many situations in which this problem occurs. I will explicitly add this as one of the possibilities. See also TAG_EXPRESSION_REPEATED.json which are unit tests that we maintain that all of our validators have to pass. We are working on an AI assistant that uses this information to better explain errors, but this is a work in progress.

FYI @happy5214

[CodyCBakerPhD]

Could you clarify what error the python validator is returning that is causing the problem?

Oh, I see - if I run the code snippet above as-is, I see

Namespace(data_path='000003', error_limit=None, errors_by_file=False, format='text', log_level='WARNING', log_file=None, log_quiet=False, output_file='', print_output=False, suffixes=['events', 'participants'], verbose=False, check_for_warnings=False, exclude_dirs=['sourcedata', 'derivatives', 'code', 'stimuli'])
2025-12-18 17:46:27 - hed.bids_file_group - WARNING - Found 186 bad files in group 'events'
Validation completed.
        Found 0 issues.

I had assumed it would find the same issue but I suppose it skips further validation of content for bad filenames

If the code snippet is adjusted to

git clone https://github.com/bids-dandisets/000003 -b basic_sanitization

hedpy validate-bids 000003

then the output is indeed

Namespace(data_path='000003', error_limit=None, errors_by_file=False, format='text', log_level='WARNING', log_file=None, log_quiet=False, output_file='', print_output=False, suffixes=['events', 'participants'], verbose=False, check_for_warnings=False, exclude_dirs=['sourcedata', 'derivatives', 'code', 'stimuli'])
Validation completed.
        Found 4 issues.
Code counts: TAG_EXPRESSION_REPEATED(4)

which is less verbose than the main BIDS validator (which at least displayed the specific tag it thought was repeated and how many times it was repeated)

But again, what was confusing about this issue is the only place in the dataset where such tags are written (https://github.com/bids-dandisets/000003/blob/basic_sanitization/sub-YutaMouse42/ses-YutaMouse42%2B151102/ecephys/sub-YutaMouse42_ses-YutaMouse42%2B151102_events.json#L33) does not show any repetition - this was inferred, as you describe in the originating issue (bids-standard/bids-validator#326), by the duplicate columns in the companion TSV, which the validator automatically de-duplicates but thereby repeats the tag in the process

Reading through the main documentation on the error code, nothing seems to indicate this aspect other than the one line of correction pattern: https://github.com/hed-standard/hed-specification/blob/main/tests/json_tests/TAG_EXPRESSION_REPEATED.json#L41 but even that is vague to the cause

My suggestion would be to make a separate error code for duplicated table entries (if that is something you even want to capture) and not ephemerally concatenate tags from a duplicated event line