Discussion: replace ConsensusID with a more robust multi-engine rescoring path

[ypriverol]

Context

In the multi-engine search path (e.g. Comet + MSGF+, optionally + Sage), quantms currently merges per-engine PSMs with CONSENSUSID after per-engine PercolatorAdapter rescoring, then applies FDR. This generates per-file CONSENSUSID fan-out and relies on IDRipper when the rescoring step was previously merged per-sample or per-project.

With ms2features_enable=true (quantms-rescoring via ms2rescore), every engine's PSMs receive the same run-invariant features (MS²PIP, DeepLC, precursor-ppm). Those features are numerically comparable across engines by construction, which raises the question: can we simplify the multi-engine path by dropping CONSENSUSID and merging engines directly into a single PercolatorAdapter call?

This issue is a discussion thread to decide which direction to take.

What we learned from the OpenMS source

Short version: the naive path IDMerger → PercolatorAdapter is not supported by OpenMS.

• IDMerger is a dumb concatenator. It preserves per-PeptideHit meta-values verbatim as a sparse flat-map. A Comet PSM carries xcorr and no hyperscore; a Sage PSM carries hyperscore and no xcorr. There is no unified feature schema in the merged idXML — just ragged per-hit meta-values (IDMergerAlgorithm.cpp:170-237).
• PercolatorAdapter explicitly refuses multi-engine input (PercolatorAdapter.cpp:493-498):

  if (se1 != se2) { OPENMS_LOG_ERROR << "... Use TOPP_PSMFeatureExtractor to merge ...";
                    return INPUT_FILE_CORRUPT; }
  

• Even if you bypassed that check, PercolatorInfile::store silently drops any PSM missing any feature column (PercolatorInfile.cpp:531-566). Ragged rows never reach the percolator binary.

The sanctioned multi-engine path in OpenMS is:

IDMerger -merge_proteins_add_PSMs
  → PSMFeatureExtractor -multiple_search_engines -impute
  → PercolatorAdapter

PSMFeatureExtractor -impute fills missing features with observed min/max (or float::max with -limit_imputation) — that's the "NaN handling" step.

Gap: PercolatorFeatureSetHelper::addMULTISEFeatures (line 502-529) branches only on Comet, MSGF+, X!Tandem, Mascot. Sage is not in this code path. PSMFeatureExtractor is also marked @experimental.

Candidate approaches for discussion

1. Adopt the OpenMS-sanctioned path. Use IDMerger -merge_proteins_add_PSMs → PSMFeatureExtractor -multiple_search_engines -impute → PercolatorAdapter. Works for Comet + MSGF+ (+ X!Tandem, Mascot). Does not work for Sage without an upstream patch to PercolatorFeatureSetHelper. Conservative, uses well-tested OpenMS primitives.

2. Bypass per-engine features, rely only on rescoring features. With ms2rescore enabled, every engine's PSMs carry identical run-invariant features (MS²PIP / DeepLC / precursor-ppm). Feed the merged idXML to PercolatorAdapter with -generic-feature-set (or a similar restriction) so only the shared columns drive the SVM, ignoring engine-native scores. Works for any engine including Sage. Loses engine-native features from the SVM, which may or may not hurt PSM recovery — needs a benchmark.

3. Keep CONSENSUSID. It was purpose-built for harmonising scores across engines. The per-file fan-out is real but the step is cheap. In light of the blockers above, this may remain the right tool and the scope of this issue shrinks to "stop discussing the replacement, invest elsewhere."

4. Contribute upstream. Open a PR against OpenMS to add Sage support to PercolatorFeatureSetHelper::addMULTISEFeatures and lift PSMFeatureExtractor's experimental label. This unblocks option 1 for every engine and benefits the broader OpenMS user base. Long timeline.

What would help

• Benchmark evidence on Comet + MSGF+ datasets comparing options 1, 2, and 3 at 1 % FDR (PSM / peptide counts, calibration). No published benchmark covers this at quantms scale.
• Scientific input on whether option 2 (rescoring-only feature set) is defensible — is it equivalent to option 1 empirically, or does dropping engine-native features cost us?
• Maintainer preference on upstreaming Sage support vs. working around it in quantms.

Prerequisite already in flight

#706 removes ms2features_range = by_sample | by_project and the IDRipper Nextflow module. These exist only to enable pooled Percolator runs that get split back per-run for CONSENSUSID, and removing them is a no-op for the default path while simplifying any future CONSENSUSID replacement work. That PR is independent of the discussion here — the multi-engine topology is untouched.

Not in scope for this issue

• Changes to the DIA workflow (no CONSENSUSID usage).
• Pooling Percolator across files (separate discussion, requires its own benchmark).
• Native-Percolator bypass (running percolator outside PercolatorAdapter on merged pin files).

[daichengxin]

Hi all, I tried and benchmark some methods to replace consensusid:

First, use PSMFeatureExtractor to merge the search results instead of using consensusID. Then, add additional features using quantms-rescoring. Finally, use Percolator to calculate PEP/FDR. The results as follows on PXD001819 datasets:

PSMFeatureExtractor \\
    -in ${id_files.join(' ')} \\
    -out ${meta.mzml_id}_feat.idXML \\
    -threads $task.cpus \\
    -multiple_search_engines \\
    -skip_db_check \\
    -impute \\
    $args \\
    2>&1 | tee ${meta.mzml_id}_extract_psm_feature.log

• comet_rescoring: Comet search combined quantms-rescoring
• consensusid: Comet & MSGF, and then consensusid
• merged: Comet & MSGF and then merge features
• consensusid_rescoring: consensusid combined quantms-rescoring
• merged_rescoring: merge idXML features and quantms-rescoring

As can see, the change is acceptable and we can further refine. And we’ve cut the number of temporary files and the computational load in half. Subsequently, EXTRACTPSMFEATURES will be replaced by a Python script in quantms-rescoring to address certain issues and further reduce computational load and temporary files. What do you think?

[ypriverol]

@timosachsenberg @jpfeuffer As we discussed before, we don't like consensusID; in some cases it behaves well and in others no; Additionally, I think it would be good to treat search engines as properties to Percolator and have Percolator decide who is better.

In addition, I would love to remove some of the idXML intermediate steps to reduce intermediate files and conversions; like for example most of the search engines now; are taking the parameters from quantms; and from the commandline arguments; and all of them MSGF+; Comet and ProSE produces pin for percolator; we are not using in anayway the pipeline without percolator; why not producing the idXML only at the end of the percolator; and use pin all the way; is simpler; smaller and easy to use; this will reduce a lot the space that we can run.

In my mind, we have two workflows now:

search engine (pin) -> rescoring (optional) (pin) -> percolator (idFilter here even) (idXML)
search engine (pin) -> percolator (idXML)
multiple search engine (pin) -> rescoring (required) (pin) -> percolator (id Filter) (idXML)

We can do this by combining within the same process in nextflow or in OpenMS, but we should start moving toward fewer processes; less intermdiate files; less file sizes.

@timosachsenberg @jpfeuffer ?

[jpfeuffer]

No. I don't think using pin solves anything. Percolator should accept our parquet instead..
In your plot consensusID is best. Why not benchmark on something where consensusID is actually is worse? What do you actually want? Get better? Or you just hate consensusID so much, now that you know how it works, that you want to get rid of it even if it's better?

Regarding combining steps. Yes, I voted for this for a long time already. That's why I did the IsobaricWorkflow for example

[timosachsenberg]

That’s a nice result and I think the drop is acceptable. Never really liked the hacks we had to do to perform consensusid in OpenMS. Maybe we could think about framing this as our new quantms-consensusid ? Merge does not sound exciting…and it is a bit more

[ypriverol]

@jpfeuffer, a couple of things:

• If we have parquet, even better; if the parquet @timosachsenberg is already supported everywhere, we are ready to support it.
• We discussed recently that ConsesusID introduces a new step that algorithmically can perform better but sometimes worse; as @timosachsenberg mentioned, ConsesusID boosts one search engine, giving more priority to hits that are more consensus, rather than hits that are "better"; what I aim to have less steps, fewer big temporary files; less steps that don't give us an advantage.
• As I mentioned before, after the rescoring, what we see is that multiple search engines are marginal; then, if we see in the future less and less use of this feature, I want to have fewer steps that make the process difficult to trace; for example, recently we discussed PepMatrix vs the default in ConsensusID.
  -@jpfeuffer after I refine all the ID part, including cleaning the intermediate step; I myself and @daichengxin can also help will go over the TMT workflow. I agree with you, it should be a priority.

Just for a bit of background, recently I have been running some really large plasma and tissue datasets with more than 300 RAW files, all with multiple search engines; for every raw file (2 Search engines + 2 rescoring + 2 percolator + 1 ConsensusID + 1 ID Filter) X 300 = 2400 idXMLs and process.

My proposal:

(2 Search engines + 1 Merge rescoring + Percorlator Filter) X 300: 1200 All with parquet in OpenMS.

@timosachsenberg I think we can integrate that as a feature of quantms-rescoring because we always need to do rescoring to be able to do this.