drain host is not filling target OSDs evenly

[hydro-b]

First: thanks for the tool. It's really useful. We use it to drain one host at a time (with pgremapper looping over its OSDs). We noticed however that the target OSDs are not evenly filled up (from a PG count view, or usage for that matter). This in-balance might get really big (i.e. more that 40 PGs). We can straighten that out mid-process by stopping new remaps and use the "balance-host" option. But it's a waste of time. It would be better to get those PGs on the right OSD the first time.

I have taken a look at the code and if I understand correctly the decision to make what OSD is used as the target is handled in function "calcPgMappingsToUndoUpmaps". It does not seem to take into account how many PGs are already on the OSD. Is that correct? Or is there some heuristic that does take this into account?

Note: we do not clear any upmaps on the target host before running pgremapper. Might this influence mapping decisions?

[baergj]

Hello!

Indeed, that is a weakness of drain that is noted in the readme: No attempt is made to balance the fullness of the target OSDs; rather, the least busy target OSDs and PGs will be selected. Internally, we've typically set up constrained source->target relationships that avoid this issue, but you're right that it's not ideal.

I hesitated to implement this into the initial tool because it seemed it would add considerable complexity to calcPgMappingsToDrainOsd() and/or remapLeastBusyPg(). This could either be done opportunistically, adjusting the scoring algorithm in remapLeastBusyPg() to prefer to fill less full OSDs, or as a hard constraint. If the system has similarly-performing disks and starts balanced, then the former approach would likely be sufficient and not too hard to implement, and would retain higher drain performance in the case where a slight imbalance exists.

I'll add this to our internal TODO list, without any promises about turnaround time, but, as always, contribution are welcome!

[hydro-b]

Sorry, I missed that part obviously.

Internally, we've typically set up constrained source->target relationships that avoid this issue, but you're right that it's not ideal.

A Constraint as in that you drain from a fixed OSD -> fixed OSD? That is what I'm up to next. Create a loop over pgremapper and have 1 backfill per OSD to 1 fixed other OSD. In that case you know beforehand you are not going to fill up an OSD too full, or create imbalance.

This could either be done opportunistically, adjusting the scoring algorithm in remapLeastBusyPg() to prefer to fill less full OSDs, or as a hard constraint. If the system has similarly-performing disks and starts balanced, then the former approach would likely be sufficient and not too hard to implement, and would retain higher drain performance in the case where a slight imbalance exists.

Adjusting the scoring algorithm seems like the best approach to me.

[baergj]

A Constraint as in that you drain from a fixed OSD -> fixed OSD?

Exactly, and the scheme you describe is what we have historically done.