KPConv Bias Toward Dominant Classes — How to Break Semantic Collapse?

[TokyoWarfare]

Hi everyone,

I'm currently running my KPConv pipeline for the first times using a small sample of real training data. The dataset comes from a Mobile Mapping System (MMS) and is set along a mountain road, not an urban scene.

I've run into a persistent issue: the model tends to fixate on the most represented class, ignoring the rest of the semantic space.

Dataset imbalance
Initially, asphalt and vegetation were massively overrepresented.

I tried compensating with proportional class_w weights , but saw little improvement.

To isolate the issue, I created a testing dataset excluding those dominant classes, focusing only on objects.

Still, the model showed a strong bias toward the class with the most points. Only till I started greatly penalizing biased classed weights tarted to show action, yet going bias after 300 epoch for example.

What I’ve tried
Only after drastically reducing the weight of the dominant class (class 5) did other classes begin to appear in predictions.

However, around epoch 300, the model tends to collapse back into a single-class bias, ignoring minority classes again.

I haven’t used augmentations yet, I initially avoided them, fearing they might distort object geometry in this type of dataset. But now I’m reconsidering whether that’s contributing to the issue.

Validation behavior
mean IoU hovers around 9%, and preds unique during validation are consistently [0, 2], even when truth unique includes up to 5 classes.

I suspect the current val_radius and max_val_points might be limiting the representativity of validation chunks. I’ve increased them and am running a new experiment.

What I’m considering
Activating light augmentations to break geometric repetition.

Increasing val_radius to 1.0 and max_val_points to 50k for richer validation chunks.

Running inference on the training cloud itself to visualize segmentation in the best-case scenario.

Possibly increasing dataset size — currently it covers about a 200m stretch of road.

Has anyone encountered similar behavior with KPConv on imbalanced datasets? Is this kind of semantic collapse expected in outdoor scenes like this? Or is it just a matter of carefully tuning weights, chunk parameters, and augmentations until it stabilizes?

Any tips or insights would be hugely appreciated!

training.txt

parameters.txt

notice parameters says layers=1 but I've tested that all architecture runs , this is because itseems the log prints prior to loading architecture, but during training I see go trough it all.

[TokyoWarfare]

OK!, I've found an issue, that very likely is , if not the cause, a cause for the training to break...

while the C++ subsampling plugin works great for me.
the C++ nieghbor search pluging took forever to compute neighbors for my data, may be it is waay to dense or may be I did something wrong while compiling.

The thing is that I moved instead to baltree.

BUT, did not check for the neighbors to be ordered by distance.

And I just did, and they're not :D

I'm not sure how much of influence this will 've but aparently the c++ plugin does so... so I'll doo and cross fingers. Will report when I find solution in case anyone else falls in the same trap.

[TokyoWarfare]

-Fixed the neighbor ordering
-increased the learning rate to 0.001
-readjusted weights
-made the chunk size, the equivalent of the ball radius much bigger to do batches of 1 of 15x15m aprox for better context.

-nan issues gone.

some other lasses now start to show
[STEP] e000-i0044 => L=15.508 acc=10.3% class_acc={4: 92.76, 5: 0.0, 8: 10.28, 9: 0.0} / t(ms): 17187.7 18121.9 11751.3

dominant classes are 8-9 with multi millinon vs others in the tens or few hundred thousand points

Ideally I would preffer to set the weights per batch, I understand that setting
segloss_balance: class
should do the job but seemingly it does not, may be I'm failing the pass this param to config. will check.

I saw a user was able to run Kpconv on autopilot like lidar dataset creating reaaaaaly nice output on per frame basis, that gave me great hopes. will post params if I succeed. Cheers.

[TokyoWarfare]

OK, I give up :\

I spent 8+ weeks or so trying to adapt the pipeline to my data but seemingly I failed :D.

Doing inference on trained models turned out in a single class output. Meaning model collapsed as stats where pointing.

I will rather try to adapt the data to try on the custom data trianer in the repo.

On a positive note I got my hands very dirty and kind of understood how works what initially was a black box.

But I failed.

My idea was to be able to process large mobile mapping datasets and not be limited by memory. But overtime I understood the repo followed a somewhat similar principle with the ball data extraction yet limited by loading all on RAM.

What I did was break the workflow in 2 parts, data preprocessing 2 data training

In the first I did break all the dataset tiles into small chunks of nxn compute neighbors, ups, pools, normalize features and pack everythign with correct type as .pkl

On a second stage I would feed the trainer with such pkl after converting to torch tensors, and chekcing everything was up to standard... And after the collate tbh I've not touched anything other than create a custom validator.

I really can´t see where I failed. But during training it would allways massively bias towards a class. Even early on it would stick to a calss unless I did input massive weight balancing.

But even then if I killed on bias it would stick to next most overepresented class by point numbers.

So it felt like I would need to provide weights per batch, which I was specting segloss_balance: class to do when no weights provided (may be it does, I just noticed in tye yaml I was accientally also providing class_w) so may be it had bene ignored, doing a final test run , just in case....
But as I was saying I allways felt the training did biass early on towards a class and would stick to that as that would provide an easy win in accuracy score due to "getting rigth" the massively dominang class. (vegetation usually)
so I noticed in teh yaml last time I left the weights enabled, that would have nerfed the segloss_balance param so I'm doing a final run, I noticed some ok-ish batches
[STEP] e000-i0005 => L=31.355 acc=14.6% class_acc={0: 81.65, 3: 0.0, 4: 59.15, 8: 15.48, 9: 0.0} / t(ms): 6289.9 6684.5 4946.1

like at least is trying to classify other than class 8, but later on it seems like he gets he can cheat the system by going all in with class 8
[STEP] e000-i0012 => L=25.992 acc=94.0% class_acc={8: 99.11, 9: 0.0} / t(ms): 11726.5 11803.5 9260.2
[STEP] e000-i0015 => L=19.629 acc=80.7% class_acc={3: 0.0, 4: 0.0, 8: 100.0, 9: 0.0} / t(ms): 11020.7 11491.2 9020.5

, so, at this poit, I may be my data or my adaptation. whatever it is, I'm teared to bits by this.

May try the custom dataset on a clean repo download but in some time.

I'll share my scripts once I cleanup them a bit, may be something can be salvaged.

cheers.

[TokyoWarfare]

OK...

I was cleaning up the code when I found a very stupid bug I made when reading neighbors data when reading pkl. This would for sure break the training, no questions asked. Wether it would would be the only cause to break the training, that was yet to be seen.

Ok so I ran a simple XYZ + RGB training run just to see if anything changed.

The output was horrible, BUT, inference was no longer single feature . It was totally broken, but most importantly the broken data was following some RGB cues which gave me peace of mind as I had fear the feature index could be missaligned with points etc... it wasnt.

SO. I did another test with a more radical feature that is Verticality, which is very telling on what is asphalt and what is not.

I failed again, yet, by tuning the weights I was finally able to make the model see asphalt/ground + vegetation, somewhat which are the main classes, that sum up to 90%+ of the points. So the model has started to understand.

theese are the weights
class_w: [4, 3, 3, 5, 2, 2.6, 2, 2, 0.0015, 0.25]

the crazy thing is that slightly touching the weights breaks the balance.
It is as sensible that if I take class 4 and make weight as high as 20 because it is not showing up. Oddly because class 4 are the powerline masts, which in verticality field are very obvious, but in the training they're not seen so I attemped to make the weight higher to force it to "see" them, but that will break the balance of main 8-9 classes. So it feels like handling nitroglicerine.

My main concern at this point is wether this hypersinsitivity is the reflect of something else other than the class density inbalance...

One extra issue I've located and that I've yet to fix is tht I was doing the normalization wrong, not at data tile level, but at chunk level, so that would inevitably normalize a field unevenly sinze in a chunk min max levels won´t be the same as in other chunk, yet, sinze on theese runs I've been using massive 50x50m chunks , by using a higher initial dl, that should have been smoothed. I've to fix that.

Also having started with bigger dl (0.07) vs ( 0.02) may or may not have a positive influence. To be tested.

In short. I'm not abandoing this yet, I've pending one final test after fixing normalization and may be adding another field like surface variation.

Also, a problem with theese computed field in a non uniform density clud as are the mms that degrade as get away form road, is that in cloud compare there allways will be "gray" points where the scale was too small. I fix theese nans giving a 0 value, which will inevitably enter noise in the training... May be removing theese points from the data, at least for training would be interesting, at least to see what is their impact....

Tbh, I'm concernet about the hypersensitivity to the weights. I'll again test the segloss_balance calss to see if anything changed since the last time, else I'll try to compute weigts per chunk on the fly rather than having some global weights. or ty to get a print of the weights when using "class" balance mode to see why theese seem to have no effect...

One final test that would tell wether this adaptation is working is to use one of the known to work datasets like toronto 3D , use same params as in the train files ans wee what the hell happnes...

whatever...

[TokyoWarfare]

ok guys

made some progress

I was hitting the wall, clearly due to an issue with the global weights

I've bery dominant classes and I need to balance out the weights of each class.

Global weights where not providing enough fine tuning and slight changes in weights where causing major changes in the training metrics

SO
what I did was a per batch class weights, will measeure how many points for each class there are and build weights.

the difference in ressults was clear.

the output is still VERY noisy

ressults are ressults of inference of a model of just 2 epoch so I'm leaving the model overnight to do inference later on and see if any bettter comes out.

there are still 2 things on my end that are wrong for sure

• feature normalization still has errors.
• features still have a lot of nans since I'm computing them in cloud compare
• verticality & surface variation. Hence a lot of low res values are give value 0 which can likely cause input noise... not sure yet how to handle this. Interpolate or multiscale in cloud compare would be great... I may interpolate manually in a few files to see if has any good effect or get rid of the points...
• still not tested with the original dl 0.02 radius still on 0.07

as of now it kind of sees asphalt-ground vegetation manmade, (trunks are not yet seen as vegetation, I gess verticality makes them too desirable to ignore...) despite the labels are provided with thinner detail. if I could get rid of the noise, that would be great.

theese ressults kind of gave me a bit of hope after all.

on a positive note, even a 1080 does inference pretty fast!

I've noticed in this last shot that the seams of the chunking are qite noticeable not just for the density but for the ressults, else, they seem seamless.

[HuguesTHOMAS]

Nice work @TokyoWarfare! Happy to see you finally got some results.

Given what I see from your chunk borders, I think you should change your input strategy. If you have enough RAM on your machine, I would load all the datasets in memory, merging the chunks into big point clouds of each scene. Then I would use the input sphere dataloader. See the NPM3D dataset and also the NPM3D training parameters that could help you, as this is also a road environment dataset.

Note that in the case of a huge imbalanced class, I found that setting use_potentials=False for the dataset could help because I implemented a sampler that takes the same number of input spheres per class. So, for example, if you have three classes in your dataset and they have, respectively 1000, 1000000, and 100000 points, and you need to pick 600 spheres for the epoch, the sampler will choose 200 spheres centered on a point of each class, increasing the number of times it sees under-represented classes.

You can play around with using potential sampling or class-balanced sampling, and with or without your smart class weights, to see what is best.. If you need more details about all this, the best way is to search in my PhD thesis, which has a lot of details that are not in the paper: https://pastel.hal.science/tel-02458455/document

[TokyoWarfare]

Hello Thomas,

Thank you so much for your tips.
On my current PC I've 64gb and in the new one 190Gb or RAM. In the new one, as is, should be able to test bigger dataset wihout having to adapt any code, that is a big plus!.

My data is very heavy. Some tiles are 10+Gb per tile, in the 1-200M point range. Of course I could try to thin. I could even run a voxelization of 1cm first as source data, to fit more data without loosing that much detail. Voxelization really works nice in not loosing detail on more sparse features like powerlines, never tried before, works certainly great.

Now that I've understand the steps of your pipeline, I think it's time for me to dump my prototype implementation and go back to test my data in your code, as is. If I ever make it work there will be time for customizaitons. My vision was to try to adapt more to a producton environment by extracint all the config params to yaml, separate the preprocess from trianing etc... but somewhere in the middle I messed things up.

That paper was really impresive and certainly gave me hopes as the Lille 3D looks a lot to mine in terms of class inbalance as it does my MMS mountain road pass data.

I need a break tho. I got into this as deep as I could, and I failed and this took me already a lot of weeks. I've to move on on other areas of my project.

I'm interesting in classifying the MMS data because I'm making a game using point clouds.
Here a sample
https://youtu.be/7JvwxPE_iq0

The way I made it technically work is to not use point clouds for biggest human made structures like asphalt & retaining walls, where A LOT of points are concentrated and a poligonal mesh does better. I rasterize the points RGB/Normal data for theese surfaces and I gain a lot of performance and integrate much better with the game engine. The ground itself could be relatively easy extracted with CSF and is ready after some manual cleanup.

Extracting the rest is where the real pain is. The reason I needed to segment theese are various but esentially color artifact corrections & thining. Trees where they meet the sky allways have that white aura/edge that ruins the visual perception, this is because the RGB sensor is not perfectly aligned and allways creates theese around edges that have sky as bkg. To automatically filter theese I need vegetation as unique element else I would damage other white areas like could be guard rails.

Having vegetation and other elements independently I can also subsample vegetation, why I try to do is to empty the "inner" part of the vegetation so that the most visible points of the "outer shell" remain, hence the most visible parts are retained and the performance increases by not rendering points not so important.

And having less dense objects like guardrais etc separated I can render them further distance in the game.

To fill in the voids in vegetation due to thining, what I do is to create a polygon mesh with vertex colors of the vegetation only and use it as bkg. Some wort of cheapo splats, before the splats where a thing :D

And then, out of the blue, gaussian splats showed up.

At this point, I'm puting a pause in the segmentation part, to focus in the game developement and more in the splats research as splats may solve a lot of the problems I've with the point cloud plus give better visual fidelity.

YET

Yet... the segmentation may still play an important role, as the splats seem to be some sort of "extended" point cloud, there may be elements that would still be better replaced by polygonal mesh. I've dome some tests with on car mounted camera road capture to splats and the guardraisl seem to break while the bkg vegetations holds very well. But when it works, it's like you're there.

There are 2 hardware cameras that are hitting the market just noe at nearly consumer prices 4500Us for hte PortalCam. Cameras with a lidar and 2 front cams + 2 wide 180ºcams built in, that take 800K points per sec lidar and then built splats. Theese two worlds are merging and I will try to adapt my workflow with the MMS to this new hardware.

If you or anyone that works in research of clouds you think is interested in how I built the world from MMS data I can share my step by step power points without problem. The most reusable part is probably the asphalt-ground generation as the other part will likely go obsolte soon adn also was a ressult of only having the end MMS data.

if I have had access tot he original RGB cams I woulc have created better RGB blending, splats etc... But the ground generation is still reusable and quite automated with scripts.

Best

Pablo.