Validation def missing! :D

[TokyoWarfare]

Hello,

I'm trying to run my own MMS dataset.

After a lot of pain I've managed to get out of the pit just to find a funny missing piece in trainer.py.

SensatUrban and Toronto3D both use 'segmentation' as their dataset_task, but in trainer.py we have:

def validation(self, net, val_loader, config: Config):
    if config.dataset_task == 'classification':
        self.object_classification_validation(net, val_loader, config)
    elif config.dataset_task == 'segmentation':
        self.object_segmentation_validation(net, val_loader, config)
    elif config.dataset_task == 'cloud_segmentation':
        self.cloud_segmentation_validation(net, val_loader, config)
    elif config.dataset_task == 'slam_segmentation':
        self.slam_segmentation_validation(net, val_loader, config)
    else:
        raise ValueError('No validation method implemented for this network type')

But... object_segmentation_validation def is missing! 😄

Wondering what the intended behavior was here.

In my case, the MMS dataset is originally divided into large tiles (100+ million points each), which I chunk into smaller subtiles using a grid and a max point cap. So what goes into training are individual chunks, batch size 1.

For validation, my idea is to randomly select some of these chunks, run inference on them, and compare predictions against their ground truth labels. Would that be a reasonable approach to estimate validation loss in this context?

Thanks!

[HuguesTHOMAS]

Yes indeed, it should be 'cloud_segmentation' probably for these datasets. The function self.cloud_segmentation_validation(net, val_loader, config) probably already does what you want. It actually should handle testing on small spheres in the big point cloud already so you don't have to chunk the dataset yourself.

[TokyoWarfare]

for the record thsiis what I've done in the end as I've already all chunk data stored in .pkl due to a pre process stage and I've no access to the whole tile unless I reconstruct it form theese .pkl, so the nice ball data extraction method would require so.

`def mms_segmentation_validation(self, net, val_loader, config, debug=False):
"""
Lightweight validation for MMS chunks.
Runs inference on a subset of validation chunks and compares predictions against ground truth.
"""

    print("----------------VALIDATION-----------------")

    try:
        import time
        from utils.metrics import fast_confusion, IoU_from_confusions

        t0 = time.time()
        softmax = torch.nn.Softmax(dim=1)

        # Acceder al dataset original desde el BatchLoader
        label_values = np.array(val_loader.loader.dataset.label_values)
        ignored_labels = val_loader.loader.dataset.ignored_labels
        valid_labels = [l for l in label_values if l not in ignored_labels]
        nc_tot = len(label_values)

        predictions = []
        targets = []
        chunk_names = []

        net.eval()
        with torch.no_grad():
            for i, batch in enumerate(val_loader):
                if i >= config.validation_size:
                    break

                if 'cuda' in self.device.type:
                    batch.to(self.device)

                outputs = net(batch, config)
                probs = softmax(outputs).cpu().numpy()
                labels = batch.labels.cpu().numpy()
                chunk_name = batch.get("chunk_name", f"chunk_{i}")

                predictions.append(probs)
                targets.append(labels)
                chunk_names.append(chunk_name)

                if debug:
                    print(f"[VAL] Validando chunk: {chunk_name}")

        # Confusion por chunk
        Confs = np.zeros((len(predictions), nc_tot, nc_tot), dtype=np.int32)
        for i, (probs, truth) in enumerate(zip(predictions, targets)):

            for l_ind, label_value in enumerate(label_values):
                if label_value in ignored_labels:
                    probs = np.insert(probs, l_ind, 0, axis=1)

            preds = label_values[np.argmax(probs, axis=1)]

            # --- DEBUG LOGS ---
            print(f"\n[VALIDATION DEBUG] Batch {i}")
            print("  → truth unique:", np.unique(truth))
            print("  → preds unique:", np.unique(preds))
            print("  → truth shape:", truth.shape)
            print("  → preds shape:", preds.shape)
            print("  → num_classes:", nc_tot)
            print("  → label_values:", label_values)

            # --- SANITY FILTER ---
            valid_mask = (truth >= 0) & (truth < nc_tot)
            truth = truth[valid_mask]
            preds = preds[valid_mask]

            # --- CONFUSION MATRIX ---
            vec_conf = fast_confusion(truth, preds, label_values)

            # --- SHAPE CHECK ---
            if vec_conf.size != nc_tot * nc_tot:
                print("⚠️ Confusion vector size mismatch:", vec_conf.size)
                print("→ Cannot reshape to", (nc_tot, nc_tot))
                print("→ Skipping this batch")
                Confs[i] = np.zeros((nc_tot, nc_tot), dtype=np.int32)
            else:
                Confs[i] = vec_conf.reshape((nc_tot, nc_tot)).astype(np.int32)

        # Sumar confusiones
        C = np.sum(Confs, axis=0).astype(np.float32)

        # Eliminar labels ignorados
        for l_ind, label_value in reversed(list(enumerate(label_values))):
            if label_value in ignored_labels:
                C = np.delete(C, l_ind, axis=0)
                C = np.delete(C, l_ind, axis=1)

        # Calcular IoUs
        IoUs = IoU_from_confusions(C)
        mIoU = 100 * np.mean(IoUs)

        print(f'[MMS VALIDATION] mean IoU = {mIoU:.2f}%')
        if debug:
            print('Per-class IoUs:')
            for i, iou in enumerate(IoUs):
                print(f'  Class {valid_labels[i]}: {iou:.3f}')

        net.train()

    except AttributeError:
        print("[VAL] VALIDATION ERROR: No se pudo acceder a 'label_values' o 'ignored_labels'.")
        return

`