out of memory for instrument "all"

[playgithub]

❓ Questions and Help

hareware

CPU memory 32GB
GPU memory 8GB

problem

In the following sample
for instrument "all", it'll be out of memory
for instrument "csi300", it's ok

sample

out of memroy by dataset = init_instance_by_config(task["dataset"]) below

...
stock_pool = 'all'

train_start_date = '2017-7-12'
train_end_date = '2020-12-31'
valid_start_date = '2021-01-01'
valid_end_date = '2023-12-31'
test_start_date = '2024-01-01'
test_end_date = '2025-12-31'
...
task = {
    "model": {
        "class": "LGBModel",
        "module_path": "qlib.contrib.model.gbdt",
        "kwargs": {
            "loss": "mse",
            "colsample_bytree": 0.8879,
            "learning_rate": 0.0421,
            "subsample": 0.8789,
            "lambda_l1": 205.6999,
            "lambda_l2": 580.9768,
            "max_depth": 8,
            "num_leaves": 210,
            "num_threads": 20,
            "early_stopping_rounds": 50,
            "num_boost_round": 1000
        }
    },
    "dataset": {
        "class": "DatasetH",
        "module_path": "qlib.data.dataset",
        "kwargs": {
            "handler": {
                "class": "Alpha158",
                "module_path": "qlib.contrib.data.handler",
                "kwargs": data_handler_config
            },
            "segments": {
                "train": (train_start_date, train_end_date),
                "valid": (valid_start_date, valid_end_date),
                "test": (test_start_date, test_end_date)
            }
        }
    }
}
...
dataset = init_instance_by_config(task["dataset"])
...

quesiton

any solution?

[SunsetWolf]

Hi, @playgithub , This is expected behavior. When instrument="all", init_instance_by_config(task["dataset"]) needs to load and process the full market universe, which significantly increases memory usage.

Based on memory profiling, the peak usage is ~28.5 GB during dataset initialization alone. After accounting for OS and other processes, a 32 GB machine is not sufficient.

[playgithub]

expected, any workaround? e.g.:

• a compressed dataframe in memory, which only uncompressed when get data
• or map data on disk to memory

memory is qutie expensive now ;)

[SunsetWolf]

Hi @playgithub , thanks for the follow-up.

In theory, yes—there is room to reduce memory usage. However, in practice this would require substantial refactoring across the data pipeline, along with careful profiling and re-organization of how data is loaded and cached. It’s not a small or localized change.
At the moment, the practical workarounds are still to:

• use a smaller instrument universe (e.g. csi300)
• shorten the time range
• or scale up memory / use a larger machine

That said, contributions in this direction are definitely welcome if someone is interested in tackling this optimization.

[playgithub]

OK, now a small instrument makes more sense to me than a large mixed one.
If I find a large instrument is better than a small one, I'll re-mention this topic.
Thanks

[WindBT47]

You can write a small script to solve this.
Steps:
1. Use qlib.data.D.features() to compute base features (d=0)
2. Use qlib.data.D.features() to compute labels
3. Shift features by d=1..window-1 using pandas groupby().shift()
4. Apply GaussRankNorm per-day cross-section
5. Fillna + ProcessInf
6. Save as float32 npz
7. Train with your calculated features cache