allocation checks: check branch.status instead of making call to k8s

[boddumanohar]

Currently during branch create we have checks to check against provisioning limits. These changes takes time because it goes through all the branches and queries k8s for the resources information.

This is very expensive operation. So removed _collect_branch_statuses function call and replaced it with logic to get the same from branch.status

Instead we could query branch.status. The tradeoff here is branch.status is around 1 minute old.

[mxsrc]

Do you have numbers on how long the current approach takes?

[boddumanohar]

Generally create branch API takes around 10-15 secs to return 201 response.

[mxsrc]

Is all of that due to the provisioning check? On second thought, probably not. I suggest profiling what actually takes most of the time before optimizing parts and chosing trade-offs. This should be possible using py-spy in the deployed container: py-spy record -o profile.svg --pid 12345, or by editing the code:

import cProfile
...
    profiler = cProfile.Profile()
    profiler.enable()
    ....
    profiler.disable()
    stats = pstats.Stats(profiler)
    stats.sort_stats('cumulative')
    stats.print_stats(20)

[boddumanohar]

if a project has 100 branches, the function _aggregate_group_by_resource_type calls _collect_branch_service_health which probes all the branch services and gets its health.

From the branch status, we exclude the branches with status BranchServiceStatus.STOPPED, BranchServiceStatus.DELETING

_BRANCH_SERVICE_ENDPOINTS: dict[str, tuple[str, int]] = {
    "database": ("db", 5432),
    "pgbouncer": ("pgbouncer", 6432),
    "storage": ("storage", 5000),
    "meta": ("meta", 8080),
    "rest": ("rest", 3000),
    "pgexporter": ("pgexporter", 9187),
}

So during branch create, we make 100*6 = 600 network calls for the sake of validations. This seems very costly.

[mxsrc]

I see the point. I think 100 branches is on the upper limit of what we can expect, and we should be able to parallelize this significantly. Still, it's a lot. Do you think it'd be feasible to modify the pod to include a readiness/liveness probe? I'm trying to find a way where we don't have to implement everything ourselves but can use builtins instead.

[noctarius]

The comment should be brought up tomorrow morning on the standup. I think, otherwise, this PR is ok, since the branch status is already persisted in the database.

[noctarius]

This is dangerous (and I guess it was also wrong with the old code). Stopped branches don't count towards CPU / RAM allocation, but I think storage (db, object) still count, even when stopped. Only deleted branches are out of the game. WDYT @boddumanohar?