Skip to content

feat: add --verbose flag to expose stateless installation steps of Dataform Core and identify bottlenecks #2099

@ashish10alex

Description

@ashish10alex

When specifying dataformCoreVersion in workflow_settings.yaml file dataform cli does a stateless installation. Details of this step is not visible to the user via any flag. If this step takes time or is stuck due to network issues it is unclear on whats happening. Furthermore, this step in my limited testing takes significantly longer (> 3x) on windows machine on a repository with just 30 actions. Thus making this flag available will help user identify bottlenecks in their setup. dataform --version 3.0.48 was used for testing.

1. Time taken on Mac OS M2 Pro, 16 GB ~1.5 secs

❯ time dataform compile --quiet
Compiling...

Compiled 30 action(s).
23 dataset(s).
5 assertion(s).
2 operation(s).
dataform compile --quiet  0.80s user 0.26s system 72% cpu 1.466 total

Time taken on Windows machine , 12th Gen Intel(R) Core(TM) i7-12800H, 14 cores, 64 GB RAM. ~ 5 secs

> Measure-Command { dataform-cli compile}

TotalSeconds      : 5.0940425

TotalMilliseconds : 5094.0425

Time taken on WSL Windows machine , 12th Gen Intel(R) Core(TM) i7-12800H, 14 cores, 64 GB RAM ~2.9 secs

❯ time dataform compile --quiet
Compiling...
 
Compiled 30 action(s).
23 dataset(s).
5 assertion(s).
2 operation(s).
 
real    0m2.960s
user    0m0.576s
sys     0m0.216s

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions