Skip to content
Open

HW 7 #21

Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 21 additions & 21 deletions Readme.md
Original file line number Diff line number Diff line change
@@ -1,40 +1,40 @@
# Задание №7. Оптимизация `test-suite` и сбор `DX`-метрик
# Task #7. Optimizing `test-suite` and collecting `DX`-metrics

В этом задании вам нужо попрактиковаться в оптимизации `test-suite` и сборе `DX`-метрик.
In this task you need to practice optimizing `test-suite` and collecting `DX`-metrics.

## Оптимизация test-suite
Можно оптимизировать test-suite `dev.to`, либо своего проекта.
## Test-suite optimization
You can optimize the test-suite of `dev.to`, or your own project.

В любом случае для сдачи задания нужно написать `case-study` о сделанной оптимизации.
In any case, to submit the task you need to write a `case-study` about the optimization done.

Вооружитесь инструментами, рассмотренными на лекции, и разгоните этот test-suite!
Arm yourself with the tools covered in the lecture, and speed up this test-suite!

## Сбор DX-метрики
## Collecting DX-metrics

Чтобы запечатлеть свой прогресс и в дальнейшем защитить его от деградации, сделайте сбор `DX`-метрики времени выполнения `test-suite` в `InfluxDB` и постройте график в `Chronograf`.
To capture your progress and protect it from degradation in the future, set up collection of `DX`-metrics for test-suite execution time in `InfluxDB` and build a graph in `Chronograf`.

Сделать это очень просто с помощью https://github.com/influxdata/TICK-docker и https://github.com/palkan/influxer.
This is very easy to do with https://github.com/influxdata/TICK-docker and https://github.com/palkan/influxer.

Подумайте, как бы вам было удобно прикрепить отправку метрики в `InfluxDB` к прогону тестов.
Think about how it would be convenient for you to attach sending metrics to `InfluxDB` to test runs.

## Hints

Старайтесь держать `feedback-loop` коротким. `test-suite` `dev.to` целиком в один процесс выполняется около 10 минут, это очень долго.
Try to keep the `feedback-loop` short. The entire `dev.to` `test-suite` runs in about 10 minutes in a single process, which is very long.

Используйте семплирование или работайте с конкретными наиболее долгими тестами.
Use sampling or work with specific slowest tests.

Если вы обнаружите проблему, то выберите какой-нибудь один тест, на котором она воспроизводится, и исправьте её, работая с этим тестом.
If you discover a problem, choose one test where it reproduces and fix it while working with that test.

Попробуйте все инструменты из `test-prof`!
Try all the tools from `test-prof`!

`json-flamegraph` для всего `test-suite` целиком занимает около `1Gb`. `Speedscope.app` открыть его не может.
`json-flamegraph` for the entire `test-suite` takes about `1Gb`. `Speedscope.app` cannot open it.

C отчётом `stackprof` для всего `test-suite` можно работать через `CLI stackprof`, и через `qcachegrind`.
You can work with the `stackprof` report for the entire `test-suite` through `CLI stackprof`, and through `qcachegrind`.

С `ruby-prof` надо работать ещё аккуратнее, потому что он, как `tracing profiler`, собирает очень много данных, с которыми в дальнейшем тяжело работать.
With `ruby-prof` you need to be even more careful, because as a `tracing profiler`, it collects a lot of data that is hard to work with later.


## Чек-лист для сдачи задания
- [x] PR в этот репозиторий с `case-study` о проделанной оптимизации и достигнутых результатах в описании
- [x] Если оптимизируете `dev.to`, то сделать `PR` с оптимизацией в https://github.com/hardcode-dev/rails-optimization-2-task4 и добавить в этот PR ссылку на него
- [x] Добавить скриншот с графиком изменения времени прогона `test-suite` в `Chronograf` по мере оптимизации
## Checklist for submitting the task
- [x] PR to this repository with a `case-study` about the optimization done and results achieved in the description
- [x] If optimizing `dev.to`, make a `PR` with optimization to https://github.com/hardcode-dev/rails-optimization-2-task4 and add a link to it in this PR
- [x] Add a screenshot with a graph of test-suite execution time changes in `Chronograf` as optimization progresses
28 changes: 28 additions & 0 deletions case-study.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
Decided to optimize my current project (~83% coverage and 1905 tests), tests were running in a single thread for ~10 minutes, but earlier before I started taking the course I suggested using parallel tests and the run time dropped to ~2.7 (and this is taking into account fixes made after previous lessons in the course, originally it was ~3 minutes) minutes on my local machine (12 threads), unfortunately in CI we only use one core so parallel tests didn't help there

# Step 1
started with RD_PROF and it showed me the 5 slowest tests, one of which ran for ~1 minute

used let_it_be in the necessary places and the overall run time dropped to 1.4 minutes

there were problems with let_it_be, it didn't want to work with tests nested in multiple contexts, it didn't want to work with timecop and so on, but the main problem was that if you run a file with tests they passed but if you run all tests in the project some tests failed, because for some reason it reset associations for factories that were needed in the test, couldn't find a native solution and had to rewrite the test so that associations were created directly inside the test.
but overall let_it_be is a really cool find.


# Step 2
Then went to check factories and the picture was quite sad, didn't redo the factories since there wasn't enough time, but I already know roughly what can be done with this.

# Results
Overall the time for running tests was evenly spread and pulling each one separately would take a lot of time.

also tried running all tools from the screencasts to feel them out.

let_it_be helped solve the problem when after parallel run at the end there was a very slow test that took a whole minute to complete.

Of course in terms of test optimization in the project there's a lot of untouched ground, but the project is due soon and there's no time for this(((

as a challenge in the future I'll try running tests directly in RAM by keeping the database there

Optimization result:

![Optimization result](https://i.imgur.com/HIptMAH.png)