shiny tests using chromote #209

biplab-sutradhar · 2025-07-07T15:32:33Z

Closes #143

This PR migrates RSelenium tests to Chromote for animint2, covering examples/shiny, examples/shiny-WorldBank, and examples/rmarkdown visualizations.

…helper functions for shiny, RMarkdown app to helper-function.R

…ium_to_chromote

codecov · 2025-07-10T20:34:06Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.04%. Comparing base (940cf7d) to head (7f86cb8).
⚠️ Report is 12 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff             @@
##           master     #209       +/-   ##
===========================================
- Coverage   77.76%   62.04%   -15.73%     
===========================================
  Files         164      164               
  Lines        8771     8776        +5     
  Branches      564      181      -383     
===========================================
- Hits         6821     5445     -1376     
- Misses       1950     3331     +1381

Flag	Coverage Δ
javascript	`45.48% <ø> (-49.91%)`	⬇️
r	`69.62% <ø> (-0.08%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

tdhock · 2025-07-11T15:54:50Z

tests/testthat/test-shiny.R

+if (!requireNamespace("this.path", quietly = TRUE)) install.packages("this.path")
+library(this.path)
+if (exists("this.path")) {
+  setwd(dirname(dirname(dirname(this.path::this.path()))))
+} else {
+  setwd(normalizePath(file.path("..", "..", "..")))


please avoid adding dep on this.path

tdhock · 2025-07-11T15:55:58Z

tests/testthat/helper-functions.R

+  if (!file.exists(rmd_file)) stop("RMarkdown file does not exist: ", rmd_file)
+  if (!requireNamespace("rmarkdown")) stop("Package 'rmarkdown' is not installed")
+  app_url <- sprintf("http://127.0.0.1:%d", port)
+  proc <- callr::r_bg(function(rmd_file, port) {


please add callr to Suggests to avoid

* checking for unstated dependencies in ‘tests’ ... WARNING Warning: '::' or ':::' imports not declared from: ‘callr’ ‘this.path’ 'library' or 'require' calls not declared from: ‘callr’ ‘this.path’

biplab-sutradhar · 2025-07-11T15:57:51Z

This pr have more issues I will update

tdhock · 2025-07-14T10:41:35Z

R/z_knitr.R

 #' @export
 #' 
+
+


please undo addition of empty spaces

tdhock · 2025-07-14T10:41:56Z

R/z_knitr.R

    list(jsonFile = "plot.json")
  }
-  shiny::markRenderFunction(animint2::animintOutput, renderFunc)
+  shiny::markRenderFunction(animint2::animintOutput, renderFunc) 


please undo addition of white space at end of line

biplab-sutradhar · 2025-07-25T19:53:22Z

@tdhock Can you please check if any changes are needed?

tdhock · 2025-07-27T16:32:46Z

@biplab-sutradhar can you please review this PR?

tdhock · 2025-07-27T16:33:03Z

I meant
@suhaani-agarwal can you please review this PR?

suhaani-agarwal · 2025-07-27T17:02:57Z

Hi! I noticed that there are some test failures and warnings in the test-shiny -test suite ,

However, the test suite is still passing successfully without throwing an error. Could you please take a look at that?

── Error ('test-shiny.R:27:3'): animint plot renders in a shiny app ────────────
<error_stop_port_search/rlang_error/error/condition>
Error in `startup(port = port, ...)`: Chrome debugging port not open after 10 seconds.
Backtrace:
     ▆
  1. └─ChromoteSession$new() at test-shiny.R:27:3
  2.   └─chromote (local) initialize(...)
  3.     └─chromote::default_chromote_object()
  4.       ├─chromote::set_default_chromote_object(Chromote$new())
  5.       └─Chromote$new()
  6.         └─chromote (local) initialize(...)
  7.           └─Chrome$new()
  8.             └─chromote (local) initialize(...)
  9.               └─chromote:::launch_chrome(path, args)
 10.                 └─chromote:::with_random_port(...)
 11.                   ├─base::tryCatch(...)
 12.                   │ └─base (local) tryCatchList(expr, classes, parentenv, handlers)
 13.                   │   └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
 14.                   │     └─base (local) doTryCatch(return(expr), name, parentenv, handler)
 15.                   └─chromote (local) startup(port = port, ...)
 16.                     └─rlang::abort(...)

── Warning ('test-shiny.R:59:3'): WorldBank shiny app functionality ────────────
localhost:4473 cannot be opened
Backtrace:
    ▆
 1. └─start_shiny_app(worldbank_dir, port) at test-shiny.R:59:3
 2.   ├─base::try(...) at tests/testthat/helper-functions.R:385:5
 3.   │ └─base::tryCatch(...)
 4.   │   └─base (local) tryCatchList(expr, classes, parentenv, handlers)
 5.   │     └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
 6.   │       └─base (local) doTryCatch(return(expr), name, parentenv, handler)
 7.   └─base::socketConnection("localhost", port, open = "r+", timeout = 5) at tests/testthat/helper-functions.R:385:5

── Warning ('test-shiny.R:118:3'): animint plot renders in an interactive document ──
localhost:4656 cannot be opened
Backtrace:
    ▆
 1. └─start_rmd_app(rmd_file, port) at test-shiny.R:118:3
 2.   ├─base::try(...) at tests/testthat/helper-functions.R:408:5
 3.   │ └─base::tryCatch(...)
 4.   │   └─base (local) tryCatchList(expr, classes, parentenv, handlers)
 5.   │     └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
 6.   │       └─base (local) doTryCatch(return(expr), name, parentenv, handler)
 7.   └─base::socketConnection("localhost", port, open = "r+", timeout = 5)

[ FAIL 1 | WARN 3 | SKIP 0 | PASS 8 ]

biplab-sutradhar · 2025-07-27T17:17:12Z

@suhaani-agarwal check after I rerun no test fails but I will look into why the error is not appearing after the test failure.

suhaani-agarwal · 2025-07-27T17:36:45Z

tests/testthat/test-shiny.R

+  b$view()
+  on.exit(b$close(), add = TRUE)
+  b$Page$navigate(app_info$url)
+  b$Page$loadEventFired(wait_ = TRUE, timeout = 30000)


I noticed that a new ChromoteSession is being initialized within each test. Would it be possible to use the existing tests_init() function instead? It already starts a single ChromoteSession that's used for the renderer and compiler tests

to run shiny test I don't use tests_init() directly use testthat::test_file("test-shiny.R")

shiny test need a separate Shiny server via start_shiny_app()/start_rmd_app()

suhaani-agarwal · 2025-07-29T19:04:47Z

tests/testthat/test-shiny.R

+    skip("Tests skipped for mock app")
+  }
+
+  port <- sample(3000:9999, 1)


I think instead of random ports, fixed ports (eg. 3147 , 3148 etc.) should be used (like were being used earlier). The usage of these random ports might be the reason of the WARNINGS being shown currently in the test-shiny workflow

…ium_to_chromote

…per-HTML.R

biplab-sutradhar · 2025-08-28T05:16:42Z

Shiny tests generate JS dynamically in the page and run on different port 3147. An if/else block like this won’t work

if (is_shiny_test) {
    stop_js_coverage('shiny')
} else {
    stop_js_coverage('static')
}

because stop_js_coverage() overwrites the coverage file, and we need to collect from both static and Shiny tests. The paths also don’t match. hence added

# 1. Run static tests
start_js_coverage()
tests_run(filter = "renderer")
stop_js_coverage('static')

# 2. Run Shiny tests
start_js_coverage()
tests_run(filter = "shiny")
stop_js_coverage('shiny')

So I thought the flow to collect static coverage first, collect Shiny coverage second, then merge both into one report.

biplab-sutradhar · 2025-08-28T05:58:35Z

I added renderer to test-shiny.R, to run using tests_run(filter = 'shiny') and when call tests_run(filter = 'renderer') and tests_run(filter = 'shiny') for coverage, during that test-renderer-shiny code runs twice. To avoid double-running, I use testthat::test_file('test-shiny.R') directly.

tdhock · 2025-08-28T13:15:08Z

lter = 'shiny') for coverage, during that test-renderer-shiny code runs twice. To avoid double-running, I use testthat::test_file('test-shiny.R') directly.

if we put shiny tests in test-renderer-shiny.R then it should be possible to run only filter="renderer" and then we don't have to run filter="shiny" separately.

it should be possible to test shiny without making it a special case, which would be preferable for simplicity.

biplab-sutradhar · 2025-08-28T14:50:27Z

It gets stuck here, and even after rerunning, the issue remains the same. However, it works fine locally.

tests_run(filter = "interactivity") 
[1] "interactivity" 
[1] "tooltip-interactivity" 
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 45 ]

tdhock · 2025-08-28T15:43:09Z

@suhaani-agarwal do you know why this would happen and what we could try to fix?

suhaani-agarwal · 2025-08-29T11:00:09Z

@suhaani-agarwal do you know why this would happen and what we could try to fix?

I have rerun the test to investigate whats wrong, looking into it...

suhaani-agarwal · 2025-08-29T12:02:59Z

Currently, the Shiny tests are also collecting JS coverage. The issue is that Shiny tests are inherently heavy, and running JS coverage simultaneously adds significant overhead because every JS evaluation, DOM update, and animation gets tracked. Since the Shiny tests were previously executed before the other renderer tests, this caused Chromote to hang during the subsequent renderer tests.

To resolve this, I have renamed the Shiny test file to test-renderer4-shiny.R, which ensures that the Shiny tests run last. With this change, the tests now execute without hanging or getting stuck, and the coverage collection completes successfully.

suhaani-agarwal · 2025-08-29T12:12:39Z

I think the Shiny tests could be made a bit lighter - there are Sys.sleep calls of 10–20 seconds. Do we need such long pauses?

Also, the JS coverage conversion doesn’t seem to be happening correctly. This isn’t related to the hanging issue; it’s likely due to changes in the v8-to-istanbul file or in stop_js_coverage(). The coverage collection itself is working correctly, and the js-coverage.json file is being saved as expected:

JS coverage saved to /home/runner/work/animint2/animint2/tests/testthat/js-coverage.json

It’s possible that the Shiny tests are still overwriting the JS coverage file, but I’m not certain.

tdhock · 2025-08-29T12:12:43Z

JS coverage still fails on Github actions?
is a possible solution running shiny tests separately from renderer?

biplab-sutradhar · 2025-08-31T17:53:37Z

is a possible solution running shiny tests separately from renderer?

using testthat::test_file('test-shiny.R') separately was working so it can be added

biplab-sutradhar · 2025-09-04T14:24:50Z

@tdhock What should I do?

tdhock · 2025-09-04T16:12:40Z

please try to investigate #209 (comment) and respond

biplab-sutradhar · 2025-09-05T16:15:29Z

@tdhock Extracting animint.js coverage entries and replacing their URLs with the file path let handle both Shiny and static test servers and now JS coverage is not failing

biplab-sutradhar added 9 commits July 6, 2025 19:12

two visualization moved from rselenium to chromote

f7a093e

Add RMarkdown app tests

f26a3a2

Update GitHub Actions workflow to include Shiny test suite and moved …

6e8d467

…helper functions for shiny, RMarkdown app to helper-function.R

testing helper functions

a69605d

Refactor app startup function

fbb0d6a

Add library call for animint2 in shiny test execution

da9bd9c

moved helper functions to helper-functions.R

13c4469

Merge branch 'master' of github.com:animint/animint2 into test/rselen…

3562dbe

…ium_to_chromote

refactor: test.yaml file

0275814

biplab-sutradhar added 4 commits July 11, 2025 02:32

prevent skipping files

b4addeb

refactor code to get correct file directory

944c4d6

install packages this.path

5915e13

update test.yaml

dac3fc8

tdhock reviewed Jul 11, 2025

View reviewed changes

refactor code to prevent test failures

7b41cad

tdhock reviewed Jul 14, 2025

View reviewed changes

R/z_knitr.R Outdated

#' @export

#'

Copy link

Collaborator

tdhock Jul 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please undo addition of empty spaces

tdhock reviewed Jul 14, 2025

View reviewed changes

biplab-sutradhar added 2 commits July 14, 2025 19:42

removed empty lines and undo comments

c161768

removed redundant function definition

68d8924

suhaani-agarwal reviewed Jul 27, 2025

View reviewed changes

suhaani-agarwal mentioned this pull request Jul 27, 2025

Combining renderer and compiler tests #212

Merged

suhaani-agarwal reviewed Jul 29, 2025

View reviewed changes

biplab-sutradhar added 6 commits August 27, 2025 21:21

fix: correct typo in stop_js_coverage function

8a408d0

refactor: remove unused Shiny coverage processing code

9e7c032

refactor: enhance JS coverage handling for Shiny tests

99537ca

Merge branch 'master' of github.com:animint/animint2 into test/rselen…

b58e032

…ium_to_chromote

refactor: improve JS coverage processing in v8-to-istanbul.js and hel…

0bfe931

…per-HTML.R

refactor shiny test coverage

bf35fbb

refactor for js coverage

f16f8a8

added single unified test and coverage workflow

e3979c4

trying to run shiny tests at the end in renderer tests

4714f79

tdhock changed the title ~~Migration of RSelenium test to chromote test~~ shiny tests using chromote Aug 29, 2025

tdhock mentioned this pull request Aug 29, 2025

shiny example errors #218

Open

biplab-sutradhar added 6 commits September 4, 2025 23:27

refactor coverage

6a592bd

refactor js coverage

6f09f27

refactor js coverage

7ac7203

refactor stop_js_coverage function

5e11b3c

delay time

dadb0ce

undo versions

7f86cb8

shiny tests using chromote #209

Are you sure you want to change the base?

shiny tests using chromote #209

Uh oh!

Conversation

biplab-sutradhar commented Jul 7, 2025

Uh oh!

codecov bot commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

biplab-sutradhar commented Jul 11, 2025 via email • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

biplab-sutradhar commented Jul 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tdhock commented Jul 27, 2025

Uh oh!

tdhock commented Jul 27, 2025

Uh oh!

suhaani-agarwal commented Jul 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

biplab-sutradhar commented Jul 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

biplab-sutradhar commented Aug 28, 2025

Uh oh!

biplab-sutradhar commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tdhock commented Aug 28, 2025

Uh oh!

biplab-sutradhar commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tdhock commented Aug 28, 2025

Uh oh!

suhaani-agarwal commented Aug 29, 2025

Uh oh!

suhaani-agarwal commented Aug 29, 2025

Uh oh!

suhaani-agarwal commented Aug 29, 2025

Uh oh!

tdhock commented Aug 29, 2025

Uh oh!

biplab-sutradhar commented Aug 31, 2025

Uh oh!

biplab-sutradhar commented Sep 4, 2025

Uh oh!

tdhock commented Sep 4, 2025

Uh oh!

biplab-sutradhar commented Sep 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Jul 10, 2025 •

edited

Loading

biplab-sutradhar commented Jul 11, 2025 via email •

edited

Loading

biplab-sutradhar commented Jul 25, 2025 •

edited

Loading

suhaani-agarwal commented Jul 27, 2025 •

edited

Loading

biplab-sutradhar commented Aug 28, 2025 •

edited

Loading

biplab-sutradhar commented Aug 28, 2025 •

edited

Loading