Skip to content

Comments

dev: updates bulkexport script to batch exports#3125

Open
gabestein wants to merge 22 commits intomainfrom
gs/batch-export
Open

dev: updates bulkexport script to batch exports#3125
gabestein wants to merge 22 commits intomainfrom
gs/batch-export

Conversation

@gabestein
Copy link
Member

@gabestein gabestein commented Jul 11, 2024

Issue(s) Resolved

  • Batches collection exports so they don't take up all the available db connections
  • Excludes pubs with no releases from exports by default

Test Plan

  • Export a large collection using the CLI tool
  • See that when exports are missing and need to be generate, it only tries to do ~5 at a time

Screenshots (if applicable)

Optional

Notes/Context/Gotchas

Supporting Docs

@gabestein gabestein requested a review from tefkah July 11, 2024 20:05
@pubpubBot pubpubBot temporarily deployed to pubpub-pipel-gs-batch-e-wormby July 11, 2024 20:19 Inactive
@pubpubBot pubpubBot temporarily deployed to pubpub-pipel-gs-batch-e-wormby July 11, 2024 21:22 Inactive
@gabestein gabestein changed the title updates bulkexport script to batch exports dev: updates bulkexport script to batch exports Jul 16, 2024
@pubpubBot pubpubBot temporarily deployed to pubpub-pipel-gs-batch-e-ldphaz September 10, 2024 22:37 Inactive
gabestein and others added 18 commits September 20, 2024 11:18
@gabestein
still not perfect but better
fa081c0
@gabestein @isTravis
update: Langing page for Platform (#3129)
31a8af9 
* redo landing page and remove some old routes

* get rid of landing page lint errors

* remove community creation links

* require superadmin to create community

* update links to kf.org

* Adjust jumbotron height

* update footer link

* fix tests

* better test fix

* ok actually do real tests fine gabe does tod now

* sigh spell it right

* I love tests

---------

Co-authored-by: Travis Rich <travis.s.rich@gmail.com>
@gabestein
debounce user query and prevent null (#3133)
3245366
@gabestein
debounce collection search (#3136)
66c432f
@gabestein
dev: Remove TOS banner (#3176)
3538520
@gabestein
use preprint for arcadia (#3173)
3a8fcca
@gabestein
use timestamp for crossref deposits (#3199)
ee833d8
@tefkah @gabestein
feat: add code attrs such as lang to statically rendered html code bl…
1814a82 
…ocks (#3206)

* feat: add code attrs such as lang to statically rendered html code blocks

* fix: use data-lang instead of lang
@gabestein
don't strip iframe url if target is html (#3207)
57c2d6a
@gabestein
return public comments from querymany if pub has release (#3209)
f8ba23f
@isTravis @gabestein
fix: reerse ordering of wrapped marks in static export (#3221)
bd07e03
@gabestein
add redirects (#3228)
f10a1e5
@isTravis @gabestein
update: Add hideLabel value to output HTML (#3227)
aab05e8
@3mcd @gabestein
chore: upgrade heroku stack (#3235)
93e54ac 
* chore: upgrade heroku stack

* chore: try heroku 22

* chore: remove repo specifier

* chore: back to 24

* chore: deploy commit
@tefkah @gabestein
feat: add read replicas (#3236)
9a056cf 
* feat: add connection pooling

* fix: make sure DATABASE_READ_REPLICA_1 is set

* fix: actually set read replica env var correctly during test

* fix: always set the read_replica_db to the main db in tests

* fix: require second read replica
@tefkah @gabestein
fix: make scopesummaries always use master (#3249)
1205cc2
@tefkah @gabestein
fix: fix many read replica issues (#3251)
7125ba5 
* fix: fix signup flow (ideally)

* fix: fix activity items

* fix: fix pubAttribution error

* fix: fix createThreadComment

* fix: useMaster for increment attemptCount
@erkannt @davidcmoulton @gabestein
Attempt to add the browse page link to eLife navigation (#3258)
87db0ef 
elifesciences/issues#9146

Co-authored-by: David Moulton <d.moulton@elifesciences.org>
@pubpubBot pubpubBot temporarily deployed to pubpub-pipel-gs-batch-e-tugfgs April 1, 2025 19:57 Inactive
@pubpubBot pubpubBot temporarily deployed to pubpub-pipel-gs-batch-e-tugfgs April 1, 2025 20:25 Inactive
@gabestein gabestein mentioned this pull request Apr 17, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tefkah tefkah Awaiting requested review from tefkah

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@gabestein @pubpubBot @tefkah @isTravis @3mcd @erkannt