Remove intermediate files from updating-sources
#14
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
curl (instead of wget) and streaming decompressors & processors allow creating the final files without actually needing intermediate artefacts and needing to clean them up, so when possible do that. Update `kym_scrape` to not need `requests`, and also to shove the phrases to stdout directly, and move the update tracking to stderr, that way it can also be used in a "streaming" manner. The log isn't really useful anymore as the stdout essentially provides the same tracking (just deduplicated). POI can't be made streaming in a portable manner: *some* variants of unzip apparently allow passing an archive as stdin but not all, KSH-style process substitution opens a named pipe which generic unzip also dislike, and ZSH's `=` variant works but is ZSH-specific (also the lack of feedback is a bit disturbing, as it needs to download the entire content into a temporary file which it can then hand out, and the POI dataset is quite large). However we can use the `-p` feature to avoid the intermediate `allCountries.txt`.
Author
|
Nota: if you're using a tool which supports PEP 723 (pdm, uv, ...) then it might make sense to put requests back in (probably using a Also not sure if it changed since the script was created but it looks like KYM does not really have a page 0, it's the same content as page 1, I think KYM just interprets it as not specifying a page which is why it's not a 404, so might make sense to start at 1. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
curl (instead of wget) and streaming decompressors & processors allow creating the final files without actually needing intermediate artefacts and needing to clean them up, so when possible do that.
Update
kym_scrapeto not needrequests, and also to shove the phrases to stdout directly, and move the update tracking to stderr, that way it can also be used in a "streaming" manner. The log isn't really useful anymore as the stdout essentially provides the same tracking (just deduplicated).POI can't be made streaming in a portable manner: some variants of unzip apparently allow passing an archive as stdin but not all, KSH-style process substitution opens a named pipe which generic unzip also dislike, and ZSH's
=variant works but is ZSH-specific (also the lack of feedback is a bit disturbing, as it needs to download the entire content into a temporary file which it can then hand out, and the POI dataset is quite large). However we can use the-pfeature to avoid the intermediateallCountries.txt.