Skip to content

Conversation

@lee-hodg
Copy link

Implements

  • Allow custom headers setting, which is useful for some sites you may wish to scrape
  • Also when doing a crawl we may want the process url to have the &norender parameter to stop javascript rendering to speed up performance of the crawl, so here allow for that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant