Skip to content

Add 'requester pays' options for data downloads #48

@chadokruse

Description

@chadokruse
  • Create new bucket and turn requester pays option to 'on' (docs)
  • Add additional step to deploy scripts to also send data to requester pays bucket
  • Detailed README with instructions (including limitations of dataset)

Size of files (gzipped):
1.5GB Normalized filings (2011-current)
0.2GB Grants (2011-current)

Storage cost negligible (only $0.05 / mo...$0.026 / GB)
Network costs and egress fees are the concern (fee schedule)


Alternative:

  1. Use G-Suite storage (API docs)
    Q: What are the consequences if a developer constantly hammers the endpoint(s) programmatically?

  2. Use a cloud function to disallow programmatic access. Would allow good actors access, but prevent runaway costs from programmatic hammering of endpoint.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions