Skip to content

Add ENABLE_PARTIAL_ASYNC_IMPORT variable #121

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Qwerty5Uiop wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add ENABLE_PARTIAL_ASYNC_IMPORT variable #121

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions setup/env-variables/backend-env-variables.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@
| `HACKNEY_DEFAULT_POOL_SIZE` | Size of `default` hackney pool. Implemented in [#12406](https://github.com/blockscout/blockscout/pull/12406). | Version: v8.1.0\+ <br />Default: `1000` <br />Applications: API, Indexer |
| `UNIVERSAL_PROXY_CONFIG_URL` | Config URL for universal proxy. Implemented in [#12442](https://github.com/blockscout/blockscout/pull/12442). | Version: v8.1.0\+ <br />Default: `https://raw.githubusercontent.com/blockscout/backend-configs/refs/heads/main/universal-proxy-config.json` <br />Applications: API |
| `UNIVERSAL_PROXY_CONFIG` | JSON-encoded configuration string used to define settings for the universal proxy. Implemented in [#13787](https://github.com/blockscout/blockscout/pull/13787). | Version: v10.0.0\+ <br />Default: (empty) <br />Applications: API |
| `K8S_SERVICE` | Kubernetes service name for Elixir nodes clusterization, more info on how to configure it can be found here https://hexdocs.pm/libcluster/Cluster.Strategy.Kubernetes.DNS.html. Implemented in [#13080](https://github.com/blockscout/blockscout/pull/13080). | Version: v9.2.0\+ <br />Default: (empty) <br />Applications: API, Indexer |

Check warning on line 144 in setup/env-variables/backend-env-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (blockscout) - vale-spellcheck

setup/env-variables/backend-env-variables.mdx#L144 

Did you really mean 'clusterization'?

## Database Management

Expand Down Expand Up @@ -274,7 +274,7 @@
| `INDEXER_RECEIPTS_BATCH_SIZE` | Batch size for transaction receipts fetcher. Implemented in [#6454](https://github.com/blockscout/blockscout/pull/6454). | Version: v5.0.0\+ <br />Default: `250` <br />Applications: Indexer |
| `INDEXER_RECEIPTS_CONCURRENCY` | Concurrency for transaction receipts fetcher. Implemented in [#6454](https://github.com/blockscout/blockscout/pull/6454). | Version: v5.0.0\+ <br />Default: `10` <br />Applications: Indexer |
| `INDEXER_SIGNED_AUTHORIZATION_STATUS_BATCH_SIZE` | Batch size (number of blocks) for EIP7702 authorizations status fetcher. Implemented in [#12451](https://github.com/blockscout/blockscout/pull/12451). | Version: v9.0.0\+ <br />Default: `10` <br />Applications: Indexer |
| `INDEXER_FHE_OPERATIONS_ENABLED` | Flag to enable parsing of Fully Homomorphic Encryption (FHE) operations from transactions. Implemented in [#13742](https://github.com/blockscout/blockscout/pull/13742). | Version: v11.0.0\+ <br />Default: `false` <br />Applications: Indexer |

Check warning on line 277 in setup/env-variables/backend-env-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (blockscout) - vale-spellcheck

setup/env-variables/backend-env-variables.mdx#L277 

Did you really mean 'Homomorphic'?
| `INDEXER_SYSTEM_MEMORY_PERCENTAGE` | Percentage of total memory available to the VM that an application can use if `INDEXER_MEMORY_LIMIT` is not set. Implemented in [#10697](https://github.com/blockscout/blockscout/pull/10697). | Version: v6.9.0\+ <br />Default: `60` <br />Applications: Indexer |
| `INDEXER_ARCHIVAL_TOKEN_BALANCES_BATCH_SIZE` | Batch size for archival token balances fetcher. Implemented in [#7439](https://github.com/blockscout/blockscout/pull/7439). | Version: v10.0.0\+ <br />Default: `100` <br />Applications: Indexer |
| `INDEXER_ARCHIVAL_TOKEN_BALANCES_CONCURRENCY` | Concurrency for archival token balances fetcher. Implemented in [#8167](https://github.com/blockscout/blockscout/pull/8167). | Version: v10.0.0\+ <br />Default: `10` <br />Applications: Indexer |
Expand Down Expand Up @@ -303,6 +303,7 @@
| `INDEXER_CURRENT_TOKEN_BALANCES_BATCH_SIZE` | Batch size for current token balances fetcher. Implemented in [#13923](https://github.com/blockscout/blockscout/pull/13923). | Version: v10.0.0\+ <br />Default: `100` <br />Applications: Indexer |
| `INDEXER_CURRENT_TOKEN_BALANCES_CONCURRENCY` | Concurrency for current token balances fetcher. Implemented in [#13923](https://github.com/blockscout/blockscout/pull/13923). | Version: v10.0.0\+ <br />Default: `10` <br />Applications: Indexer |
| `INDEXER_CURRENT_TOKEN_BALANCES_IMPORT_CHUNK_SIZE` | Number of CurrentTokenBalances items processed per chunk in token balances import. Default is 50; effective minimum is 1. Implemented in [#14250](https://github.com/blockscout/blockscout/pull/14250). | Version: v11.0.0\+ <br />Default: `50` <br />Applications: Indexer |
| `ENABLE_PARTIAL_ASYNC_IMPORT` | If `true`, addresses, current token balances, tokens and token instances are imported asynchronously. Implemented in [#14277](https://github.com/blockscout/blockscout/pull/14277). | Version: master <br />Default: `false` <br />Applications: Indexer |

## DB Migrations & Sanitizer Management

Expand Down Expand Up @@ -520,21 +521,21 @@
## CSV Export

<Info>
When `CSV_EXPORT_ASYNC_ENABLED` is `true`, supported CSV export endpoints create a tracked request, enqueue a background job, write the CSV to a temporary file, upload it to Gokapi, and return a `request_id` for polling instead of streaming the CSV directly in the HTTP response.

Check warning on line 524 in setup/env-variables/backend-env-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (blockscout) - vale-spellcheck

setup/env-variables/backend-env-variables.mdx#L524 

Did you really mean 'Gokapi'?
</Info>

| Variable | Description | Parameters |
| ------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| `CSV_EXPORT_LIMIT` | Changes the max number of rows pulled by the CSV export logic. Applies to both sync and async CSV export paths. Implemented in [#10497](https://github.com/blockscout/blockscout/pull/10497) | Version: v6.8.0\+ <br />Default: `10000` <br />Applications: API |
| `CSV_EXPORT_ASYNC_ENABLED` | Enables async CSV export for supported endpoints. When enabled, the API returns `202 Accepted` with a `request_id` and processes exports through Oban instead of streaming them directly. Implemented in [#14028](https://github.com/blockscout/blockscout/pull/14028) | Version: v11.0.0\+ Required: No <br />Default: `false` <br />Applications: API |

Check warning on line 530 in setup/env-variables/backend-env-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (blockscout) - vale-spellcheck

setup/env-variables/backend-env-variables.mdx#L530 

Did you really mean 'Oban'?
| `CSV_EXPORT_ASYNC_OBAN_CONCURRENCY` | Sets Oban concurrency for the `csv_export` queue used by async CSV exports. Implemented in [#14028](https://github.com/blockscout/blockscout/pull/14028) | Version: v11.0.0\+ Required: No <br />Default: `10` <br />Applications: API |

Check warning on line 531 in setup/env-variables/backend-env-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (blockscout) - vale-spellcheck

setup/env-variables/backend-env-variables.mdx#L531 

Did you really mean 'Oban'?
| `CSV_EXPORT_ASYNC_GOKAPI_URL` | Base URL of the Gokapi instance used to store completed async CSV exports. Trailing slash is stripped during validation. Implemented in [#14028](https://github.com/blockscout/blockscout/pull/14028) | Version: v11.0.0\+ Required: Yes, if async export is enabled <br />Default: (empty) <br />Applications: API |

Check warning on line 532 in setup/env-variables/backend-env-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (blockscout) - vale-spellcheck

setup/env-variables/backend-env-variables.mdx#L532 

Did you really mean 'Gokapi'?
| `CSV_EXPORT_ASYNC_GOKAPI_API_KEY` | API key sent to Gokapi in the `apikey` header for async CSV export uploads. Implemented in [#14028](https://github.com/blockscout/blockscout/pull/14028) | Version: v11.0.0\+ Required: Yes, if async export is enabled <br />Default: (empty) <br />Applications: API |

Check warning on line 533 in setup/env-variables/backend-env-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (blockscout) - vale-spellcheck

setup/env-variables/backend-env-variables.mdx#L533 

Did you really mean 'Gokapi'?
| `CSV_EXPORT_ASYNC_MAX_PENDING_TASKS_PER_IP` | Maximum number of pending async CSV export requests allowed per client IP at once. Implemented in [#14028](https://github.com/blockscout/blockscout/pull/14028) | Version: v11.0.0\+ Required: No <br />Default: `3` <br />Applications: API |
| `CSV_EXPORT_ASYNC_UPLOAD_CHUNK_SIZE` | Chunk size in bytes for reading the generated CSV file and uploading it to Gokapi. Should be synchronized with Gokapi settings. Implemented in [#14028](https://github.com/blockscout/blockscout/pull/14028) | Version: v11.0.0\+ Required: No <br />Default: `47185920` <br />Applications: API |

Check warning on line 535 in setup/env-variables/backend-env-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (blockscout) - vale-spellcheck

setup/env-variables/backend-env-variables.mdx#L535 

Did you really mean 'Gokapi'?

Check warning on line 535 in setup/env-variables/backend-env-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (blockscout) - vale-spellcheck

setup/env-variables/backend-env-variables.mdx#L535 

Did you really mean 'Gokapi'?
| `CSV_EXPORT_DB_TIMEOUT` | Timeout for CSV export database work. Follows the [time format](/setup/env-variables/backend-env-variables#time-format). Implemented in [#14028](https://github.com/blockscout/blockscout/pull/14028) | Version: v11.0.0\+ Required: No <br />Default: `1h` if async export is enabled, otherwise `5m` <br />Applications: API |
| `CSV_EXPORT_ASYNC_TMP_DIR` | Directory used for in-progress async CSV export files before they are uploaded to Gokapi. Implemented in [#14028](https://github.com/blockscout/blockscout/pull/14028) | Version: v11.0.0\+ Required: No <br />Default: `/tmp/csv_export` <br />Applications: API |

Check warning on line 537 in setup/env-variables/backend-env-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (blockscout) - vale-spellcheck

setup/env-variables/backend-env-variables.mdx#L537 

Did you really mean 'Gokapi'?
| `CSV_EXPORT_ASYNC_GOKAPI_TIMEOUT` | HTTP timeout and `recv_timeout` used for Gokapi requests during async CSV export. Follows the [time format](/setup/env-variables/backend-env-variables#time-format). Implemented in [#14028](https://github.com/blockscout/blockscout/pull/14028) | Version: v11.0.0\+ Required: No <br />Default: `60s` <br />Applications: API |

Check warning on line 538 in setup/env-variables/backend-env-variables.mdx

View check run for this annotation

Mintlify / Mintlify Validation (blockscout) - vale-spellcheck

setup/env-variables/backend-env-variables.mdx#L538 

Did you really mean 'Gokapi'?
| `CSV_EXPORT_ASYNC_GOKAPI_UPLOAD_EXPIRY_DAYS` | Sets Gokapi `expiryDays` for completed async CSV export uploads. Implemented in [#14028](https://github.com/blockscout/blockscout/pull/14028) | Version: v11.0.0\+ Required: No <br />Default: `1` <br />Applications: API |
| `CSV_EXPORT_ASYNC_GOKAPI_UPLOAD_ALLOWED_DOWNLOADS` | Sets Gokapi `allowedDownloads` for completed async CSV export uploads. Implemented in [#14028](https://github.com/blockscout/blockscout/pull/14028) | Version: v11.0.0\+ Required: No <br />Default: `1` <br />Applications: API |

Expand Down