Skip to content

Rework migrate data docs (Take 2)#5244

Open
kilfoyle wants to merge 24 commits intoelastic:mainfrom
kilfoyle:4353/rework-ech2ess-mig-v2
Open

Rework migrate data docs (Take 2)#5244
kilfoyle wants to merge 24 commits intoelastic:mainfrom
kilfoyle:4353/rework-ech2ess-mig-v2

Conversation

@kilfoyle
Copy link
Contributor

@kilfoyle kilfoyle commented Feb 20, 2026

This is a redo of #4914 to reorganize the "Migrate your Elasticsearch data" section of the docs. It:

Note: one source file was renamed so a few of the files here are included just for link fixes.


Closes: #4353
Rel: https://github.com/elastic/docs-team/issues/30
Rel: https://github.com/elastic/docs-content-internal/issues/467

Generative AI disclosure

  1. Did you use a generative AI (GenAI) tool to assist in creating this contribution?
  • Yes
  • No

@github-actions
Copy link
Contributor

github-actions bot commented Feb 20, 2026

Vale Linting Results

Summary: 2 warnings, 2 suggestions found

⚠️ Warnings (2)
File Line Rule Message
manage-data/migrate-data.md 55 Elastic.Spelling 'Snaphot' is a possible misspelling.
manage-data/migrate/migrate-data-using-reindex-api.md 69 Elastic.Spelling 'multiiple' is a possible misspelling.
💡 Suggestions (2)
File Line Rule Message
deploy-manage/deploy/cloud-enterprise/create-deployment.md 50 Elastic.Semicolons Use semicolons judiciously.
manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md 53 Elastic.WordChoice Consider using 'can, might' instead of 'may', unless the term is in the UI.

The Vale linter checks documentation changes against the Elastic Docs style guide.

To use Vale locally or report issues, refer to Elastic style guide for Vale.


The [reindex API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex) offers a convenient way for you to migrate your {{es}} documents from a source index, data stream, or alias in one deployment to another.

This guide gives the example of reindexing a full index from an {{ech}} deployment an {{serverless-full}} project using the remote host parameters as shown in the [reindex from remote](elasticsearch://reference/elasticsearch/rest-apis/reindex-indices.md#reindex-from-remote) example. These steps can be adapted to other deployment types according to the tables listed in [User data migration guides](/manage-data/migrate-data.md#data-migration-guides).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"from an ECH deployment to an Elastic Cloud Serverless project"


For more advanced use cases, including data modification using scripts or ingest pipelines, refer to the [Reindex indices examples](elasticsearch://reference/elasticsearch/rest-apis/reindex-indices.md#reindex-from-remote) and the [reindex API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex) documentation.

:::{tip}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This tip doesn't apply for serverless destinations (as serverless currently only supports ECH sources). This is a general page and the example just happens to be ECH -> ECS, so the tip itself is still valid. But there's something about the placement right here that causes a bit of a context switch for me - or might tend to have readers expecting that it could apply in the example. I wonder if it could fit better somewhere else - maybe even back on the main migration page.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point! I've moved this and the whitelisting guidance to a "Notes for migrating between other deployment types" section at the bottom of the page. On the main "Migrate your Elasticsearch Data" page there's also a note guiding people using private CA to the page for that.


Basic authentication can be used in place of an API key, but an API key is recommended as a more secure option.

- The target deployment must be able to access your original source cluster to perform the reindex operation. Access is controlled by the {{es}} `reindex.remote.whitelist` user setting.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This section does not apply for serverless destinations - in serverless, we automatically allow access to all ECH endpoints as source. But this is great information and it needs to be kept somewhere. Do you think we could take this (and perhaps the tip about the certificates above) and shift it to a different section of this page? Somewhere clearly delineated from the guided example?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For sure! I've put this in a new "Notes for migrating between other deployment types" section at the bottom of the page, along with the note for folks using private CA.

:::{important}
Kibana assets must be migrated separately using the {{kib}} [export/import APIs](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-saved-objects) or recreated manually. Refer to [Migration options](/manage-data/migrate-data.md#migration-options) for details about migrating different types of {{es}} data.

Index templates, data stream definitions, and ILM policies, must be in place _before_ you start reindexing data. However, be mindful that if you have any [ingest pipelines](/manage-data/ingest/transform-enrich/ingest-pipelines.md) configured, it's typically best to add these _after_ data migration so as to avoid re-transforming data that had already been transformed at the time that it was ingested into your source deployment (though if the data is idempotent, re-transforming is not a concern).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ILM does not apply for serverless - perhaps we should mention DLM too? Or just generalize with something like "data lifecycle settings" ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to Index templates, data stream definitions, and data lifecycle settings must be in place...

@kilfoyle
Copy link
Contributor Author

kilfoyle commented Mar 2, 2026

Thanks for the superb suggestions @pete-naylor! All fixed. :-)

Copy link
Collaborator

@shainaraskas shainaraskas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

really great work.

I think that there are some surprisingly quick wins left in terms of recontextualizing the tutorials that currently only look at serverless/ech ... otherwise, some super nice improvements and a win for migrators everywhere.

left a bunch of comments so you prob want a re-review after taking a look. I am super slow so don't consider getting my review to be a requirement (but I can come do another round if you like)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this title was too gory ... we might avoid renaming it to retain some seo signals even if the title was not perfectly ideal?

## Step 5: Reindex from remote `Source` cluster [ec-remote-reindex-step5]

You can now run a remote reindex operation on the {{ech}} `Destination` cluster from the `Source` cluster, as described in the [migration guide](/manage-data/migrate.md#ech-reindex-remote):
You can now run a `reindex from remote` reindex operation on the {{ech}} `Destination` cluster from the `Source` cluster, as described in [Migrate Elasticsearch data using the reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would leave these blank so the titles stay in sync

Suggested change
You can now run a `reindex from remote` reindex operation on the {{ech}} `Destination` cluster from the `Source` cluster, as described in [Migrate Elasticsearch data using the reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md):
You can now run a `reindex from remote` reindex operation on the {{ech}} `Destination` cluster from the `Source` cluster, as described in [](/manage-data/migrate/migrate-data-using-reindex-api.md):

## Migrate system indices using snapshot and restore

To restore system indices from a snapshot, follow the same procedure described in [](../migrate.md#ec-restore-snapshots) and select the appropriate feature states when preparing the restore operation, such as `kibana` or `security`.
To restore system indices from a snapshot, follow the same procedure described in [Migrate Elasticsearch data with minimal downtime using snapshots](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md) and select the appropriate feature states when preparing the restore operation, such as `kibana` or `security`.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
To restore system indices from a snapshot, follow the same procedure described in [Migrate Elasticsearch data with minimal downtime using snapshots](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md) and select the appropriate feature states when preparing the restore operation, such as `kibana` or `security`.
To restore system indices from a snapshot, follow the same procedure described in [](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md) and select the appropriate feature states when preparing the restore operation, such as `kibana` or `security`.


# Migrate your {{es}} data [migrate-your-elasticsearch-data]

Transitioning between Elastic deployment types involves migrating your {{es}} data. This page helps you plan your migration by describing the main categories of data you may need to move (ingested user data, {{es}} system data, {{kib}} saved objects, and feature-specific data), the migration methods available for each, and where to find step-by-step guides for your scenario.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is close enough to the next section that i'd skip the parenthetical personally

Suggested change
Transitioning between Elastic deployment types involves migrating your {{es}} data. This page helps you plan your migration by describing the main categories of data you may need to move (ingested user data, {{es}} system data, {{kib}} saved objects, and feature-specific data), the migration methods available for each, and where to find step-by-step guides for your scenario.
Transitioning between Elastic deployment types involves migrating your {{es}} data. This page helps you plan your migration by describing the main categories of data you may need to move, the migration methods available for each, and where to find step-by-step guides for your scenario.


- **Ingested user data**: All of the data that you've added into {{es}}, structured or unstructured, for your own applications.

- **{{es}} system data**: Configuration and state information stored in {{es}} [system indices](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#system-indices). This data is used by {{es}} for its internal operations.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when I read "configuration data" I assume things like users might be involved, but that looks like it's feature/component data. can we be more precise here?

Comment on lines +106 to +108
- If you're migrating from a self-managed cluster that uses non–publicly trusted TLS certificates, including self-signed certificates and certificates signed by a private certificate authority (CA), refer to our guide [Reindex from a self-managed cluster using a private CA](/manage-data/migrate/migrate-from-a-self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex.md).

- The target deployment must be able to access your original source cluster to perform the reindex operation. When you migrate to {{serverless-short}}, access to all {{ech}} endpoints is allowed automatically. For migrating to other deployment types, access is controlled by the {{es}} `reindex.remote.whitelist` user setting.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am surprised there are so few "differences" here. it would really improve this tutorial to have an "ECH -> serverless" and "anything -> anything" pathway because there are so few differences. I think that the prerequisites could easily be generalized.

thinking harder about this, it just feels like there is one non-serverless step that needs to happen before the reindex call. could we just make this a tutorial with two major steps?

we could then turn the console step into a tabbed experience with one tab for serverless and one for everything else (if the API calls are different)

navigation_title: Migrate data using Logstash
applies_to:
serverless:
deployment:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

switch to stack?

# Migrate {{ech}} data to {{serverless-full}} with {{ls}} [migrate-with-ls]
# Migrate {{es}} data using {{ls}} [migrate-with-ls]

[{{ls}}](logstash://reference/index.md) is a data collection engine that uses a large ecosystem of [plugins](logstash-docs-md://lsr/index.md) to collect, process, and forward data from a variety of sources to a variety of destinations. Here we focus on using the [Elasticsearch input](logstash-docs-md://lsr/plugins-inputs-elasticsearch.md) plugin to read from your {{ech}} deployment, and the [Elasticsearch output](logstash-docs-md://lsr/plugins-outputs-elasticsearch.md) plugin to write to your {{{serverless-full}} project.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you update the admonition on line 19 to make it clearer that it's about deployment types? the title "basic migration" isn't very meaningful

Kibana assets much be migrated separately using the {{kib}} [export/import APIs](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-saved-objects) or recreated manually. Refer to [Migration options](/manage-data/migrate-data.md#migration-options) for details about migrating different types of {{es}} data.
Templates, data stream definitions, and ILM policies, must be in place _before_ you start data migration.

Visual components, such dashboard and visualizations, can be migrated after you have migrated the data.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

again, lower on this page, we only have 3 config options called out. I wonder how specific to ech/serverless this tutorial actually is, and how it could be generalized. feels like it wouldn't need much at all.

might be worth considering as our only link we are using on the migration overview page.


## User data migration guides [data-migration-guides]

To migrate your {{es}} ingested user data, choose one of the available migration options depending on your source and target deployment types.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

consider hinting that some of these tutorials use specific deployment types as examples, but can be altered for your scenario (less shock/disorientation)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Migration] Rework and expand data migration section

3 participants