Add info on how to manually apply dlq rows #21440

peachdawnleach · 2025-12-02T18:56:29Z

Fixes DOC-15407

Added info from Confluence on how to manually apply dlq rows

Added section on how to manually resolve entries in the dlq

typo

github-actions · 2025-12-02T18:56:54Z

Files changed:

netlify · 2025-12-02T18:56:55Z

✅ Deploy Preview for cockroachdb-interactivetutorials-docs canceled.

Name	Link
🔨 Latest commit	`ae3ab5a`
🔍 Latest deploy log	https://app.netlify.com/projects/cockroachdb-interactivetutorials-docs/deploys/693877635ef52b0008fcdc73

netlify · 2025-12-02T18:56:57Z

✅ Deploy Preview for cockroachdb-api-docs canceled.

Name	Link
🔨 Latest commit	`ae3ab5a`
🔍 Latest deploy log	https://app.netlify.com/projects/cockroachdb-api-docs/deploys/69387763fb10730009a5b6f6

netlify · 2025-12-02T19:11:40Z

✅ Netlify Preview

Name	Link
🔨 Latest commit	`ae3ab5a`
🔍 Latest deploy log	https://app.netlify.com/projects/cockroachdb-docs/deploys/69387763a81bde00089cf6e8
😎 Deploy Preview	https://deploy-preview-21440--cockroachdb-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

peachdawnleach · 2025-12-02T19:12:58Z

Preview link: https://deploy-preview-21440--cockroachdb-docs.netlify.app/docs/stable/manage-logical-data-replication#dead-letter-queue-dlq

msbutler · 2025-12-05T15:14:11Z

src/current/v25.4/manage-logical-data-replication.md

+    106677386757203 | 2025-04-25 25:32:28.435439+00 | {"created_at": "2025-04-25:35:00.499499", "payload": "blahblahblah=", "my_id": 207}
+    ~~~
+
+1. Check the value of the row and the replicated time:


nit: like in the kb, it might be clearer to the user if you state what the purpose of the next 3 steps are before showing the 3 steps: e.g. Check if the value of the row on source and destination are the same. If they are, the DLQ’d row can be removed:

Added nested steps and ported to 26.1

netlify · 2025-12-08T15:33:41Z

❌ Deploy Preview for cockroachdb-docs-noble-test failed. Why did it fail? →

Name	Link
🔨 Latest commit	`de022e2`
🔍 Latest deploy log	https://app.netlify.com/projects/cockroachdb-docs-noble-test/deploys/6936efd0428436000877335d

…om:cockroachdb/docs into 20251125-doc-15407-manually-apply-dlq-rows

jhlodin

Left some feedback to help clarify the process and what objects we're talking about. Should be specific when referring to "source" and "destination" whether we're talking about a table or a cluster.

jhlodin · 2025-12-08T17:04:32Z

src/current/v25.4/manage-logical-data-replication.md

 )
 ~~~

+#### Resolve rows in the DLQ


Suggested change

#### Resolve rows in the DLQ

#### Manage rows in the DLQ

Feels better as a verb for the header, since it's not clear from the previous section that anything is in an "unresolved" state.

jhlodin · 2025-12-08T17:05:22Z

src/current/v25.4/manage-logical-data-replication.md


+#### Resolve rows in the DLQ
+
+LDR does not pause when writes are sent to the DLQ. You must manage the DLQ manually by examining each entry in the DLQ and either manually reinserting the row or deleting the entry from the DLQ. If you have multiple DLQ entries, resolve them in order from most recent to least recent. 


Suggested change

LDR does not pause when writes are sent to the DLQ. You must manage the DLQ manually by examining each entry in the DLQ and either manually reinserting the row or deleting the entry from the DLQ. If you have multiple DLQ entries, resolve them in order from most recent to least recent.

LDR does not pause when writes are sent to the DLQ. You must manage the DLQ manually by examining each entry in the DLQ and either reinserting the row or deleting the entry from the DLQ. If you have multiple DLQ entries, resolve them in order from most recent to least recent.

2x "manually"

jhlodin · 2025-12-08T17:06:07Z

src/current/v25.4/manage-logical-data-replication.md

+
+LDR does not pause when writes are sent to the DLQ. You must manage the DLQ manually by examining each entry in the DLQ and either manually reinserting the row or deleting the entry from the DLQ. If you have multiple DLQ entries, resolve them in order from most recent to least recent. 
+
+To resolve a row in the DLQ:


Suggested change

To resolve a row in the DLQ:

To resolve an entry in the DLQ:

Section uses "row" and "entry" interchangeably, should stick to one

jhlodin · 2025-12-08T17:08:25Z

src/current/v25.4/manage-logical-data-replication.md

+
+To resolve a row in the DLQ:
+
+1. On the destination, find the primary key value in the `incoming_row` column.


Suggested change

1. On the destination, find the primary key value in the `incoming_row` column.

1. On the destination cluster's DLQ table, find the primary key value in the `incoming_row` column.

jhlodin · 2025-12-08T17:09:22Z

src/current/v25.4/manage-logical-data-replication.md

+    106677386757203 | 2025-04-25 25:32:28.435439+00 | {"created_at": "2025-04-25:35:00.499499", "payload": "blahblahblah=", "my_id": 207}
+    ~~~
+
+1. Determine whether the value of the row matches on the source and the destination:


Suggested change

1. Determine whether the value of the row matches on the source and the destination:

1. Determine whether the value of the row in the DLQ matches the values on the source and destination tables respectively:

jhlodin · 2025-12-08T17:12:12Z

src/current/v25.4/manage-logical-data-replication.md

+
+1. Determine whether the value of the row matches on the source and the destination:
+
+    1. Check the value of the row and the replicated time:


Clarify that this is run on the DLQ table on the destination cluster

Nit: The parent step reads "source and destination" in that order but these are ordered destination then source. Suggest making those consistent

jhlodin · 2025-12-08T17:12:22Z

src/current/v25.4/manage-logical-data-replication.md

+        SELECT replicated_time FROM show logical replication jobs;
+        ~~~
+
+    1. On the source, check the value of the row as of the replicated time:


source cluster

jhlodin · 2025-12-08T17:15:56Z

src/current/v25.4/manage-logical-data-replication.md

+        DELETE FROM crdb_replication.dlq_271_foo WHERE id = 106677386757203;
+        ~~~
+
+    1. If the row's value on the destination is different from its value on the source, but the row's value on the source equals its value in the DLQ, update the row on the destination to have the same value as on the source:


This is the same state as the following step, sounds like one of them is meant to say "but the row's value on the source is also different from its value in the DLQ" or similar.

jhlodin · 2025-12-08T17:20:00Z

src/current/v25.4/manage-logical-data-replication.md

+
+        {% include_cached copy-clipboard.html %}
+        ~~~ sql
+        SELECT * FROM foo WHERE my_id = 207;


We should be more clear throughout about whether these operations are taking place on the source table, the destination table, or the DLQ for the table. Can we use fully-qualified table names here with corresponding DB names in order to make it more clear? Or are the fully-qualified names going to be identical regardless of which cluster it's on?

I'm not sure about this, since I would think the two clusters would be as identical as possible- @msbutler do you have an opinion?

it would be fine to use fully qualified names in the example queries. i.e. sourceDB.Foo and destDB.Foo

(aside: you are allowed to run LDR within one cluster from one db to another db)

jhlodin · 2025-12-08T17:20:35Z

src/current/v26.1/manage-logical-data-replication.md

Same comments as above

peachdawnleach · 2025-12-09T15:37:41Z

@msbutler working on clarifying some things in here based on docs review, and wondering- in a DLQ entry, is "payload" the value of the row that it's trying to insert? And is that formatted as like "column=value" ? Or just "value"? Or some other way? Thanks!

msbutler · 2025-12-09T16:04:56Z

@peachdawnleach ah good question, In this toy example table, incoming_row contains the value for the pk identified by the column my_id, as well as columns created_at and payload. I updated my kb runbook with this info. So, incoming_row contains the values of the columns of the DLQ’d row.

peachdawnleach · 2025-12-09T18:45:31Z

@msbutler Awesome, that makes sense. So in that case, why is it that when performing the UPSERT in a later step, the values are in a different order than they are in the DLQ row? Is it just that the pk always comes first?

Clarifying language and examples based on docs review

peachdawnleach · 2025-12-09T20:15:32Z

@msbutler Also, this PR could generally use a re-review when you have a chance - a lot ended up changing here

peachdawnleach added 2 commits December 1, 2025 16:54

Added section

270a8de

Added section on how to manually resolve entries in the dlq

typo

6a7ecc9

typo

Merge branch 'main' into 20251125-doc-15407-manually-apply-dlq-rows

c0fef15

peachdawnleach requested review from alicia-l2 and msbutler December 3, 2025 20:01

msbutler approved these changes Dec 5, 2025

View reviewed changes

peachdawnleach and others added 4 commits December 5, 2025 12:05

Merge branch 'main' into 20251125-doc-15407-manually-apply-dlq-rows

4b5aeb4

Added nested steps and ported to 26.1

34e963b

Added nested steps and ported to 26.1

Merge branch 'main' into 20251125-doc-15407-manually-apply-dlq-rows

06bd65a

trigger build

9712731

peachdawnleach force-pushed the 20251125-doc-15407-manually-apply-dlq-rows branch from ff5e88f to de022e2 Compare December 8, 2025 15:33

Merge branch '20251125-doc-15407-manually-apply-dlq-rows' of github.c…

4315294

…om:cockroachdb/docs into 20251125-doc-15407-manually-apply-dlq-rows

peachdawnleach force-pushed the 20251125-doc-15407-manually-apply-dlq-rows branch from de022e2 to 4315294 Compare December 8, 2025 16:24

jhlodin requested changes Dec 8, 2025

View reviewed changes

peachdawnleach and others added 2 commits December 9, 2025 14:23

Clarifications based on review

a43f3fb

Clarifying language and examples based on docs review

Merge branch 'main' into 20251125-doc-15407-manually-apply-dlq-rows

ae3ab5a

peachdawnleach requested review from msbutler and removed request for alicia-l2 December 9, 2025 20:15


		#### Resolve rows in the DLQ

		LDR does not pause when writes are sent to the DLQ. You must manage the DLQ manually by examining each entry in the DLQ and either manually reinserting the row or deleting the entry from the DLQ. If you have multiple DLQ entries, resolve them in order from most recent to least recent.


		LDR does not pause when writes are sent to the DLQ. You must manage the DLQ manually by examining each entry in the DLQ and either manually reinserting the row or deleting the entry from the DLQ. If you have multiple DLQ entries, resolve them in order from most recent to least recent.

		To resolve a row in the DLQ:


		To resolve a row in the DLQ:

		1. On the destination, find the primary key value in the `incoming_row` column.

	1. On the destination, find the primary key value in the `incoming_row` column.
	1. On the destination cluster's DLQ table, find the primary key value in the `incoming_row` column.

	1. Determine whether the value of the row matches on the source and the destination:
	1. Determine whether the value of the row in the DLQ matches the values on the source and destination tables respectively:


		1. Determine whether the value of the row matches on the source and the destination:

		1. Check the value of the row and the replicated time:

Add info on how to manually apply dlq rows #21440

Are you sure you want to change the base?

Add info on how to manually apply dlq rows #21440

Conversation

peachdawnleach commented Dec 2, 2025

Uh oh!

github-actions bot commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Files changed:

Uh oh!

netlify bot commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for cockroachdb-interactivetutorials-docs canceled.

Uh oh!

netlify bot commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for cockroachdb-api-docs canceled.

Uh oh!

netlify bot commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Netlify Preview

Uh oh!

peachdawnleach commented Dec 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

netlify bot commented Dec 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ Deploy Preview for cockroachdb-docs-noble-test failed. Why did it fail? →

Uh oh!

jhlodin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

peachdawnleach commented Dec 9, 2025

Uh oh!

msbutler commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

peachdawnleach commented Dec 9, 2025

Uh oh!

peachdawnleach commented Dec 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

github-actions bot commented Dec 2, 2025 •

edited

Loading

netlify bot commented Dec 2, 2025 •

edited

Loading

netlify bot commented Dec 2, 2025 •

edited

Loading

netlify bot commented Dec 2, 2025 •

edited

Loading

netlify bot commented Dec 8, 2025 •

edited

Loading

msbutler commented Dec 9, 2025 •

edited

Loading