Skip to content

Migration fails with "ghc table does not exist" #1622

@ajm188

Description

@ajm188

This was running off of master against an 8.4.6 database.

+ ./gh-ost -allow-on-master -attempt-instant-ddl -dml-batch-size 50 -exact-rowcount -force-named-cut-over -force-named-panic -postpone-cut-over-flag-file /tmp/queries/bb2d953d-431d-4bf0-
b6b2-d7ed1a2ae9d0/postpone-cutover -throttle-additional-flag-file /tmp/queries/bb2d953d-431d-4bf0-b6b2-d7ed1a2ae9d0/throttle -timestamp-old-table --skip-metadata-lock-check -database test -alter 'add index idx_user_start_time_end_time (user_id, start_time, end_time)' -table queries -host $hostname -user root -ask-pass
[2026/01/30 17:57:10] [info] binlogsyncer.go:191 create BinlogSyncer with config {ServerID:99999 Flavor:mysql Host:[redacted] Port:3306 User:root Password: Localhost: Charset: SemiSyncEnabled:false RawModeEnabled:false TLSConfig:<nil> ParseTime:false TimestampStringLocation:UTC UseDecimal:true RecvB
ufferSize:0 HeartbeatPeriod:0s ReadTimeout:0s MaxReconnectAttempts:0 DisableRetrySync:false VerifyChecksum:false DumpCommandFlag:0 Option:<nil> Logger:0x40004c23c0 Dialer:0x329d40 RowsEv
entDecodeFunc:<nil> TableMapOptionalMetaDecodeFunc:<nil> DiscardGTIDSet:false EventCacheCount:10240 SynchronousEventHandler:<nil>}
[2026/01/30 17:57:10] [info] binlogsyncer.go:443 begin to sync binlog from position (mysql-bin-changelog.027237, 3002)
[2026/01/30 17:57:10] [info] binlogsyncer.go:409 Connected to mysql 8.4.6 server
[2026/01/30 17:57:10] [info] binlogsyncer.go:868 rotate to (mysql-bin-changelog.027237, 3002)
# Migrating `test`.`queries`; Ghost table is `test`.`_queries_gho`
# Migrating `test`.`queries`; Ghost table is `test`.`_queries_gho`
# Migrating ip-172-17-5-122:3306; inspecting ip-172-17-5-122:3306; executing on bastion-0
# Migration started at Fri Jan 30 17:57:10 +0000 2026
# chunk-size: 1000; max-lag-millis: 1500ms; dml-batch-size: 50; max-load: ; critical-load: ; nice-ratio: 0.000000
# throttle-additional-flag-file: /tmp/queries/bb2d953d-431d-4bf0-b6b2-d7ed1a2ae9d0/throttle
# postpone-cut-over-flag-file: /tmp/queries/bb2d953d-431d-4bf0-b6b2-d7ed1a2ae9d0/postpone-cutover [set]
# Serving on unix socket: /tmp/gh-ost.test.queries.sock
# Migrating ip-172-17-5-122:3306; inspecting ip-172-17-5-122:3306; executing on bastion-0
# Migration started at Fri Jan 30 17:57:10 +0000 2026
# chunk-size: 1000; max-lag-millis: 1500ms; dml-batch-size: 50; max-load: ; critical-load: ; nice-ratio: 0.000000
# throttle-additional-flag-file: /tmp/queries/bb2d953d-431d-4bf0-b6b2-d7ed1a2ae9d0/throttle
# postpone-cut-over-flag-file: /tmp/queries/bb2d953d-431d-4bf0-b6b2-d7ed1a2ae9d0/postpone-cutover [set]
# Serving on unix socket: /tmp/gh-ost.honeycomb_prod.queries.sock
Copy: 0/0 100.0%; Applied: 0; Backlog: 0/1000; Time: 0s(total), 0s(copy); streamer: mysql-bin-changelog.027237:7368; Lag: 0.07s, HeartbeatLag: 0.07s, State: migrating; ETA: due
Copy: 0/426432577 0.0%; Applied: 0; Backlog: 0/1000; Time: 0s(total), 0s(copy); streamer: mysql-bin-changelog.027237:7368; Lag: 0.07s, HeartbeatLag: 0.07s, State: migrating; ETA: N/A
CREATE TABLE `_queries_gho` (
  `id` int NOT NULL AUTO_INCREMENT,
  ...
  PRIMARY KEY (`id`),
) ENGINE=InnoDB AUTO_INCREMENT=404517417 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin
[2026/01/30 17:57:10] [info] binlogsyncer.go:225 syncer is closing...
[2026/01/30 17:57:10] [info] binlogsyncer.go:988 kill last connection id 109
[2026/01/30 17:57:10] [info] binlogsyncer.go:255 syncer is closed
2026-01-30 17:57:10 ERROR Error 1146 (42S02): Table 'test._queries_ghc' doesn't exist

Interestingly, I was not running in either "revert" or "resume" mode, so I have no idea why gh-ost didn't actually try to create the changelog table. After the gh-ost process exited, there were no gh-ost tables on the database at all:

(MySQL):test>show tables like '%gh%';
+---------------------------------+
| Tables_in_test (%gh%) |
+---------------------------------+
+---------------------------------+

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions