Skip to content

数据分散,根据时间字段报错 #1070

@czxin788

Description

@czxin788

Description

按时间字段,将数据分散到不同的mysql实例,报错

 tail -f dtle.log 
2024-09-12T04:42:31.680Z [INFO]  applier: doFullCopy: loop: rowCopyComplete: job=dtle-demo-src-dst2
2024-09-12T04:42:31.680Z [INFO]  extractor: initBinlogReader: job=dtle-demo-src-dst2
2024-09-12T04:42:31.680Z [INFO]  reader: Connecting binlog streamer: job=dtle-demo-src-dst2 file=bin.000004 pos=3297 gtid=4d7bd416-6f5a-11ef-9c56-0242ac120004:1-22
2024-09-12T04:42:31.684Z [INFO]  extractor: monitor and update job status: job=dtle-demo-src-dst2
2024-09-12T04:42:31.684Z [INFO]  extractor: Beginning streaming: job=dtle-demo-src-dst2
2024-09-12T04:42:31.684Z [INFO]  reader: Rotate to next binlog: job=dtle-demo-src-dst2 name=bin.000004
2024-09-12T04:42:31.685Z [ERROR] extractor: onError: job=dtle-demo-src-dst2 err="StreamEvents encountered unexpected error: 'where' predicate does not eval to bool. demo.demo_tbl create_time>='2023-01-01'" hasShutdown=false
2024-09-12T04:42:31.685Z [INFO]  extractor: extractor shutdown: job=dtle-demo-src-dst2
2024-09-12T04:42:31.687Z [INFO]  DestroyTask: id=4ea0319f-4101-d066-d05a-9bcafe20f734/src/9979b45c
2024-09-12T04:42:31.785Z [INFO]  extractor: Shutting down: job=dtle-demo-src-dst2

Steps to reproduce the issue

  1. 创建表结构
mysql> show create table demo_tbl \G
*************************** 1. row ***************************
       Table: demo_tbl
Create Table: CREATE TABLE `demo_tbl` (
  `id` int(11) NOT NULL,
  `name` varchar(12) DEFAULT NULL,
  `create_time` datetime DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
1 row in set (0.00 sec)
  1. 源表数据
mysql> select * from demo_tbl;
+----+------+---------------------+
| id | name | create_time         |
+----+------+---------------------+
|  1 | a    | 2023-01-02 14:00:00 |
|  2 | b    | 2023-01-02 14:01:00 |
|  3 | c    | 2023-01-02 14:02:00 |
|  4 | d    | 2024-01-02 14:02:00 |
|  5 | e    | 2024-01-03 14:02:00 |
|  6 | e    | 2024-06-03 14:02:00 |
|  7 | e    | 2024-06-03 14:02:00 |
|  8 | e    | 2024-06-03 14:02:00 |
|  9 | e    | 2022-06-03 14:02:00 |
| 10 | e    | 2022-06-03 14:02:00 |
| 11 | e    | 2024-06-03 14:02:00 |
+----+------+---------------------+
11 rows in set (0.00 sec)
  1. dtle任务文件
    cat src_dst1.json
{
  "Job": {
    "ID": "dtle-demo-src-dst1",
    "Datacenters": ["dc1"],
    "TaskGroups": [{
        "Name": "src",
        "Tasks": [{
          "Name": "src",
          "Driver": "dtle",
          "Config": {
            "Gtid": "",
            "ReplicateDoDb": [{
              "TableSchema": "demo",
              "Tables": [{
                "TableName": "demo_tbl",
                "Where":"create_time<'2023-01-01'"
              }]
            }],
            "SrcConnectionConfig": {
              "Host": "mysql-src",
              "Port": 3306,
              "User": "root",
              "Password": "pass"
            },
            "DestConnectionConfig": {
              "Host": "mysql-dst1",
              "Port": 3306,
              "User": "root",
              "Password": "pass"
            }
          }
        }]
      }, {
        "Name": "dest",
        "Tasks": [{
          "Name": "dest",
          "Driver": "dtle",
          "Config": {
            "DestType": "mysql"
          }
        }]
    }]
  }
}

cat src_dst2.json

{
  "Job": {
    "ID": "dtle-demo-src-dst2",
    "Datacenters": ["dc1"],
    "TaskGroups": [{
        "Name": "src",
        "Tasks": [{
          "Name": "src",
          "Driver": "dtle",
          "Config": {
            "Gtid": "",
            "ReplicateDoDb": [{
              "TableSchema": "demo",
              "Tables": [{
                "TableName": "demo_tbl",
                "Where":"create_time>='2023-01-01'"
              }]
            }],
            "SrcConnectionConfig": {
              "Host": "mysql-src",
              "Port": 3306,
              "User": "root",
              "Password": "pass"
            },
            "DestConnectionConfig": {
              "Host": "mysql-dst2",
              "Port": 3306,
              "User": "root",
              "Password": "pass"
            }
          }
        }]
      }, {
        "Name": "dest",
        "Tasks": [{
          "Name": "dest",
          "Driver": "dtle",
          "Config": {
            "DestType": "mysql"
          }
        }]
    }]
  }
}

Describe the results you received

 tail -f dtle.log 
2024-09-12T04:42:31.680Z [INFO]  applier: doFullCopy: loop: rowCopyComplete: job=dtle-demo-src-dst2
2024-09-12T04:42:31.680Z [INFO]  extractor: initBinlogReader: job=dtle-demo-src-dst2
2024-09-12T04:42:31.680Z [INFO]  reader: Connecting binlog streamer: job=dtle-demo-src-dst2 file=bin.000004 pos=3297 gtid=4d7bd416-6f5a-11ef-9c56-0242ac120004:1-22
2024-09-12T04:42:31.684Z [INFO]  extractor: monitor and update job status: job=dtle-demo-src-dst2
2024-09-12T04:42:31.684Z [INFO]  extractor: Beginning streaming: job=dtle-demo-src-dst2
2024-09-12T04:42:31.684Z [INFO]  reader: Rotate to next binlog: job=dtle-demo-src-dst2 name=bin.000004
2024-09-12T04:42:31.685Z [ERROR] extractor: onError: job=dtle-demo-src-dst2 err="StreamEvents encountered unexpected error: 'where' predicate does not eval to bool. demo.demo_tbl create_time>='2023-01-01'" hasShutdown=false
2024-09-12T04:42:31.685Z [INFO]  extractor: extractor shutdown: job=dtle-demo-src-dst2
2024-09-12T04:42:31.687Z [INFO]  DestroyTask: id=4ea0319f-4101-d066-d05a-9bcafe20f734/src/9979b45c
2024-09-12T04:42:31.785Z [INFO]  extractor: Shutting down: job=dtle-demo-src-dst2

Describe the results you expected

Output of ./dtle version:**

dtle 4.23.04.3

(paste your output here)

Additional information

(e.g. issue happens only occasionally)

Additional details (log, config, job config etc):

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions