Python SDK read_parquet union_by_name fail after version 1.2.2

[DaiZack]

duckdbtest.zip

What happens?

Title

Regression in 1.3.0+: union_by_name fails with "Can't change source type (NULL) to target type (VARCHAR[])" when reading parquet files with mixed NULL/LIST types

DuckDB Version

• Working version: 1.2.2
• Broken versions: 1.3.0, 1.3.1 (and later)

Environment

• OS: Linux
• Python: 3.12.9
• pandas: (latest)

Description

Starting with DuckDB 1.3.0, reading multiple parquet files with union_by_name=True fails when:

1. Some parquet files have a column stored as NULL type (because all values are null in that file)
2. Other parquet files have the same column properly typed as VARCHAR[] (array/list of strings)

This worked correctly in DuckDB 1.2.2 but now throws:

BinderException: Binder Error: Can't change source type ("NULL") to target type (VARCHAR[]), type conversion not allowed

Expected Behavior

When union_by_name=True is set, DuckDB should merge schemas gracefully, treating NULL-typed columns as compatible with any target type (similar to how pandas handles this).

Actual Behavior

DuckDB 1.3.0+ throws a BinderException and refuses to read the files, even though union_by_name=True is explicitly designed to handle schema variations across multiple files.

Root Cause Analysis

Investigation shows:

• When a parquet file has ALL NULL values for a column, it's stored with NULL type (e.g., INT32 with NullType() logical type)
• Other files with actual data store the same column as BYTE_ARRAY with StringType() or complex types like ListType()
• The error specifically mentions VARCHAR[] (array type) suggesting it happens with nested/complex types
• This regression appeared between versions 1.2.2 and 1.3.0

How to Reproduce

attached files to test see duckdbtest.zip

import duckdb
print(f"DuckDB version: {duckdb.__version__}")

# Fails with 1.3.0+
try:
    result = duckdb.read_parquet(
        "duckdb_bug_test_files/*.parquet",
        union_by_name=True
    ).df()
    print(f"SUCCESS: Read {len(result)} rows")
except Exception as e:
    print(f"FAILED: {type(e).__name__}: {e}")

To Reproduce

this is only in python SDK

OS:

Linux x86

DuckDB Version:

v1.2.2, v1.3.0 and later

DuckDB Client:

Python

Hardware:

No response

Full Name:

Zack Dai

Affiliation:

Zack Dai

Did you include all relevant configuration (e.g., CPU architecture, Linux distribution) to reproduce the issue?

• Yes, I have

Did you include all code required to reproduce the issue?

• Yes, I have

Did you include all relevant data sets for reproducing the issue?

Yes

[Tishj]

Needs to be moved to https://github.com/duckdb/duckdb-python
Could you move it @szarnyasg ?

[DaiZack]

@Tishj @szarnyasg Thanks, I created one #257 .

[paultiq]

(not sure which issue to post in, one of them is a dupe):

Doesn't this same error/inability to union occur with base duckdb?

❯ duckdb
DuckDB v1.4.3 (Andium) d1dc88f950
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D select * from read_parquet('*.parquet', union_by_name=True)
┬╖ ;
Binder Error:
Struct remap can only remap nested types, not '"NULL"'
D

Note: reproduced that this would work in 1.2.2 but not in 1.3.0+ through 1.4.3.

[DaiZack]

@paultiq yes same error raised in cli v1.4.3
I didn't notice my cli was in older version and missed this.

---

cli v.14.3 raises error

DuckDB v1.4.3 (Andium) d1dc88f950
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D select * from read_parquet('*.parquet', union_by_name=True)
  ;
Binder Error:
Struct remap can only remap nested types, not '"NULL"'

---

cli v1.2.2 was working fine

v1.2.2 7c039464e4
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D select * from read_parquet('*.parquet', union_by_name=True)
  ;
┌────────┬──────────────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Index  │         Name         │                                                                                          data_detail                                                                                          │
│ int64  │       varchar        │ struct(address varchar, address_en varchar, alt_business_names varchar[], business_name varchar, business_name_en integer, city varchar, city_en varchar, country_code varchar, easy_number…  │
├────────┼──────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 177000 │ NULL                 │ NULL                                                                                                                                                                                          │
│ 177001 │ NULL                 │ NULL                                                                                                                                                                                          │
│ 177002 │ NULL                 │ NULL                                                                                                                                                                                          │
│ 177003 │ NULL                 │ NULL                                                                                                                                                                                          │
│ 177004 │ NULL                 │ NULL                                                                                                                                                                                          │
│ 177005 │ NULL                 │ NULL                                                                                                                                                                                          │
│   4061 │ NULL                 │ NULL                                                                                                                                                                                          │
│   4062 │ NULL                 │ NULL                                                                                                                                                                                          │
│   4063 │ NULL                 │ NULL                                                                                                                                                                                          │
│   4064 │ AGUILA CAPITAL GMBH  │ {'address': NULL, 'address_en': am homberg 2 b, 'alt_business_names': [alessu beteiligungs, einhundertdreissigste stv vermogensverwaltungsgesellschaft], 'business_name': aguila capital, '…  │
│   4065 │ NULL                 │ NULL                                                                                                                                                                                          │
│   4066 │ NULL                 │ NULL                                                                                                                                                                                          │
│   5042 │ NULL                 │ NULL                 

[evertlammerts]

Looks like this is fixed in 1.5.x?

❯ duckdb
DuckDB v1.5.1 (Variegata)
Enter ".help" for usage hints.
memory D SELECT * FROM read_parquet('file*.parquet', union_by_name=true);
┌───────┬────────────┐
│  id   │    tags    │
│ int32 │ varchar[]  │
├───────┼────────────┤
│     1 │ [foo, bar] │
│     2 │ NULL       │
└───────┴────────────┘