Merge branch 'main' into postgres-regression-2

Author: LucaCappelletti94

AGENTS.md

@@ -0,0 +1,41 @@
+# Extensible SQL Lexer and Parser for Rust Agents Guidelines
+
+## General Agent Workflow
+1. You will write unit tests to ensure your code change is working as expected.
+2. You will run the commands in the Pre Commit Checks section below to ensure your change is ready for a pull request.
+3. When instructed to open a PR, you will follow the instructions in the Pull Request Guidelines section below.
+
+## General Coding Guidelines
+1. Refrain from adding conditions on specific dialects, such as `dialect_is!(...)` or `dialect_of!(... | ...)`. Instead, define a new function in the `Dialect` trait that describes the condition, so that dialects can turn this condition on more easily.
+2. Make targeted code changes and refrain from refactoring, unless it's absolutely required.
+
+## Unit Tests Guidelines
+- New unit tests should be added to the `tests` module in the corresponding dialect file (e.g., `tests/sqlparser_redshift.rs` for Redshift), and should be placed at the end of the file.
+- If the new functionality is gated using a dialect function, and the SQL is likely relevant in most dialects, tests should be placed under `tests/sqlparser_common.rs`.
+- When testing a multi-line SQL statement, use a raw string literal, i.e. `r#"..."#` to preserve formatting.
+- The parser builds an abstract syntax tree (AST) from the SQL statement and has functionality to display the tree as SQL. Use the following template for simple unit tests where you expect the SQL created from the AST to be the same as the input SQL:
+```rust
+<dialect>().verified_stmt(r#"..."#);
+```
+For example: `snowflake().verified_stmt(r#"SELECT * FROM my_table"#)`. Use `one_statement_parses_to` instead of `verified_stmt` when you expect the SQL created by the AST to differ than the input SQL. For example:
+```rust
+snowflake().one_statement_parses_to(
+    "SELECT * FROM my_table t",
+    "SELECT * FROM my_table AS t",
+)
+```
+
+## Analyzing Parsing Issues
+You can try to simplify the SQL statement to identify the root cause of the parsing issue. This may involve removing certain clauses or components of the SQL statement to see if it can be parsed successfully. Additionally, you can compare the problematic SQL statement with similar statements that are parsed correctly to identify any differences that may be causing the issue.
+
+## Pre Commit Checks
+Run the following commands before you commit to ensure the change will pass the CI process:
+```bash
+cargo test --all-features
+cargo fmt --all
+cargo clippy --all-targets --all-features -- -D warnings
+```
+
+## Pull Request Guidelines
+1. PR title should follow this format: `<DIALECT>: <SHORT DESCRIPTION>`, For example, `Showflake: Add support for casting to VARIANT`.
+2. Make the PR comment short, provide an example of what was not working and a short description of the fix. Be succint.

Cargo.toml

@@ -18,7 +18,7 @@
 [package]
 name = "sqlparser"
 description = "Extensible SQL Lexer and Parser with support for ANSI SQL:2011"
-version = "0.61.0"
+version = "0.62.0"
 authors = ["Apache DataFusion <dev@datafusion.apache.org>"]
 homepage = "https://github.com/apache/datafusion-sqlparser-rs"
 documentation = "https://docs.rs/sqlparser/"

changelog/0.62.0.md

@@ -0,0 +1,143 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# sqlparser-rs 0.62.0 Changelog
+
+This release consists of 79 commits from 27 contributors. See credits at the end of this changelog for more information.
+
+**Implemented enhancements:**
+
+- feat(clickhouse): support PARTITION BY after ORDER BY and ARRAY JOIN [#2283](https://github.com/apache/datafusion-sqlparser-rs/pull/2283) (cristhiank)
+
+**Other:**
+
+- Fix panic on incomplete REGEXP/MATCH expressions in SQLite dialect [#2197](https://github.com/apache/datafusion-sqlparser-rs/pull/2197) (LucaCappelletti94)
+- chore: add confirmation before tarball is released [#2208](https://github.com/apache/datafusion-sqlparser-rs/pull/2208) (milenkovicm)
+- Databricks: Add support for `OPTIMIZE`, `PARTITIONED BY`, and `STRUCT` [#2170](https://github.com/apache/datafusion-sqlparser-rs/pull/2170) (funcpp)
+- Fix panic in `SET AUTHORIZATION` parsing when scope modifier is missing [#2201](https://github.com/apache/datafusion-sqlparser-rs/pull/2201) (LucaCappelletti94)
+- Snowflake: Add multi table insert support [#2148](https://github.com/apache/datafusion-sqlparser-rs/pull/2148) (finchxxia)
+- MSSQL: Support `THROW` statement [#2202](https://github.com/apache/datafusion-sqlparser-rs/pull/2202) (guan404ming)
+- Updated Fuzzer harness to address all dialects [#2203](https://github.com/apache/datafusion-sqlparser-rs/pull/2203) (RPG-Alex)
+- Snowflake: Lambda functions [#2192](https://github.com/apache/datafusion-sqlparser-rs/pull/2192) (yoavcloud)
+- Updated `parse_infix(..)` in `mysql.rs` and `sqlite.rs` to handle error rather than `unwrap()` [#2207](https://github.com/apache/datafusion-sqlparser-rs/pull/2207) (RPG-Alex)
+- MSSQL: Add support for TRAN shorthand [#2212](https://github.com/apache/datafusion-sqlparser-rs/pull/2212) (guan404ming)
+- Fixed select dollar column from stage for snowflake [#2165](https://github.com/apache/datafusion-sqlparser-rs/pull/2165) (romanoff)
+- [PIVOT] Optional AS keyword for aliases [#2209](https://github.com/apache/datafusion-sqlparser-rs/pull/2209) (xitep)
+- Fix `derive_dialect!` proc macro for use from external crates [#2219](https://github.com/apache/datafusion-sqlparser-rs/pull/2219) (alexander-beedie)
+- Resolve breaking change that set "supports_lambda_functions" on `GenericDialect` [#2224](https://github.com/apache/datafusion-sqlparser-rs/pull/2224) (alexander-beedie)
+- MSSQL: Add support for WAITFOR statement [#2210](https://github.com/apache/datafusion-sqlparser-rs/pull/2210) (guan404ming)
+- PostgreSQL: Support PRIMARY KEY/UNIQUE USING INDEX [#2213](https://github.com/apache/datafusion-sqlparser-rs/pull/2213) (guan404ming)
+- Prefer use of `peek_token_ref` over `peek_token` where valid [#2225](https://github.com/apache/datafusion-sqlparser-rs/pull/2225) (alexander-beedie)
+- PostgreSQL: Support more COMMENT ON object types [#2220](https://github.com/apache/datafusion-sqlparser-rs/pull/2220) (guan404ming)
+- Allow custom OptimizerHints [#2216](https://github.com/apache/datafusion-sqlparser-rs/pull/2216) (altmannmarcelo)
+- Optimise out string allocations and copies in keyword lookup [#2226](https://github.com/apache/datafusion-sqlparser-rs/pull/2226) (alexander-beedie)
+- [Oracle] Table alias for INSERTed table [#2214](https://github.com/apache/datafusion-sqlparser-rs/pull/2214) (xitep)
+- Introduce `Visit::visit_select` [#2235](https://github.com/apache/datafusion-sqlparser-rs/pull/2235) (xitep)
+- Add support for INTERVAL keyword as unquoted identifier in PostgreSQL [#2238](https://github.com/apache/datafusion-sqlparser-rs/pull/2238) (LucaCappelletti94)
+- Redshift: support wildcard select items with alias [#2230](https://github.com/apache/datafusion-sqlparser-rs/pull/2230) (yoabot-droid)
+- Snowflake: support wildcard with EXCLUDE in function arguments [#2231](https://github.com/apache/datafusion-sqlparser-rs/pull/2231) (yoabot-droid)
+- MSSQL: prevent statement-starting keywords from being consumed as implicit aliases [#2233](https://github.com/apache/datafusion-sqlparser-rs/pull/2233) (yoabot-droid)
+- Support parenthesized `CREATE TABLE ... (LIKE ... INCLUDING/EXCLUDING DEFAULTS)` in `PostgreSQL` [#2242](https://github.com/apache/datafusion-sqlparser-rs/pull/2242) (LucaCappelletti94)
+- MSSQL: support EXEC (@sql) dynamic SQL execution [#2234](https://github.com/apache/datafusion-sqlparser-rs/pull/2234) (yoabot-droid)
+- Support MySQL KEY keyword in column definitions [#2243](https://github.com/apache/datafusion-sqlparser-rs/pull/2243) (mvzink)
+- Support two-argument `TRIM(string, characters)` in PostgreSQL [#2240](https://github.com/apache/datafusion-sqlparser-rs/pull/2240) (LucaCappelletti94)
+- Add Readyset to users in README.md [#2247](https://github.com/apache/datafusion-sqlparser-rs/pull/2247) (mvzink)
+- Snowflake: parse EXCLUDE column list as ObjectName to support qualified names [#2244](https://github.com/apache/datafusion-sqlparser-rs/pull/2244) (yoabot-droid)
+- MSSQL: Add support for OUTPUT clause on INSERT/UPDATE/DELETE [#2228](https://github.com/apache/datafusion-sqlparser-rs/pull/2228) (guan404ming)
+- Redshift: Added DISTSTYLE and DISTKEY keywords parsing [#2222](https://github.com/apache/datafusion-sqlparser-rs/pull/2222) (romanoff)
+- Allow INSERT columns to be qualified [#2260](https://github.com/apache/datafusion-sqlparser-rs/pull/2260) (xitep)
+- Add support for parsing COPY statements from STDIN without a semicolon [#2245](https://github.com/apache/datafusion-sqlparser-rs/pull/2245) (LucaCappelletti94)
+- Fix credentials parsing for redshift [#2262](https://github.com/apache/datafusion-sqlparser-rs/pull/2262) (romanoff)
+- Added SORTKEY keyword parsing for redshift queries [#2261](https://github.com/apache/datafusion-sqlparser-rs/pull/2261) (romanoff)
+- Fixed transaction handling for snowflake [#2263](https://github.com/apache/datafusion-sqlparser-rs/pull/2263) (romanoff)
+- Fixed COPY GRANTS clause parsing for snowflake [#2267](https://github.com/apache/datafusion-sqlparser-rs/pull/2267) (romanoff)
+- Fixed parsing `OPTIONS(format = 'CSV')` when creating external bigquery table [#2268](https://github.com/apache/datafusion-sqlparser-rs/pull/2268) (romanoff)
+- Add support for PostgreSQL LOCK TABLE [#2273](https://github.com/apache/datafusion-sqlparser-rs/pull/2273) (mjbshaw)
+- add support for databricks JSON accessors [#2272](https://github.com/apache/datafusion-sqlparser-rs/pull/2272) (whirlun)
+- Fixed create snapshot table for bigquery [#2269](https://github.com/apache/datafusion-sqlparser-rs/pull/2269) (romanoff)
+- Fixed stage name parsing for snowflake [#2265](https://github.com/apache/datafusion-sqlparser-rs/pull/2265) (romanoff)
+- Fix STORAGE LIFECYCLE POLICY for snowflake queries [#2264](https://github.com/apache/datafusion-sqlparser-rs/pull/2264) (romanoff)
+- Fixed CHANGES keyword parsing for snowflake [#2266](https://github.com/apache/datafusion-sqlparser-rs/pull/2266) (romanoff)
+- Add SETOF support for PostgreSQL function return types [#2217](https://github.com/apache/datafusion-sqlparser-rs/pull/2217) (fmguerreiro)
+- [Oracle] Support for `INSERT INTO (<sub-query>) ...` [#2276](https://github.com/apache/datafusion-sqlparser-rs/pull/2276) (xitep)
+- Fixed BACKUP parsing for redshift [#2270](https://github.com/apache/datafusion-sqlparser-rs/pull/2270) (romanoff)
+- recursive protection for `parse_subexpr` [#2282](https://github.com/apache/datafusion-sqlparser-rs/pull/2282) (blaginin)
+- Fix the tokenization of `<` edge cases [#2280](https://github.com/apache/datafusion-sqlparser-rs/pull/2280) (ayman-sigma)
+- Expose values through ValueWithSpan [#2281](https://github.com/apache/datafusion-sqlparser-rs/pull/2281) (xitep)
+- Enable `!` as NOT operator for Databricks dialect [#2287](https://github.com/apache/datafusion-sqlparser-rs/pull/2287) (funcpp)
+- Add dollar prefix support for money constants in MSSQL [#2285](https://github.com/apache/datafusion-sqlparser-rs/pull/2285) (solontsev)
+- Allow bare columns in GROUPING SETS expressions [#2288](https://github.com/apache/datafusion-sqlparser-rs/pull/2288) (funcpp)
+- Enable numeric-prefix identifiers for Databricks dialect [#2290](https://github.com/apache/datafusion-sqlparser-rs/pull/2290) (funcpp)
+- Enable Redshift support for specifying null treatment inside window function [#2293](https://github.com/apache/datafusion-sqlparser-rs/pull/2293) (romanb)
+- Fix parsing EXECUTE (...) with a more general string expression [#2295](https://github.com/apache/datafusion-sqlparser-rs/pull/2295) (romanb)
+- Support optional AS keyword in CTE definitions for Databricks [#2286](https://github.com/apache/datafusion-sqlparser-rs/pull/2286) (funcpp)
+- MySQL: Add support for `ORDER BY` on single-table `UPDATE` [#2296](https://github.com/apache/datafusion-sqlparser-rs/pull/2296) (tpyo)
+- Support multi-column aliases in SELECT items [#2289](https://github.com/apache/datafusion-sqlparser-rs/pull/2289) (funcpp)
+- MySQL: Support `SHOW FULL PROCESSLIST` syntax [#2292](https://github.com/apache/datafusion-sqlparser-rs/pull/2292) (jstnd)
+- PostgreSQL `ALTER FUNCTION` / `ALTER AGGREGATE` [#2248](https://github.com/apache/datafusion-sqlparser-rs/pull/2248) (LucaCappelletti94)
+- Add xml '...' TypedString support for PostgreSQL [#2299](https://github.com/apache/datafusion-sqlparser-rs/pull/2299) (LucaCappelletti94)
+- Add PostgreSQL Collation DDL Support [#2249](https://github.com/apache/datafusion-sqlparser-rs/pull/2249) (LucaCappelletti94)
+- Fix COLLATE parsing after compound identifiers [#2294](https://github.com/apache/datafusion-sqlparser-rs/pull/2294) (romanb)
+- Redshift/PostgreSQL: Parse optional implicit aliases for wildcard select items [#2300](https://github.com/apache/datafusion-sqlparser-rs/pull/2300) (yoavcloud)
+- Only parse FROM identifier in CTE if using Hive [#2241](https://github.com/apache/datafusion-sqlparser-rs/pull/2241) (Viicos)
+- Add SHOW CATALOGS syntax and tests [#2284](https://github.com/apache/datafusion-sqlparser-rs/pull/2284) (Smith-Cruise)
+- Coding agents guidelines [#2298](https://github.com/apache/datafusion-sqlparser-rs/pull/2298) (yoavcloud)
+- Spark SQL: Add SparkSqlDialect [#2305](https://github.com/apache/datafusion-sqlparser-rs/pull/2305) (andygrove)
+- PostgreSQL: Add support for LATERAL ... WITH ORDINALITY [#2304](https://github.com/apache/datafusion-sqlparser-rs/pull/2304) (yoavcloud)
+- Track `Parens<T>`'s span [#2291](https://github.com/apache/datafusion-sqlparser-rs/pull/2291) (xitep)
+- Redshift: PartiQL AT <index> [#2303](https://github.com/apache/datafusion-sqlparser-rs/pull/2303) (yoavcloud)
+- Snowflake: Add support for text data type modifiers [#2297](https://github.com/apache/datafusion-sqlparser-rs/pull/2297) (yoavcloud)
+- PostgreSQL: Parse optimizer hints in leading comments [#2320](https://github.com/apache/datafusion-sqlparser-rs/pull/2320) (altmannmarcelo)
+- Add Teradata dialect [#2309](https://github.com/apache/datafusion-sqlparser-rs/pull/2309) (iffyio)
+
+## Credits
+
+Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.
+
+```
+    12	Andriy Romanov
+     9	Luca Cappelletti
+     7	xitep
+     6	Guan-Ming (Wesley) Chiu
+     6	Minjun Kim
+     6	Yoav Cohen
+     5	Yoabot
+     4	Alexander Beedie
+     3	Roman Borschel
+     2	Alex
+     2	Marcelo Altmann
+     2	Michael Victor Zink
+     1	Andy Grove
+     1	Ayman Elkfrawy
+     1	Cristhian Lopez
+     1	Dmitrii Blaginin
+     1	Donovan Schönknecht
+     1	Filipe Guerreiro
+     1	Ifeanyi Ubah
+     1	Marko Milenković
+     1	Michael Bradshaw
+     1	Sergey Olontsev
+     1	Smith Cruise
+     1	Victorien
+     1	finchxxia
+     1	jstnd
+     1	whirlun
+```
+
+Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.

dev/release/rat_exclude_files.txt

@@ -6,3 +6,4 @@ dev/release/rat_exclude_files.txt
 sqlparser_bench/img/flamegraph.svg
 **Cargo.lock
 filtered_rat.txt
+AGENTS.md