[FLINK-34088] [hotfix] fix the problems with special table name characters of postgres and oracle and sqlserver. #89

BlackPigHe · 2024-01-16T06:23:25Z

[hotfix] fix the problems with special table name characters of postgres and oracle and sqlserver.

…res and oracle and sqlserver.

boring-cyborg · 2024-01-16T06:23:29Z

Thanks for opening this pull request! Please check out our contributing guidelines. (https://flink.apache.org/contributing/how-to-contribute.html)

…res and oracle and sqlserver. update test case.

BlackPigHe · 2024-01-17T01:35:44Z

add the commit to update test case

…in oracle jdbc,Specify the required type explicitly

…roperly in oracle jdbc,Specify the required type explicitly" This reverts commit 7272a79.

snuyanzin · 2024-01-18T22:20:01Z

flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/utils/JdbcUtils.java

+    public static String handleDoubleQuotes(String identifier) {
+        String[] split = identifier.split("\\.");
+        StringBuilder builder = new StringBuilder();
+        for (String s : split) {
+            builder.append("\"").append(s).append("\"");
+            builder.append(".");
+        }
+        return builder.deleteCharAt(builder.length() - 1).toString();
+    }


I didn't get why we are going to put logic with double quotes into some generic place if for different engines there could be different quotes?

Because oracle and postgres have the same escape characters, It's all double quotes

this package is not only for oracle and postgres. E.g. MS SQL Server, MySQL are different

Do you think that the location of this method is not appropriate, I think it is OK, the specific escape characters of the data source in their own package implementation, there are public features to extract and did not find a better location

snuyanzin · 2024-01-18T22:32:18Z

...va/org/apache/flink/connector/jdbc/databases/oracle/dialect/OraclePreparedStatementTest.java

+                        "INSERT INTO \"tbl\"(\"id\", \"name\", \"email\", \"ts\", \"field1\", \"field_2\", \"__field_3__\") "
                                + "VALUES (:id, :name, :email, :ts, :field1, :field_2, :__field_3__)");
        NamedStatementMatcher.parsedSql(
-                        "INSERT INTO tbl(id, name, email, ts, field1, field_2, __field_3__) "
+                        "INSERT INTO \"tbl\"(\"id\", \"name\", \"email\", \"ts\", \"field1\", \"field_2\", \"__field_3__\") "


In issue description it was only something about table name
why does it impact columns?

the method ‘quoteIdentifier’ is common，column metadata is also by call the method.Columns should have this problem as well

can we have a failing test reproducing the problem?

snuyanzin · 2024-01-19T08:46:10Z

I guess we can extract information about current quote identifier directly from JDBCDriver
java.sql.DatabaseMetaData#getIdentifierQuoteString
and then we don't need to hardcode it for every dialect

BlackPigHe · 2024-01-19T08:55:43Z

I guess we can extract information about current quote identifier directly from JDBCDriver java.sql.DatabaseMetaData#getIdentifierQuoteString and then we don't need to hardcode it for every dialect

I get it. If there are no compatibility issues, It's better this way.

BlackPigHe · 2024-01-19T09:27:17Z

quote identifier

I thought the method Java.SQL.DatabaseMetaData#getIdentifierQuoteString to dynamically access quote identifier, the price is too big, need to introduce Connection object which is very heavy, It is not reasonable for the existing dialect class design.

snuyanzin · 2024-02-01T07:29:02Z

yep, may be it is a point to see whether we could adapt it or not

btw I haven't found any test reproducing the issue, can you add one?
Also I would say the issue is more complex than just adding sql quote identifiers
e.g. even though there are quotes queries with fields containing question marks ? will fail FLINK-34211
i guess we need to a have a number of dedicated tests for that

github-actions · 2025-11-29T06:27:17Z

This PR is being marked as stale since it has not had any activity in the last 90 days.
If you would like to keep this PR alive, please leave a comment asking for a review.
If the PR has merge conflicts, update it with the latest from the base branch.

If you are having difficulty finding a reviewer, please reach out to the
community, contact details can be found here: https://flink.apache.org/what-is-flink/community/

If this PR is no longer valid or desired, please feel free to close it.
If no activity occurs in the next 30 days, it will be automatically closed.

fengzhi09 · 2025-12-17T09:37:40Z

English Version

Title: PostgresDialect fails to quote reserved keywords (e.g., desc) in column names, causing SQL syntax errors

Body:

I encountered an issue when using the Flink JDBC connector to write data into a PostgreSQL table that contains a column named desc — which is a reserved keyword in PostgreSQL. The operation fails with the following error:

ERROR: syntax error at or near "desc"
Position: 49

Upon investigation, I found that the generated SQL statement does not wrap the column name desc in double quotes (i.e., it should be "desc"). In PostgreSQL, reserved keywords must be quoted with double quotes to be used as identifiers.

Looking at the source code, I noticed that PostgresDialect does not override the quoteIdentifier method like other dialects (e.g., MySQLDialect, OracleDialect). As a result, column names are inserted into SQL statements verbatim, leading to syntax errors when they conflict with reserved words.

Suggestion:
Please implement the quoteIdentifier method in PostgresDialect to properly quote all column names (or at least reserved keywords) using double quotes, ensuring compatibility with PostgreSQL’s identifier rules.

Relevant code location:
https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/dialect/PostgresDialect.java

中文版本

标题：PostgreSQL 方言未正确转义保留关键字字段（如 desc），导致 INSERT/UPDATE 语句语法错误

正文：

我在使用 Flink JDBC connector 写入 PostgreSQL 表时遇到了一个问题：当表中包含名为 desc 的列（PostgreSQL 保留关键字）时，写入操作失败，报错如下：

ERROR: syntax error at or near "desc"
Position: 49

经排查，Flink 生成的 SQL 语句未对列名 desc 进行双引号转义（即应写作 "desc"），而 PostgreSQL 要求保留关键字必须用双引号引用才能作为标识符使用。

我查看了源码，发现 PostgresDialect 并未像其他数据库方言（如 MySQL、Oracle）那样重写 quoteIdentifier 方法来自动处理此类保留字。这导致在构建 INSERT 或 ON CONFLICT ... DO UPDATE 语句时，直接使用了原始列名，从而引发语法错误。

建议：
请为 PostgresDialect 实现 quoteIdentifier 方法，对所有列名统一使用双引号包裹（或至少对保留关键字进行转义），以确保与 PostgreSQL 的兼容性。

相关代码位置参考：
https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/dialect/PostgresDialect.java

---

CREATE TABLE IF NOT EXISTS agent_tool (
  `id` STRING,
  `type` STRING,
  `shop_id` STRING,
  `name` STRING,
  `desc` STRING,
  `key` STRING,
  `conf` STRING,
  `icon` STRING,
  `enabled` BOOLEAN NOT NULL,
  `is_deleted` BOOLEAN NOT NULL,
  `created_at` TIMESTAMP,
  `updated_at` TIMESTAMP,
  `deleted_at` TIMESTAMP,
  `creator` STRING,
  PRIMARY KEY ( `id` ) NOT ENFORCED
) WITH (
  'connector' = 'jdbc',
  'url' = 'jdbc:postgresql://aaaa:5432/bbbb?sslmode=disable&stringtype=unspecified&currentSchema=public',
  'username' = 'cccc',
  'password'='dddd',
  'table-name' = 'agent_tool',
  'sink.buffer-flush.max-rows' = '1000',
  'sink.buffer-flush.interval' = '2s',
  'sink.max-retries' = '3'
);

[hotfix] fix the problems with special table name characters of postg…

b13a3b9

…res and oracle and sqlserver.

boring-cyborg bot added the component=Connectors/JDBC label Jan 16, 2024

BlackPigHe changed the title ~~[hotfix] fix the problems with special table name characters of postgres and oracle and sqlserver.~~ [FLINK-34088] [hotfix] fix the problems with special table name characters of postgres and oracle and sqlserver. Jan 16, 2024

[hotfix] fix the problems with special table name characters of postg…

f35971e

…res and oracle and sqlserver. update test case.

[email protected] added 2 commits January 17, 2024 15:39

[FLINK-34128] [bugfix] Some jdbc objects cannot be obtained properly …

7272a79

…in oracle jdbc,Specify the required type explicitly

Revert "[FLINK-34128] [bugfix] Some jdbc objects cannot be obtained p…

b28197e

…roperly in oracle jdbc,Specify the required type explicitly" This reverts commit 7272a79.

snuyanzin reviewed Jan 18, 2024

View reviewed changes

github-actions bot added the stale label Nov 29, 2025

github-actions bot removed the stale label Dec 18, 2025

[FLINK-34088] [hotfix] fix the problems with special table name characters of postgres and oracle and sqlserver. #89

Are you sure you want to change the base?

[FLINK-34088] [hotfix] fix the problems with special table name characters of postgres and oracle and sqlserver. #89

Uh oh!

Conversation

BlackPigHe commented Jan 16, 2024

Uh oh!

boring-cyborg bot commented Jan 16, 2024

Uh oh!

BlackPigHe commented Jan 17, 2024

Uh oh!

snuyanzin Jan 18, 2024

Choose a reason for hiding this comment

Uh oh!

BlackPigHe Jan 19, 2024

Choose a reason for hiding this comment

Uh oh!

snuyanzin Jan 19, 2024

Choose a reason for hiding this comment

Uh oh!

BlackPigHe Jan 19, 2024

Choose a reason for hiding this comment

Uh oh!

snuyanzin Jan 18, 2024

Choose a reason for hiding this comment

Uh oh!

BlackPigHe Jan 19, 2024

Choose a reason for hiding this comment

Uh oh!

snuyanzin Jan 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

snuyanzin commented Jan 19, 2024

Uh oh!

BlackPigHe commented Jan 19, 2024

Uh oh!

BlackPigHe commented Jan 19, 2024

Uh oh!

snuyanzin commented Feb 1, 2024

Uh oh!

github-actions bot commented Nov 29, 2025

Uh oh!

fengzhi09 commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

English Version

中文版本

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

snuyanzin Jan 19, 2024 •

edited

Loading

fengzhi09 commented Dec 17, 2025 •

edited

Loading