Package org.apache.spark.sql.jdbc

Class PostgresDialect

Object
org.apache.spark.sql.jdbc.JdbcDialect
org.apache.spark.sql.jdbc.PostgresDialect
All Implemented Interfaces:
Serializable, org.apache.spark.internal.Logging, org.apache.spark.sql.catalyst.SQLConfHelper, NoLegacyJDBCError, scala.Equals, scala.Product
public class PostgresDialect extends JdbcDialect implements org.apache.spark.sql.catalyst.SQLConfHelper, NoLegacyJDBCError, scala.Product, Serializable
See Also:

  • Constructor Details

    • PostgresDialect

      public PostgresDialect()

  • Method Details

    • apply

      public abstract static R apply()

    • toString

      public static String toString()

    • canHandle

      public boolean canHandle(String url)
      Description copied from class: JdbcDialect
      Check if this dialect instance can handle a certain jdbc url.
      Specified by:
      canHandle in class JdbcDialect
      Parameters:
      url - the jdbc url.
      Returns:
      True if the dialect can be applied on the given jdbc url.

    • isSupportedFunction

      public boolean isSupportedFunction(String funcName)
      Description copied from class: JdbcDialect
      Returns whether the database supports function.
      Overrides:
      isSupportedFunction in class JdbcDialect
      Parameters:
      funcName - Upper-cased function name
      Returns:
      True if the database supports function.

    • isObjectNotFoundException

      public boolean isObjectNotFoundException(SQLException e)
      Overrides:
      isObjectNotFoundException in class JdbcDialect

    • getCatalystType

      public scala.Option<DataType> getCatalystType(int sqlType, String typeName, int size, MetadataBuilder md)
      Description copied from class: JdbcDialect
      Get the custom datatype mapping for the given jdbc meta information.

      Guidelines for mapping database defined timestamps to Spark SQL timestamps:

      • TIMESTAMP WITHOUT TIME ZONE if preferTimestampNTZ -> TimestampNTZType
      • TIMESTAMP WITHOUT TIME ZONE if !preferTimestampNTZ -> TimestampType(LTZ)
      • TIMESTAMP WITH TIME ZONE -> TimestampType(LTZ)
      • TIMESTAMP WITH LOCAL TIME ZONE -> TimestampType(LTZ)
      • If the TIMESTAMP cannot be distinguished by sqlType and typeName, preferTimestampNTZ is respected for now, but we may need to add another option in the future if necessary.

      Overrides:
      getCatalystType in class JdbcDialect
      Parameters:
      sqlType - Refers to Types constants, or other constants defined by the target database, e.g. -101 is Oracle's TIMESTAMP WITH TIME ZONE type. This value is returned by ResultSetMetaData.getColumnType(int).
      typeName - The column type name used by the database (e.g. "BIGINT UNSIGNED"). This is sometimes used to determine the target data type when sqlType is not sufficient if multiple database types are conflated into a single id. This value is returned by ResultSetMetaData.getColumnTypeName(int).
      size - The size of the type, e.g. the maximum precision for numeric types, length for character string, etc. This value is returned by ResultSetMetaData.getPrecision(int).
      md - Result metadata associated with this type. This contains additional information from ResultSetMetaData or user specified options.
      Returns:
      An option the actual DataType (subclasses of DataType) or None if the default type mapping should be used.

    • convertJavaTimestampToTimestampNTZ

      public LocalDateTime convertJavaTimestampToTimestampNTZ(Timestamp t)
      Description copied from class: JdbcDialect
      Convert java.sql.Timestamp to a LocalDateTime representing the same wall-clock time as the value stored in a remote database. JDBC dialects should override this function to provide implementations that suit their JDBC drivers.
      Overrides:
      convertJavaTimestampToTimestampNTZ in class JdbcDialect
      Parameters:
      t - Timestamp returned from JDBC driver getTimestamp method.
      Returns:
      A LocalDateTime representing the same wall clock time as the timestamp in database.

    • convertTimestampNTZToJavaTimestamp

      public Timestamp convertTimestampNTZToJavaTimestamp(LocalDateTime ldt)
      Description copied from class: JdbcDialect
      Converts a LocalDateTime representing a TimestampNTZ type to an instance of java.sql.Timestamp.
      Overrides:
      convertTimestampNTZToJavaTimestamp in class JdbcDialect
      Parameters:
      ldt - representing a TimestampNTZType.
      Returns:
      A Java Timestamp representing this LocalDateTime.

    • getJDBCType

      public scala.Option<JdbcType> getJDBCType(DataType dt)
      Description copied from class: JdbcDialect
      Retrieve the jdbc / sql type for a given datatype.
      Overrides:
      getJDBCType in class JdbcDialect
      Parameters:
      dt - The datatype (e.g. StringType)
      Returns:
      The new JdbcType if there is an override for this DataType

    • isCascadingTruncateTable

      public scala.Option<Object> isCascadingTruncateTable()
      Description copied from class: JdbcDialect
      Return Some[true] iff TRUNCATE TABLE causes cascading default. Some[true] : TRUNCATE TABLE causes cascading. Some[false] : TRUNCATE TABLE does not cause cascading. None: The behavior of TRUNCATE TABLE is unknown (default).
      Overrides:
      isCascadingTruncateTable in class JdbcDialect
      Returns:
      (undocumented)

    • getTruncateQuery

      public String getTruncateQuery(String table, scala.Option<Object> cascade)
      The SQL query used to truncate a table. For Postgres, the default behaviour is to also truncate any descendant tables. As this is a (possibly unwanted) side-effect, the Postgres dialect adds 'ONLY' to truncate only the table in question
      Overrides:
      getTruncateQuery in class JdbcDialect
      Parameters:
      table - The table to truncate
      cascade - Whether or not to cascade the truncation. Default value is the value of isCascadingTruncateTable(). Cascading a truncation will truncate tables with a foreign key relationship to the target table. However, it will not truncate tables with an inheritance relationship to the target table, as the truncate query always includes "ONLY" to prevent this behaviour.
      Returns:
      The SQL query to use for truncating a table

    • beforeFetch

      public void beforeFetch(Connection connection, scala.collection.immutable.Map<String,String> properties)
      Description copied from class: JdbcDialect
      Override connection specific properties to run before a select is made. This is in place to allow dialects that need special treatment to optimize behavior.
      Overrides:
      beforeFetch in class JdbcDialect
      Parameters:
      connection - The connection object
      properties - The connection properties. This is passed through from the relation.

    • getUpdateColumnTypeQuery

      public String getUpdateColumnTypeQuery(String tableName, String columnName, String newDataType)
      Overrides:
      getUpdateColumnTypeQuery in class JdbcDialect

    • getUpdateColumnNullabilityQuery

      public String getUpdateColumnNullabilityQuery(String tableName, String columnName, boolean isNullable)
      Overrides:
      getUpdateColumnNullabilityQuery in class JdbcDialect

    • createIndex

      public String createIndex(String indexName, Identifier tableIdent, NamedReference[] columns, Map<NamedReference,Map<String,String>> columnsProperties, Map<String,String> properties)
      Description copied from class: JdbcDialect
      Build a create index SQL statement.

      Overrides:
      createIndex in class JdbcDialect
      Parameters:
      indexName - the name of the index to be created
      tableIdent - the table on which index to be created
      columns - the columns on which index to be created
      columnsProperties - the properties of the columns on which index to be created
      properties - the properties of the index to be created
      Returns:
      the SQL statement to use for creating the index.

    • isSyntaxErrorBestEffort

      public boolean isSyntaxErrorBestEffort(SQLException exception)
      Description copied from class: JdbcDialect
      Attempts to determine if the given SQLException is a SQL syntax error.

      This check is best-effort: it may not detect all syntax errors across all JDBC dialects. However, if this method returns true, the exception is guaranteed to be a syntax error.

      This is used to decide whether to wrap the exception in a more appropriate Spark exception.

      Overrides:
      isSyntaxErrorBestEffort in class JdbcDialect
      Parameters:
      exception - (undocumented)
      Returns:
      true if the exception is confidently identified as a syntax error; false otherwise.

    • indexExists

      public boolean indexExists(Connection conn, String indexName, Identifier tableIdent, org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions options)
      Description copied from class: JdbcDialect
      Checks whether an index exists

      Overrides:
      indexExists in class JdbcDialect
      Parameters:
      conn - (undocumented)
      indexName - the name of the index
      tableIdent - the table on which index to be checked
      options - JDBCOptions of the table
      Returns:
      true if the index with indexName exists in the table with tableName, false otherwise

    • dropIndex

      public String dropIndex(String indexName, Identifier tableIdent)
      Description copied from class: JdbcDialect
      Build a drop index SQL statement.

      Overrides:
      dropIndex in class JdbcDialect
      Parameters:
      indexName - the name of the index to be dropped.
      tableIdent - the table on which index to be dropped.
      Returns:
      the SQL statement to use for dropping the index.

    • classifyException

      public Throwable classifyException(Throwable e, String condition, scala.collection.immutable.Map<String,String> messageParameters, String description, boolean isRuntime)
      Description copied from class: JdbcDialect
      Gets a dialect exception, classifies it and wraps it by AnalysisException.
      Specified by:
      classifyException in interface NoLegacyJDBCError
      Overrides:
      classifyException in class JdbcDialect
      Parameters:
      e - The dialect specific exception.
      condition - The error condition assigned in the case of an unclassified e
      messageParameters - The message parameters of errorClass
      description - The error description
      isRuntime - Whether the exception is a runtime exception or not.
      Returns:
      SparkThrowable + Throwable or its sub-class.

    • compileExpression

      public scala.Option<String> compileExpression(Expression expr)
      Description copied from class: JdbcDialect
      Converts V2 expression to String representing a SQL expression.
      Overrides:
      compileExpression in class JdbcDialect
      Parameters:
      expr - The V2 expression to be converted.
      Returns:
      Converted value.

    • compileValue

      public Object compileValue(Object value)
      Description copied from class: JdbcDialect
      Converts value to SQL expression.
      Overrides:
      compileValue in class JdbcDialect
      Parameters:
      value - The value to be converted.
      Returns:
      Converted value.

    • supportsLimit

      public boolean supportsLimit()
      Description copied from class: JdbcDialect
      Returns ture if dialect supports LIMIT clause.

      Note: Some build-in dialect supports LIMIT clause with some trick, please see: OracleDialect.OracleSQLQueryBuilder and MsSqlServerDialect.MsSqlServerSQLQueryBuilder.

      Overrides:
      supportsLimit in class JdbcDialect
      Returns:
      (undocumented)

    • supportsOffset

      public boolean supportsOffset()
      Description copied from class: JdbcDialect
      Returns ture if dialect supports OFFSET clause.

      Note: Some build-in dialect supports OFFSET clause with some trick, please see: OracleDialect.OracleSQLQueryBuilder and MySQLDialect.MySQLSQLQueryBuilder.

      Overrides:
      supportsOffset in class JdbcDialect
      Returns:
      (undocumented)

    • supportsTableSample

      public boolean supportsTableSample()
      Overrides:
      supportsTableSample in class JdbcDialect

    • supportsJoin

      public boolean supportsJoin()
      Description copied from class: JdbcDialect
      Returns true if dialect supports JOIN operator.
      Overrides:
      supportsJoin in class JdbcDialect
      Returns:
      (undocumented)

    • getTableSample

      public String getTableSample(org.apache.spark.sql.execution.datasources.v2.TableSampleInfo sample)
      Overrides:
      getTableSample in class JdbcDialect

    • renameTable

      public String renameTable(Identifier oldTable, Identifier newTable)
      Description copied from class: JdbcDialect
      Rename an existing table.

      Overrides:
      renameTable in class JdbcDialect
      Parameters:
      oldTable - The existing table.
      newTable - New name of the table.
      Returns:
      The SQL statement to use for renaming the table.

    • convertJavaTimestampToTimestamp

      public Timestamp convertJavaTimestampToTimestamp(Timestamp t)
      java.sql timestamps are measured with millisecond accuracy (from Long.MinValue milliseconds to Long.MaxValue milliseconds), while Spark timestamps are measured at microseconds accuracy. For the "infinity values" in PostgreSQL (represented by big constants), we need clamp them to avoid overflow. If it is not one of the infinity values, fall back to default behavior.
      Overrides:
      convertJavaTimestampToTimestamp in class JdbcDialect
      Parameters:
      t - (undocumented)
      Returns:
      (undocumented)

    • convertJavaDateToDate

      public Date convertJavaDateToDate(Date d)
      Description copied from class: JdbcDialect
      Converts an instance of java.sql.Date to a custom java.sql.Date value.
      Overrides:
      convertJavaDateToDate in class JdbcDialect
      Parameters:
      d - the date value returned from JDBC ResultSet getDate method.
      Returns:
      the date value after conversion

    • updateExtraColumnMeta

      public void updateExtraColumnMeta(Connection conn, ResultSetMetaData rsmd, int columnIdx, MetadataBuilder metadata)
      Description copied from class: JdbcDialect
      Get extra column metadata for the given column.

      Overrides:
      updateExtraColumnMeta in class JdbcDialect
      Parameters:
      conn - The connection currently connection being used.
      rsmd - The metadata of the current result set.
      columnIdx - The index of the column.
      metadata - The metadata builder to store the extra column information.