object SchemaInferenceUtils
- Alphabetic
- By Inheritance
- SchemaInferenceUtils
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- def diffSchemas(currentSchema: StructType, targetSchema: StructType): Seq[TableChange]
Determines the column changes needed to transform the current schema into the target schema.
Determines the column changes needed to transform the current schema into the target schema.
This function compares the current schema with the target schema and produces a sequence of TableChange objects representing: 1. New columns that need to be added 2. Existing columns that need type updates
- currentSchema
The current schema of the table
- targetSchema
The target schema that we want the table to have
- returns
A sequence of TableChange objects representing the necessary changes
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def inferSchemaFromFlows(flows: Seq[ResolvedFlow], userSpecifiedSchema: Option[StructType]): StructType
Given a set of flows that write to the same destination and possibly a user-specified schema, we infer the schema of the destination dataset.
Given a set of flows that write to the same destination and possibly a user-specified schema, we infer the schema of the destination dataset. The logic is as follows: 1. If there are no incoming flows, return the user-specified schema (if provided) or an empty schema. 2. If there are incoming flows, we merge the schemas of all flows that write to the same destination. 3. If a user-specified schema is provided, we merge it with the inferred schema. The user-specified schema will take precedence over the inferred schema. Returns an error if encountered during schema inference or merging the inferred schema with the user-specified one.
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)