Package org.apache.spark.sql

Interface SparkSessionExtensionsProvider

All Superinterfaces:
scala.Function1<SparkSessionExtensions,scala.runtime.BoxedUnit>
All Known Implementing Classes:
InternalFunctionRegistration
public interface SparkSessionExtensionsProvider extends scala.Function1<SparkSessionExtensions,scala.runtime.BoxedUnit>
:: Unstable ::

Base trait for implementations used by SparkSessionExtensions

For example, now we have an external function named Age to register as an extension for SparkSession: 


   package org.apache.spark.examples.extensions

   import org.apache.spark.sql.catalyst.expressions.{CurrentDate, Expression, RuntimeReplaceable, SubtractDates}

   case class Age(birthday: Expression, child: Expression) extends RuntimeReplaceable {

     def this(birthday: Expression) = this(birthday, SubtractDates(CurrentDate(), birthday))
     override def exprsReplaced: Seq[Expression] = Seq(birthday)
     override protected def withNewChildInternal(newChild: Expression): Expression = copy(newChild)
   }

We need to create our extension which inherits SparkSessionExtensionsProvider Example: 


   package org.apache.spark.examples.extensions

   import org.apache.spark.sql.{SparkSessionExtensions, SparkSessionExtensionsProvider}
   import org.apache.spark.sql.catalyst.FunctionIdentifier
   import org.apache.spark.sql.catalyst.expressions.{Expression, ExpressionInfo}

   class MyExtensions extends SparkSessionExtensionsProvider {
     override def apply(v1: SparkSessionExtensions): Unit = {
       v1.injectFunction(
         (new FunctionIdentifier("age"),
           new ExpressionInfo(classOf[Age].getName, "age"),
           (children: Seq[Expression]) => new Age(children.head)))
     }
   }

Then, we can inject MyExtensions in three ways,

  • withExtensions of SparkSession.Builder
  • Config - spark.sql.extensions
  • ServiceLoader - Add to src/main/resources/META-INF/services/org.apache.spark.sql.SparkSessionExtensionsProvider

Since:
3.2.0
See Also:

  • Nested Class Summary

    Nested classes/interfaces inherited from interface scala.Function1

    scala.Function1.UnliftOps<A extends Object,B extends Object>, scala.Function1.UnliftOps$

  • Method Summary

    Methods inherited from interface scala.Function1

    andThen, apply, apply$mcDD$sp, apply$mcDF$sp, apply$mcDI$sp, apply$mcDJ$sp, apply$mcFD$sp, apply$mcFF$sp, apply$mcFI$sp, apply$mcFJ$sp, apply$mcID$sp, apply$mcIF$sp, apply$mcII$sp, apply$mcIJ$sp, apply$mcJD$sp, apply$mcJF$sp, apply$mcJI$sp, apply$mcJJ$sp, apply$mcVD$sp, apply$mcVF$sp, apply$mcVI$sp, apply$mcVJ$sp, apply$mcZD$sp, apply$mcZF$sp, apply$mcZI$sp, apply$mcZJ$sp, compose, toString