Package org.apache.spark.sql.artifact

Class ArtifactManager

Object
org.apache.spark.sql.artifact.ArtifactManager
All Implemented Interfaces:
org.apache.spark.internal.Logging
public class ArtifactManager extends Object implements org.apache.spark.internal.Logging
This class handles the storage of artifacts as well as preparing the artifacts for use.

Artifacts belonging to different SparkSessions are isolated from each other with the help of the sessionUUID.

Jars and classfile artifacts are stored under "jars", "classes" and "pyfiles" sub-directories respectively while other types of artifacts are stored under the root directory for that particular SparkSession.

param: session The object used to hold the Spark Connect session state.

  • Nested Class Summary

    Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging

    org.apache.spark.internal.Logging.LogStringContext, org.apache.spark.internal.Logging.SparkShellLoggingFilter

  • Constructor Summary

    Constructors
    Constructor
    Description
    ArtifactManager(SparkSession session)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    addArtifact(Path remoteRelativePath, Path serverLocalStagingPath, scala.Option<String> fragment, boolean deleteStagedFile)
    Add and prepare a staged artifact (i.e an artifact that has been rebuilt locally from bytes over the wire) for use.
    static String
    ARTIFACT_DIRECTORY_PREFIX()
     
    ClassLoader
    classloader()
    Returns a ClassLoader for session-specific jar/class file resources.
    static String
    forwardToFSPrefix()
     
    scala.collection.immutable.Seq<URL>
    getAddedJars()
    Get the URLs of all jar artifacts.
    scala.collection.immutable.Seq<String>
    getPythonIncludes()
    Get the py-file names added to this SparkSession.
    static org.apache.spark.internal.Logging.LogStringContext
    LogStringContext(scala.StringContext sc)
     
    static org.slf4j.Logger
    org$apache$spark$internal$Logging$$log_()
     
    static void
    org$apache$spark$internal$Logging$$log__$eq(org.slf4j.Logger x$1)
     
    void
    uploadArtifactToFs(Path remoteRelativePath, Path serverLocalStagingPath)
     
    <T> T
    withResources(scala.Function0<T> f)
     

    Methods inherited from class java.lang.Object

    equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.spark.internal.Logging

    initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isTraceEnabled, log, logDebug, logDebug, logDebug, logDebug, logError, logError, logError, logError, logInfo, logInfo, logInfo, logInfo, logName, LogStringContext, logTrace, logTrace, logTrace, logTrace, logWarning, logWarning, logWarning, logWarning, org$apache$spark$internal$Logging$$log_, org$apache$spark$internal$Logging$$log__$eq, withLogContext

  • Constructor Details

    • ArtifactManager

      public ArtifactManager(SparkSession session)

  • Method Details

    • forwardToFSPrefix

      public static String forwardToFSPrefix()

    • ARTIFACT_DIRECTORY_PREFIX

      public static String ARTIFACT_DIRECTORY_PREFIX()

    • org$apache$spark$internal$Logging$$log_

      public static org.slf4j.Logger org$apache$spark$internal$Logging$$log_()

    • org$apache$spark$internal$Logging$$log__$eq

      public static void org$apache$spark$internal$Logging$$log__$eq(org.slf4j.Logger x$1)

    • LogStringContext

      public static org.apache.spark.internal.Logging.LogStringContext LogStringContext(scala.StringContext sc)

    • withResources

      public <T> T withResources(scala.Function0<T> f)

    • getAddedJars

      public scala.collection.immutable.Seq<URL> getAddedJars()
      Get the URLs of all jar artifacts.
      Returns:
      (undocumented)

    • getPythonIncludes

      public scala.collection.immutable.Seq<String> getPythonIncludes()
      Get the py-file names added to this SparkSession.

      Returns:

    • addArtifact

      public void addArtifact(Path remoteRelativePath, Path serverLocalStagingPath, scala.Option<String> fragment, boolean deleteStagedFile)
      Add and prepare a staged artifact (i.e an artifact that has been rebuilt locally from bytes over the wire) for use.

      Parameters:
      remoteRelativePath -
      serverLocalStagingPath -
      fragment -
      deleteStagedFile -

    • classloader

      public ClassLoader classloader()
      Returns a ClassLoader for session-specific jar/class file resources.
      Returns:
      (undocumented)

    • uploadArtifactToFs

      public void uploadArtifactToFs(Path remoteRelativePath, Path serverLocalStagingPath)