Package org.apache.spark.util

Class DependencyUtils

Object
org.apache.spark.util.DependencyUtils
public class DependencyUtils extends Object

  • Constructor Details

    • DependencyUtils

      public DependencyUtils()

  • Method Details

    • getIvyProperties

      public static org.apache.spark.util.IvyProperties getIvyProperties()

    • resolveMavenDependencies

      public static scala.collection.Seq<String> resolveMavenDependencies(URI uri)
      Download Ivy URI's dependency jars.

      Parameters:
      uri - Ivy URI need to be downloaded. The URI format should be: ivy://group:module:version[?query] Ivy URI query part format should be: parameter=value&parameter=value... Note that currently Ivy URI query part support two parameters: 1. transitive: whether to download dependent jars related to your Ivy URI. transitive=false or transitive=true, if not set, the default value is true. 2. exclude: exclusion list when download Ivy URI jar and dependency jars. The exclude parameter content is a ',' separated group:module pair string : exclude=group:module,group:module...
      Returns:
      List of jars downloaded.

    • resolveMavenDependencies

      public static scala.collection.Seq<String> resolveMavenDependencies(boolean packagesTransitive, String packagesExclusions, String packages, String repositories, String ivyRepoPath, scala.Option<String> ivySettingsPath)

    • resolveAndDownloadJars

      public static String resolveAndDownloadJars(String jars, String userJar, SparkConf sparkConf, org.apache.hadoop.conf.Configuration hadoopConf)

    • addJarsToClassPath

      public static void addJarsToClassPath(String jars, MutableURLClassLoader loader)

    • downloadFileList

      public static String downloadFileList(String fileList, File targetDir, SparkConf sparkConf, org.apache.hadoop.conf.Configuration hadoopConf)
      Download a list of remote files to temp local files. If the file is local, the original file will be returned.

      Parameters:
      fileList - A comma separated file list.
      targetDir - A temporary directory for which downloaded files.
      sparkConf - Spark configuration.
      hadoopConf - Hadoop configuration.
      Returns:
      A comma separated local files list.

    • downloadFile

      public static String downloadFile(String path, File targetDir, SparkConf sparkConf, org.apache.hadoop.conf.Configuration hadoopConf)
      Download a file from the remote to a local temporary directory. If the input path points to a local path, returns it with no operation.

      Parameters:
      path - A file path from where the files will be downloaded.
      targetDir - A temporary directory for which downloaded files.
      sparkConf - Spark configuration.
      hadoopConf - Hadoop configuration.
      Returns:
      Path to the local file.

    • resolveGlobPaths

      public static String resolveGlobPaths(String paths, org.apache.hadoop.conf.Configuration hadoopConf)

    • addJarToClasspath

      public static void addJarToClasspath(String localJar, MutableURLClassLoader loader)

    • mergeFileLists

      public static String mergeFileLists(scala.collection.Seq<String> lists)
      Merge a sequence of comma-separated file lists, some of which may be null to indicate no files, into a single comma-separated string.
      Parameters:
      lists - (undocumented)
      Returns:
      (undocumented)

    • org$apache$spark$internal$Logging$$log_

      public static org.slf4j.Logger org$apache$spark$internal$Logging$$log_()

    • org$apache$spark$internal$Logging$$log__$eq

      public static void org$apache$spark$internal$Logging$$log__$eq(org.slf4j.Logger x$1)