Class DependencyUtils


public class DependencyUtils extends Object
  • Constructor Details

    • DependencyUtils

      public DependencyUtils()
  • Method Details

    • getIvyProperties

      public static org.apache.spark.util.IvyProperties getIvyProperties()
    • resolveMavenDependencies

      public static scala.collection.Seq<String> resolveMavenDependencies(URI uri)
      Download Ivy URI's dependency jars.

      uri - Ivy URI need to be downloaded. The URI format should be: ivy://group:module:version[?query] Ivy URI query part format should be: parameter=value&parameter=value... Note that currently Ivy URI query part support two parameters: 1. transitive: whether to download dependent jars related to your Ivy URI. transitive=false or transitive=true, if not set, the default value is true. 2. exclude: exclusion list when download Ivy URI jar and dependency jars. The exclude parameter content is a ',' separated group:module pair string : exclude=group:module,group:module...
      List of jars downloaded.
    • resolveMavenDependencies

      public static scala.collection.Seq<String> resolveMavenDependencies(boolean packagesTransitive, String packagesExclusions, String packages, String repositories, String ivyRepoPath, scala.Option<String> ivySettingsPath)
    • resolveAndDownloadJars

      public static String resolveAndDownloadJars(String jars, String userJar, SparkConf sparkConf, org.apache.hadoop.conf.Configuration hadoopConf)
    • addJarsToClassPath

      public static void addJarsToClassPath(String jars, MutableURLClassLoader loader)
    • downloadFileList

      public static String downloadFileList(String fileList, File targetDir, SparkConf sparkConf, org.apache.hadoop.conf.Configuration hadoopConf)
      Download a list of remote files to temp local files. If the file is local, the original file will be returned.

      fileList - A comma separated file list.
      targetDir - A temporary directory for which downloaded files.
      sparkConf - Spark configuration.
      hadoopConf - Hadoop configuration.
      A comma separated local files list.
    • downloadFile

      public static String downloadFile(String path, File targetDir, SparkConf sparkConf, org.apache.hadoop.conf.Configuration hadoopConf)
      Download a file from the remote to a local temporary directory. If the input path points to a local path, returns it with no operation.

      path - A file path from where the files will be downloaded.
      targetDir - A temporary directory for which downloaded files.
      sparkConf - Spark configuration.
      hadoopConf - Hadoop configuration.
      Path to the local file.
    • resolveGlobPaths

      public static String resolveGlobPaths(String paths, org.apache.hadoop.conf.Configuration hadoopConf)
    • addJarToClasspath

      public static void addJarToClasspath(String localJar, MutableURLClassLoader loader)
    • mergeFileLists

      public static String mergeFileLists(scala.collection.Seq<String> lists)
      Merge a sequence of comma-separated file lists, some of which may be null to indicate no files, into a single comma-separated string.
      lists - (undocumented)
    • org$apache$spark$internal$Logging$$log_

      public static org.slf4j.Logger org$apache$spark$internal$Logging$$log_()
    • org$apache$spark$internal$Logging$$log__$eq

      public static void org$apache$spark$internal$Logging$$log__$eq(org.slf4j.Logger x$1)