Class HadoopCodecStreams

Object
org.apache.spark.io.HadoopCodecStreams

public class HadoopCodecStreams extends Object
An utility object to look up Hadoop compression codecs and create input streams. In addition to standard Hadoop codecs, it also supports Spark's Zstandard codec if Hadopp is not compiled with Zstandard support. Additionally, it supports non-standard file extensions like .zstd and .gzip for Zstandard and Gzip codecs.
  • Constructor Details

    • HadoopCodecStreams

      public HadoopCodecStreams()
  • Method Details

    • getDecompressionCodec

      public static scala.Option<org.apache.hadoop.io.compress.CompressionCodec> getDecompressionCodec(org.apache.hadoop.conf.Configuration config, org.apache.hadoop.fs.Path file)
    • createZstdInputStream

      public static scala.Option<InputStream> createZstdInputStream(org.apache.hadoop.fs.Path file, InputStream inputStream)
    • createInputStream

      public static InputStream createInputStream(org.apache.hadoop.conf.Configuration config, org.apache.hadoop.fs.Path file)