Package org.apache.spark.io
Class HadoopCodecStreams
Object
org.apache.spark.io.HadoopCodecStreams
An utility object to look up Hadoop compression codecs and create input streams.
In addition to standard Hadoop codecs, it also supports Spark's Zstandard codec
if Hadopp is not compiled with Zstandard support. Additionally, it supports
non-standard file extensions like
.zstd
and .gzip
for Zstandard and Gzip codecs.-
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionstatic InputStream
createInputStream
(org.apache.hadoop.conf.Configuration config, org.apache.hadoop.fs.Path file) static scala.Option<InputStream>
createZstdInputStream
(org.apache.hadoop.fs.Path file, InputStream inputStream) static scala.Option<org.apache.hadoop.io.compress.CompressionCodec>
getDecompressionCodec
(org.apache.hadoop.conf.Configuration config, org.apache.hadoop.fs.Path file)
-
Constructor Details
-
HadoopCodecStreams
public HadoopCodecStreams()
-
-
Method Details
-
getDecompressionCodec
public static scala.Option<org.apache.hadoop.io.compress.CompressionCodec> getDecompressionCodec(org.apache.hadoop.conf.Configuration config, org.apache.hadoop.fs.Path file) -
createZstdInputStream
public static scala.Option<InputStream> createZstdInputStream(org.apache.hadoop.fs.Path file, InputStream inputStream) -
createInputStream
public static InputStream createInputStream(org.apache.hadoop.conf.Configuration config, org.apache.hadoop.fs.Path file)
-