Package org.apache.spark.io

Class HadoopCodecStreams

Object

org.apache.spark.io.HadoopCodecStreams

public class HadoopCodecStreams
extends Object

An utility object to look up Hadoop compression codecs and create input streams. In addition to standard Hadoop codecs, it also supports Spark's Zstandard codec if Hadopp is not compiled with Zstandard support. Additionally, it supports non-standard file extensions like .zstd and .gzip for Zstandard and Gzip codecs.

Constructor Summary

Constructors

Constructor	Description
HadoopCodecStreams()

Method Summary

Modifier and Type	Method	Description
static InputStream	createInputStream(org.apache.hadoop.conf.Configuration config, org.apache.hadoop.fs.Path file)
static scala.Option<InputStream>	createZstdInputStream(org.apache.hadoop.fs.Path file, InputStream inputStream)
static scala.Option<org.apache.hadoop.io.compress.CompressionCodec>	getDecompressionCodec(org.apache.hadoop.conf.Configuration config, org.apache.hadoop.fs.Path file)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

HadoopCodecStreams

public HadoopCodecStreams()

Method Details

getDecompressionCodec

public static scala.Option<org.apache.hadoop.io.compress.CompressionCodec> getDecompressionCodec(org.apache.hadoop.conf.Configuration config,
 org.apache.hadoop.fs.Path file)

createZstdInputStream

public static scala.Option<InputStream> createZstdInputStream(org.apache.hadoop.fs.Path file,
 InputStream inputStream)

createInputStream

public static InputStream createInputStream(org.apache.hadoop.conf.Configuration config,
 org.apache.hadoop.fs.Path file)