Package org.apache.flink.runtime.fs.hdfs

Class HadoopFileSystem

java.lang.Object

org.apache.flink.core.fs.FileSystem

org.apache.flink.runtime.fs.hdfs.HadoopFileSystem

All Implemented Interfaces:

org.apache.flink.core.fs.IFileSystem

public class HadoopFileSystem
extends org.apache.flink.core.fs.FileSystem

A FileSystem that wraps an Hadoop File System.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.core.fs.FileSystem

org.apache.flink.core.fs.FileSystem.FSKey, org.apache.flink.core.fs.FileSystem.WriteMode

Constructor Summary

Constructors

Constructor	Description
HadoopFileSystem(org.apache.hadoop.fs.FileSystem hadoopFileSystem)	Wraps the given Hadoop File System object as a Flink File System object.

Method Summary

Modifier and Type	Method	Description
HadoopDataOutputStream	create(org.apache.flink.core.fs.Path f, boolean overwrite, int bufferSize, short replication, long blockSize)
HadoopDataOutputStream	create(org.apache.flink.core.fs.Path f, org.apache.flink.core.fs.FileSystem.WriteMode overwrite)
org.apache.flink.core.fs.RecoverableWriter	createRecoverableWriter()
org.apache.flink.core.fs.RecoverableWriter	createRecoverableWriter(Map<String,String> conf)
boolean	delete(org.apache.flink.core.fs.Path f, boolean recursive)
boolean	exists(org.apache.flink.core.fs.Path f)
long	getDefaultBlockSize()
org.apache.flink.core.fs.BlockLocation[]	getFileBlockLocations(org.apache.flink.core.fs.FileStatus file, long start, long len)
org.apache.flink.core.fs.FileStatus	getFileStatus(org.apache.flink.core.fs.Path f)
org.apache.hadoop.fs.FileSystem	getHadoopFileSystem()	Gets the underlying Hadoop FileSystem.
org.apache.flink.core.fs.Path	getHomeDirectory()
URI	getUri()
org.apache.flink.core.fs.Path	getWorkingDirectory()
boolean	isDistributedFS()
org.apache.flink.core.fs.FileStatus[]	listStatus(org.apache.flink.core.fs.Path f)
boolean	mkdirs(org.apache.flink.core.fs.Path f)
HadoopDataInputStream	open(org.apache.flink.core.fs.Path f)
HadoopDataInputStream	open(org.apache.flink.core.fs.Path f, int bufferSize)
boolean	rename(org.apache.flink.core.fs.Path src, org.apache.flink.core.fs.Path dst)
static org.apache.hadoop.fs.Path	toHadoopPath(org.apache.flink.core.fs.Path path)

Methods inherited from class org.apache.flink.core.fs.FileSystem

create, get, getDefaultFsUri, getLocalFileSystem, getUnguardedFileSystem, initialize, initialize, initOutPathDistFS, initOutPathLocalFS

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.core.fs.IFileSystem

canCopyPaths

Constructor Detail

HadoopFileSystem

public HadoopFileSystem(org.apache.hadoop.fs.FileSystem hadoopFileSystem)

Wraps the given Hadoop File System object as a Flink File System object. The given Hadoop file system object is expected to be initialized already.

Parameters:

hadoopFileSystem - The Hadoop FileSystem that will be used under the hood.

Method Detail

getHadoopFileSystem

public org.apache.hadoop.fs.FileSystem getHadoopFileSystem()

Gets the underlying Hadoop FileSystem.

Returns:

The underlying Hadoop FileSystem.

getWorkingDirectory

public org.apache.flink.core.fs.Path getWorkingDirectory()

Specified by:

getWorkingDirectory in interface org.apache.flink.core.fs.IFileSystem

Specified by:

getWorkingDirectory in class org.apache.flink.core.fs.FileSystem

getHomeDirectory

public org.apache.flink.core.fs.Path getHomeDirectory()

Specified by:

getHomeDirectory in interface org.apache.flink.core.fs.IFileSystem

Specified by:

getHomeDirectory in class org.apache.flink.core.fs.FileSystem

getUri

public URI getUri()

Specified by:

getUri in interface org.apache.flink.core.fs.IFileSystem

Specified by:

getUri in class org.apache.flink.core.fs.FileSystem

getFileStatus

public org.apache.flink.core.fs.FileStatus getFileStatus(org.apache.flink.core.fs.Path f)
                                                  throws IOException

Specified by:

getFileStatus in interface org.apache.flink.core.fs.IFileSystem

Specified by:

getFileStatus in class org.apache.flink.core.fs.FileSystem

Throws:

IOException

getFileBlockLocations

public org.apache.flink.core.fs.BlockLocation[] getFileBlockLocations(org.apache.flink.core.fs.FileStatus file,
                                                                      long start,
                                                                      long len)
                                                               throws IOException

Specified by:

getFileBlockLocations in interface org.apache.flink.core.fs.IFileSystem

Specified by:

getFileBlockLocations in class org.apache.flink.core.fs.FileSystem

Throws:

IOException

open

public HadoopDataInputStream open(org.apache.flink.core.fs.Path f,
                                  int bufferSize)
                           throws IOException

Specified by:

open in interface org.apache.flink.core.fs.IFileSystem

Specified by:

open in class org.apache.flink.core.fs.FileSystem

Throws:

IOException

open

public HadoopDataInputStream open(org.apache.flink.core.fs.Path f)
                           throws IOException

Specified by:

open in interface org.apache.flink.core.fs.IFileSystem

Specified by:

open in class org.apache.flink.core.fs.FileSystem

Throws:

IOException

create

public HadoopDataOutputStream create(org.apache.flink.core.fs.Path f,
                                     boolean overwrite,
                                     int bufferSize,
                                     short replication,
                                     long blockSize)
                              throws IOException

Overrides:

create in class org.apache.flink.core.fs.FileSystem

Throws:

IOException

create

public HadoopDataOutputStream create(org.apache.flink.core.fs.Path f,
                                     org.apache.flink.core.fs.FileSystem.WriteMode overwrite)
                              throws IOException

Specified by:

create in interface org.apache.flink.core.fs.IFileSystem

Specified by:

create in class org.apache.flink.core.fs.FileSystem

Throws:

IOException

delete

public boolean delete(org.apache.flink.core.fs.Path f,
                      boolean recursive)
               throws IOException

Specified by:

delete in interface org.apache.flink.core.fs.IFileSystem

Specified by:

delete in class org.apache.flink.core.fs.FileSystem

Throws:

IOException

exists

public boolean exists(org.apache.flink.core.fs.Path f)
               throws IOException

Specified by:

exists in interface org.apache.flink.core.fs.IFileSystem

Overrides:

exists in class org.apache.flink.core.fs.FileSystem

Throws:

IOException

listStatus

public org.apache.flink.core.fs.FileStatus[] listStatus(org.apache.flink.core.fs.Path f)
                                                 throws IOException

Specified by:

listStatus in interface org.apache.flink.core.fs.IFileSystem

Specified by:

listStatus in class org.apache.flink.core.fs.FileSystem

Throws:

IOException

mkdirs

public boolean mkdirs(org.apache.flink.core.fs.Path f)
               throws IOException

Specified by:

mkdirs in interface org.apache.flink.core.fs.IFileSystem

Specified by:

mkdirs in class org.apache.flink.core.fs.FileSystem

Throws:

IOException

rename

public boolean rename(org.apache.flink.core.fs.Path src,
                      org.apache.flink.core.fs.Path dst)
               throws IOException

Specified by:

rename in interface org.apache.flink.core.fs.IFileSystem

Specified by:

rename in class org.apache.flink.core.fs.FileSystem

Throws:

IOException

getDefaultBlockSize

public long getDefaultBlockSize()

Overrides:

getDefaultBlockSize in class org.apache.flink.core.fs.FileSystem

isDistributedFS

public boolean isDistributedFS()

Specified by:

isDistributedFS in interface org.apache.flink.core.fs.IFileSystem

Specified by:

isDistributedFS in class org.apache.flink.core.fs.FileSystem

createRecoverableWriter

public org.apache.flink.core.fs.RecoverableWriter createRecoverableWriter()
                                                                   throws IOException

Specified by:

createRecoverableWriter in interface org.apache.flink.core.fs.IFileSystem

Overrides:

createRecoverableWriter in class org.apache.flink.core.fs.FileSystem

Throws:

IOException

createRecoverableWriter

public org.apache.flink.core.fs.RecoverableWriter createRecoverableWriter(Map<String,String> conf)
                                                                   throws IOException

Specified by:

createRecoverableWriter in interface org.apache.flink.core.fs.IFileSystem

Overrides:

createRecoverableWriter in class org.apache.flink.core.fs.FileSystem

Throws:

IOException

toHadoopPath

public static org.apache.hadoop.fs.Path toHadoopPath(org.apache.flink.core.fs.Path path)