Package org.apache.flink.runtime.fs.hdfs

Class HadoopRecoverableWriter

java.lang.Object

org.apache.flink.runtime.fs.hdfs.HadoopRecoverableWriter

All Implemented Interfaces:

org.apache.flink.core.fs.RecoverableWriter

@Internal
public class HadoopRecoverableWriter
extends Object
implements org.apache.flink.core.fs.RecoverableWriter

An implementation of the RecoverableWriter for Hadoop's file system abstraction.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.core.fs.RecoverableWriter

org.apache.flink.core.fs.RecoverableWriter.CommitRecoverable, org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable

Field Summary

Fields

Modifier and Type	Field	Description
protected org.apache.hadoop.fs.FileSystem	fs	The Hadoop file system on which the writer operates.

Constructor Summary

Constructors

Constructor	Description
HadoopRecoverableWriter(org.apache.hadoop.fs.FileSystem fs)	Creates a new Recoverable writer.
HadoopRecoverableWriter(org.apache.hadoop.fs.FileSystem fs, boolean noLocalWrite)

Method Summary

Modifier and Type	Method	Description
protected void	checkSupportedFSSchemes(org.apache.hadoop.fs.FileSystem fs)
boolean	cleanupRecoverableState(org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable resumable)
org.apache.flink.core.io.SimpleVersionedSerializer<org.apache.flink.core.fs.RecoverableWriter.CommitRecoverable>	getCommitRecoverableSerializer()
protected org.apache.flink.core.fs.RecoverableFsDataOutputStream	getRecoverableFsDataOutputStream(org.apache.hadoop.fs.Path targetFile, org.apache.hadoop.fs.Path tempFile)
org.apache.flink.core.io.SimpleVersionedSerializer<org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable>	getResumeRecoverableSerializer()
org.apache.flink.core.fs.RecoverableFsDataOutputStream	open(org.apache.flink.core.fs.Path filePath)
org.apache.flink.core.fs.RecoverableFsDataOutputStream	recover(org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable recoverable)
org.apache.flink.core.fs.RecoverableFsDataOutputStream.Committer	recoverForCommit(org.apache.flink.core.fs.RecoverableWriter.CommitRecoverable recoverable)
boolean	requiresCleanupOfRecoverableState()
boolean	supportsResume()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

fs

protected final org.apache.hadoop.fs.FileSystem fs

The Hadoop file system on which the writer operates.

Constructor Detail

HadoopRecoverableWriter

public HadoopRecoverableWriter(org.apache.hadoop.fs.FileSystem fs)

Creates a new Recoverable writer.

Parameters:

fs - The Hadoop file system on which the writer operates.

HadoopRecoverableWriter

public HadoopRecoverableWriter(org.apache.hadoop.fs.FileSystem fs,
                               boolean noLocalWrite)

Method Detail

checkSupportedFSSchemes

protected void checkSupportedFSSchemes(org.apache.hadoop.fs.FileSystem fs)

open

public org.apache.flink.core.fs.RecoverableFsDataOutputStream open(org.apache.flink.core.fs.Path filePath)
                                                            throws IOException

Specified by:

open in interface org.apache.flink.core.fs.RecoverableWriter

Throws:

IOException

getRecoverableFsDataOutputStream

protected org.apache.flink.core.fs.RecoverableFsDataOutputStream getRecoverableFsDataOutputStream(org.apache.hadoop.fs.Path targetFile,
                                                                                                  org.apache.hadoop.fs.Path tempFile)
                                                                                           throws IOException

Throws:

IOException

recover

public org.apache.flink.core.fs.RecoverableFsDataOutputStream recover(org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable recoverable)
                                                               throws IOException

Specified by:

recover in interface org.apache.flink.core.fs.RecoverableWriter

Throws:

IOException

requiresCleanupOfRecoverableState

public boolean requiresCleanupOfRecoverableState()

Specified by:

requiresCleanupOfRecoverableState in interface org.apache.flink.core.fs.RecoverableWriter

cleanupRecoverableState

public boolean cleanupRecoverableState(org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable resumable)
                                throws IOException

Specified by:

cleanupRecoverableState in interface org.apache.flink.core.fs.RecoverableWriter

Throws:

IOException

recoverForCommit

public org.apache.flink.core.fs.RecoverableFsDataOutputStream.Committer recoverForCommit(org.apache.flink.core.fs.RecoverableWriter.CommitRecoverable recoverable)
                                                                                  throws IOException

Specified by:

recoverForCommit in interface org.apache.flink.core.fs.RecoverableWriter

Throws:

IOException

getCommitRecoverableSerializer

public org.apache.flink.core.io.SimpleVersionedSerializer<org.apache.flink.core.fs.RecoverableWriter.CommitRecoverable> getCommitRecoverableSerializer()

Specified by:

getCommitRecoverableSerializer in interface org.apache.flink.core.fs.RecoverableWriter

getResumeRecoverableSerializer

public org.apache.flink.core.io.SimpleVersionedSerializer<org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable> getResumeRecoverableSerializer()

Specified by:

getResumeRecoverableSerializer in interface org.apache.flink.core.fs.RecoverableWriter

supportsResume

public boolean supportsResume()

Specified by:

supportsResume in interface org.apache.flink.core.fs.RecoverableWriter