Package org.apache.flink.streaming.api.operators.async

Class AsyncWaitOperator<IN,​OUT>

  • Type Parameters:
    IN - Input type for the operator.
    OUT - Output type for the operator.
    All Implemented Interfaces:
    Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.BoundedOneInput, org.apache.flink.streaming.api.operators.Input<IN>, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.KeyContextHandler, org.apache.flink.streaming.api.operators.OneInputStreamOperator<IN,​OUT>, org.apache.flink.streaming.api.operators.OutputTypeConfigurable<OUT>, org.apache.flink.streaming.api.operators.StreamOperator<OUT>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, org.apache.flink.streaming.api.operators.UserFunctionProvider<AsyncFunction<IN,​OUT>>, org.apache.flink.streaming.api.operators.YieldingOperator<OUT>
    @Internal
public class AsyncWaitOperator<IN,​OUT>
extends org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator<OUT,​AsyncFunction<IN,​OUT>>
implements org.apache.flink.streaming.api.operators.OneInputStreamOperator<IN,​OUT>, org.apache.flink.streaming.api.operators.BoundedOneInput
    The AsyncWaitOperator allows to asynchronously process incoming stream records. For that the operator creates an ResultFuture which is passed to an AsyncFunction. Within the async function, the user can complete the async collector arbitrarily. Once the async collector has been completed, the result is emitted by the operator's emitter to downstream operators.

    The operator offers different output modes depending on the chosen AsyncDataStream.OutputMode. In order to give exactly once processing guarantees, the operator stores all currently in-flight StreamElement in it's operator state. Upon recovery the recorded set of stream elements is replayed.

    The operator also support retry on predefined condition and strategy AsyncRetryStrategy. If given strategy is NO_RETRY_STRATEGY or a custom implemented AsyncRetryPredicate which defines empty retry predicate, then no retry will happen, everything keeps the same as the AsyncWaitOperator without retry support.

    In case of chaining of this operator, it has to be made sure that the operators in the chain are opened tail to head. The reason for this is that an opened AsyncWaitOperator starts already emitting recovered StreamElement to downstream operators.

    See Also:
    Serialized Form

    • Field Summary

      • Fields inherited from class org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator

        userFunction

      • Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

        combinedWatermark, config, lastRecordAttributes1, lastRecordAttributes2, latencyStats, LOG, metrics, output, processingTimeService, stateHandler, stateKeySelector1, stateKeySelector2, timeServiceManager

    • Constructor Summary

      Constructors 
      Constructor Description
      AsyncWaitOperator​(org.apache.flink.streaming.api.operators.StreamOperatorParameters<OUT> parameters, AsyncFunction<IN,​OUT> asyncFunction, long timeout, int capacity, AsyncDataStream.OutputMode outputMode, AsyncRetryStrategy<OUT> asyncRetryStrategy, org.apache.flink.streaming.runtime.tasks.ProcessingTimeService processingTimeService, org.apache.flink.api.common.operators.MailboxExecutor mailboxExecutor)  

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void endInput()  
      void initializeState​(org.apache.flink.runtime.state.StateInitializationContext context)  
      void open()  
      void processElement​(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<IN> record)  
      void processWatermark​(org.apache.flink.streaming.api.watermark.Watermark mark)  
      protected void setup​(org.apache.flink.streaming.runtime.tasks.StreamTask<?,​?> containingTask, org.apache.flink.streaming.api.graph.StreamConfig config, org.apache.flink.streaming.api.operators.Output<org.apache.flink.streaming.runtime.streamrecord.StreamRecord<OUT>> output)  
      void snapshotState​(org.apache.flink.runtime.state.StateSnapshotContext context)  

      • Methods inherited from class org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator

        close, finish, getUserFunction, getUserFunctionParameters, notifyCheckpointAborted, notifyCheckpointComplete, setOutputType

      • Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

        beforeInitializeStateHandler, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getStateKeySelector1, getStateKeySelector2, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, isAsyncKeyOrderedProcessingEnabled, isUsingCustomRawKeyedState, prepareSnapshotPreBarrier, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processRecordAttributes, processRecordAttributes1, processRecordAttributes2, processWatermark, processWatermark1, processWatermark1, processWatermark2, processWatermark2, processWatermarkStatus, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setMailboxExecutor, setProcessingTimeService, snapshotState, useInterruptibleTimers

      • Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

        notifyCheckpointAborted, notifyCheckpointComplete

      • Methods inherited from interface org.apache.flink.streaming.api.operators.Input

        processLatencyMarker, processRecordAttributes, processWatermark, processWatermarkStatus

      • Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

        getCurrentKey, setCurrentKey

      • Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

        hasKeyContext

      • Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

        setKeyContextElement

      • Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

        close, finish, getMetricGroup, getOperatorAttributes, getOperatorID, initializeState, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState

    • Constructor Detail

      • AsyncWaitOperator

        public AsyncWaitOperator​(org.apache.flink.streaming.api.operators.StreamOperatorParameters<OUT> parameters,
                         @Nonnull
                         AsyncFunction<IN,​OUT> asyncFunction,
                         long timeout,
                         int capacity,
                         @Nonnull
                         AsyncDataStream.OutputMode outputMode,
                         @Nonnull
                         AsyncRetryStrategy<OUT> asyncRetryStrategy,
                         @Nonnull
                         org.apache.flink.streaming.runtime.tasks.ProcessingTimeService processingTimeService,
                         @Nonnull
                         org.apache.flink.api.common.operators.MailboxExecutor mailboxExecutor)

    • Method Detail

      • setup

        protected void setup​(org.apache.flink.streaming.runtime.tasks.StreamTask<?,​?> containingTask,
                     org.apache.flink.streaming.api.graph.StreamConfig config,
                     org.apache.flink.streaming.api.operators.Output<org.apache.flink.streaming.runtime.streamrecord.StreamRecord<OUT>> output)
        Overrides:
        setup in class org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator<OUT,​AsyncFunction<IN,​OUT>>

      • open

        public void open()
          throws Exception
        Specified by:
        open in interface org.apache.flink.streaming.api.operators.StreamOperator<IN>
        Overrides:
        open in class org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator<OUT,​AsyncFunction<IN,​OUT>>
        Throws:
        Exception

      • processElement

        public void processElement​(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<IN> record)
                    throws Exception
        Specified by:
        processElement in interface org.apache.flink.streaming.api.operators.Input<IN>
        Throws:
        Exception

      • processWatermark

        public void processWatermark​(org.apache.flink.streaming.api.watermark.Watermark mark)
                      throws Exception
        Specified by:
        processWatermark in interface org.apache.flink.streaming.api.operators.Input<IN>
        Overrides:
        processWatermark in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
        Throws:
        Exception

      • snapshotState

        public void snapshotState​(org.apache.flink.runtime.state.StateSnapshotContext context)
                   throws Exception
        Specified by:
        snapshotState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator
        Overrides:
        snapshotState in class org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator<OUT,​AsyncFunction<IN,​OUT>>
        Throws:
        Exception

      • initializeState

        public void initializeState​(org.apache.flink.runtime.state.StateInitializationContext context)
                     throws Exception
        Specified by:
        initializeState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator
        Overrides:
        initializeState in class org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator<OUT,​AsyncFunction<IN,​OUT>>
        Throws:
        Exception

      • endInput

        public void endInput()
              throws Exception
        Specified by:
        endInput in interface org.apache.flink.streaming.api.operators.BoundedOneInput
        Throws:
        Exception