org.apache.flink.api.java.operators

Class GroupCombineOperator<IN,OUT>

java.lang.Object

org.apache.flink.api.java.DataSet<OUT>

org.apache.flink.api.java.operators.Operator<OUT,O>

org.apache.flink.api.java.operators.SingleInputOperator<IN,OUT,O>

org.apache.flink.api.java.operators.SingleInputUdfOperator<IN,OUT,GroupCombineOperator<IN,OUT>>

org.apache.flink.api.java.operators.GroupCombineOperator<IN,OUT>

Type Parameters:

IN - The type of the data set consumed by the operator.

OUT - The type of the data set created by the operator.

All Implemented Interfaces:

UdfOperator<GroupCombineOperator<IN,OUT>>

@Public
public class GroupCombineOperator<IN,OUT>
extends SingleInputUdfOperator<IN,OUT,GroupCombineOperator<IN,OUT>>

This operator behaves like the GroupReduceOperator with Combine but only runs the Combine part which reduces all data locally in their partitions. The combine part can return an arbitrary data type. This is useful to pre-combine values into an intermediate representation before applying a proper reduce operation.

Field Summary

Fields inherited from class org.apache.flink.api.java.operators.Operator

minResources, name, parallelism, preferredResources

Fields inherited from class org.apache.flink.api.java.DataSet

context

Constructor Summary

Constructors

Constructor and Description
GroupCombineOperator(DataSet<IN> input, org.apache.flink.api.common.typeinfo.TypeInformation<OUT> resultType, org.apache.flink.api.common.functions.GroupCombineFunction<IN,OUT> function, String defaultName) Constructor for a non-grouped reduce (all reduce).
GroupCombineOperator(Grouping<IN> input, org.apache.flink.api.common.typeinfo.TypeInformation<OUT> resultType, org.apache.flink.api.common.functions.GroupCombineFunction<IN,OUT> function, String defaultName) Constructor for a grouped reduce.

Method Summary

Modifier and Type	Method and Description
protected org.apache.flink.api.common.functions.GroupCombineFunction<IN,OUT>	getFunction()
org.apache.flink.api.common.operators.SingleInputSemanticProperties	getSemanticProperties() Gets the semantic properties that have been set for the user-defined functions (UDF).
protected org.apache.flink.api.common.operators.base.GroupCombineOperatorBase<?,OUT,?>	translateToDataFlow(org.apache.flink.api.common.operators.Operator<IN> input) Translates this operation to a data flow operator of the common data flow API.

Methods inherited from class org.apache.flink.api.java.operators.SingleInputUdfOperator

extractSemanticAnnotations, getAnalyzedUdfSemanticsFlag, getBroadcastSets, getParameters, returns, returns, returns, setAnalyzedUdfSemanticsFlag, setSemanticProperties, udfWithForwardedFieldsAnnotation, withBroadcastSet, withForwardedFields, withParameters

Methods inherited from class org.apache.flink.api.java.operators.SingleInputOperator

getInput, getInputType

Methods inherited from class org.apache.flink.api.java.operators.Operator

getMinResources, getName, getParallelism, getPreferredResources, getResultType, name, setParallelism

Methods inherited from class org.apache.flink.api.java.DataSet

aggregate, checkSameExecutionContext, clean, coGroup, collect, combineGroup, count, cross, crossWithHuge, crossWithTiny, distinct, distinct, distinct, distinct, fillInType, filter, first, flatMap, fullOuterJoin, fullOuterJoin, getExecutionEnvironment, getType, groupBy, groupBy, groupBy, iterate, iterateDelta, join, join, joinWithHuge, joinWithTiny, leftOuterJoin, leftOuterJoin, map, mapPartition, max, maxBy, min, minBy, output, partitionByHash, partitionByHash, partitionByHash, partitionByRange, partitionByRange, partitionByRange, partitionCustom, partitionCustom, partitionCustom, print, print, printOnTaskManager, printToErr, printToErr, project, rebalance, reduce, reduceGroup, rightOuterJoin, rightOuterJoin, runOperation, sortPartition, sortPartition, sortPartition, sum, union, write, write, writeAsCsv, writeAsCsv, writeAsCsv, writeAsCsv, writeAsFormattedText, writeAsFormattedText, writeAsText, writeAsText

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

GroupCombineOperator

public GroupCombineOperator(DataSet<IN> input,
                            org.apache.flink.api.common.typeinfo.TypeInformation<OUT> resultType,
                            org.apache.flink.api.common.functions.GroupCombineFunction<IN,OUT> function,
                            String defaultName)

Constructor for a non-grouped reduce (all reduce).

Parameters:

input - The input data set to the groupReduce function.

resultType - The type information for the resulting type.

function - The user-defined GroupReduce function.

defaultName - The operator's name.

GroupCombineOperator

public GroupCombineOperator(Grouping<IN> input,
                            org.apache.flink.api.common.typeinfo.TypeInformation<OUT> resultType,
                            org.apache.flink.api.common.functions.GroupCombineFunction<IN,OUT> function,
                            String defaultName)

Constructor for a grouped reduce.

Parameters:

input - The grouped input to be processed group-wise by the groupReduce function.

function - The user-defined GroupReduce function.

Method Detail

getFunction

protected org.apache.flink.api.common.functions.GroupCombineFunction<IN,OUT> getFunction()

Specified by:

getFunction in class SingleInputUdfOperator<IN,OUT,GroupCombineOperator<IN,OUT>>

getSemanticProperties

@Internal
public org.apache.flink.api.common.operators.SingleInputSemanticProperties getSemanticProperties()

Description copied from interface: UdfOperator

Gets the semantic properties that have been set for the user-defined functions (UDF).

Specified by:

getSemanticProperties in interface UdfOperator<GroupCombineOperator<IN,OUT>>

Overrides:

getSemanticProperties in class SingleInputUdfOperator<IN,OUT,GroupCombineOperator<IN,OUT>>

Returns:

The semantic properties of the UDF.

translateToDataFlow

protected org.apache.flink.api.common.operators.base.GroupCombineOperatorBase<?,OUT,?> translateToDataFlow(org.apache.flink.api.common.operators.Operator<IN> input)

Description copied from class: SingleInputOperator

Translates this operation to a data flow operator of the common data flow API.

Specified by:

translateToDataFlow in class SingleInputOperator<IN,OUT,GroupCombineOperator<IN,OUT>>

Parameters:

input - The data flow operator that produces this operation's input data.

Returns:

The translated data flow operator.