org.apache.flink.connector.kafka.dynamic.source

Class DynamicKafkaSource<T>

java.lang.Object

org.apache.flink.connector.kafka.dynamic.source.DynamicKafkaSource<T>

Type Parameters:

T - Record type

All Implemented Interfaces:

Serializable, org.apache.flink.api.connector.source.Source<T,DynamicKafkaSourceSplit,DynamicKafkaSourceEnumState>, org.apache.flink.api.connector.source.SourceReaderFactory<T,DynamicKafkaSourceSplit>, org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>

@Experimental
public class DynamicKafkaSource<T>
extends Object
implements org.apache.flink.api.connector.source.Source<T,DynamicKafkaSourceSplit,DynamicKafkaSourceEnumState>, org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>

Factory class for the DynamicKafkaSource components. FLIP-246: DynamicKafkaSource

This source's key difference from KafkaSource is that it enables users to read dynamically, which does not require job restart, from streams (topics that belong to one or more clusters). If using KafkaSource, users need to restart the job by deleting the job and reconfiguring the topics and clusters.

This example shows how to configure a DynamicKafkaSource that emits Integer records:

 DynamicKafkaSource<Integer> dynamicKafkaSource =
                     DynamicKafkaSource.<Integer>builder()
                             .setStreamIds(Collections.singleton("MY_STREAM_ID"))
                             // custom metadata service that resolves `MY_STREAM_ID` to the associated clusters and topics
                             .setKafkaMetadataService(kafkaMetadataService)
                             .setDeserializer(
                                     KafkaRecordDeserializationSchema.valueOnly(
                                             IntegerDeserializer.class))
                             .setStartingOffsets(OffsetsInitializer.earliest())
                             // common properties for all Kafka clusters
                             .setProperties(properties)
                             .build();
 

See more configuration options in DynamicKafkaSourceBuilder and DynamicKafkaSourceOptions.

See Also:

Serialized Form

Method Summary

Modifier and Type	Method and Description
static <T> DynamicKafkaSourceBuilder<T>	builder() Get a builder for this source.
org.apache.flink.api.connector.source.SplitEnumerator<DynamicKafkaSourceSplit,DynamicKafkaSourceEnumState>	createEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<DynamicKafkaSourceSplit> enumContext) Create the DynamicKafkaSourceEnumerator.
org.apache.flink.api.connector.source.SourceReader<T,DynamicKafkaSourceSplit>	createReader(org.apache.flink.api.connector.source.SourceReaderContext readerContext) Create the DynamicKafkaSourceReader.
org.apache.flink.api.connector.source.Boundedness	getBoundedness() Get the Boundedness.
org.apache.flink.core.io.SimpleVersionedSerializer<DynamicKafkaSourceEnumState>	getEnumeratorCheckpointSerializer() Get the DynamicKafkaSourceEnumStateSerializer.
KafkaStreamSubscriber	getKafkaStreamSubscriber()
org.apache.flink.api.common.typeinfo.TypeInformation<T>	getProducedType() Get the TypeInformation of the source.
org.apache.flink.core.io.SimpleVersionedSerializer<DynamicKafkaSourceSplit>	getSplitSerializer() Get the DynamicKafkaSourceSplitSerializer.
org.apache.flink.api.connector.source.SplitEnumerator<DynamicKafkaSourceSplit,DynamicKafkaSourceEnumState>	restoreEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<DynamicKafkaSourceSplit> enumContext, DynamicKafkaSourceEnumState checkpoint) Restore the DynamicKafkaSourceEnumerator.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

builder

public static <T> DynamicKafkaSourceBuilder<T> builder()

Get a builder for this source.

Returns:

a DynamicKafkaSourceBuilder.

getBoundedness

public org.apache.flink.api.connector.source.Boundedness getBoundedness()

Get the Boundedness.

Specified by:

getBoundedness in interface org.apache.flink.api.connector.source.Source<T,DynamicKafkaSourceSplit,DynamicKafkaSourceEnumState>

Returns:

the Boundedness.

createReader

@Internal
public org.apache.flink.api.connector.source.SourceReader<T,DynamicKafkaSourceSplit> createReader(org.apache.flink.api.connector.source.SourceReaderContext readerContext)

Create the DynamicKafkaSourceReader.

Specified by:

createReader in interface org.apache.flink.api.connector.source.SourceReaderFactory<T,DynamicKafkaSourceSplit>

Parameters:

readerContext - The context for the source reader.

Returns:

the DynamicKafkaSourceReader.

createEnumerator

@Internal
public org.apache.flink.api.connector.source.SplitEnumerator<DynamicKafkaSourceSplit,DynamicKafkaSourceEnumState> createEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<DynamicKafkaSourceSplit> enumContext)

Create the DynamicKafkaSourceEnumerator.

Specified by:

createEnumerator in interface org.apache.flink.api.connector.source.Source<T,DynamicKafkaSourceSplit,DynamicKafkaSourceEnumState>

Parameters:

enumContext - The context for the split enumerator.

Returns:

the DynamicKafkaSourceEnumerator.

restoreEnumerator

@Internal
public org.apache.flink.api.connector.source.SplitEnumerator<DynamicKafkaSourceSplit,DynamicKafkaSourceEnumState> restoreEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<DynamicKafkaSourceSplit> enumContext,
                                                                                                                                              DynamicKafkaSourceEnumState checkpoint)

Restore the DynamicKafkaSourceEnumerator.

Specified by:

restoreEnumerator in interface org.apache.flink.api.connector.source.Source<T,DynamicKafkaSourceSplit,DynamicKafkaSourceEnumState>

Parameters:

enumContext - The context for the restored split enumerator.

checkpoint - The checkpoint to restore the SplitEnumerator from.

Returns:

the DynamicKafkaSourceEnumerator.

getSplitSerializer

@Internal
public org.apache.flink.core.io.SimpleVersionedSerializer<DynamicKafkaSourceSplit> getSplitSerializer()

Get the DynamicKafkaSourceSplitSerializer.

Specified by:

getSplitSerializer in interface org.apache.flink.api.connector.source.Source<T,DynamicKafkaSourceSplit,DynamicKafkaSourceEnumState>

Returns:

the DynamicKafkaSourceSplitSerializer.

getEnumeratorCheckpointSerializer

@Internal
public org.apache.flink.core.io.SimpleVersionedSerializer<DynamicKafkaSourceEnumState> getEnumeratorCheckpointSerializer()

Get the DynamicKafkaSourceEnumStateSerializer.

Specified by:

getEnumeratorCheckpointSerializer in interface org.apache.flink.api.connector.source.Source<T,DynamicKafkaSourceSplit,DynamicKafkaSourceEnumState>

Returns:

the DynamicKafkaSourceEnumStateSerializer.

getProducedType

public org.apache.flink.api.common.typeinfo.TypeInformation<T> getProducedType()

Get the TypeInformation of the source.

Specified by:

getProducedType in interface org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>

Returns:

the TypeInformation.

getKafkaStreamSubscriber

@VisibleForTesting
public KafkaStreamSubscriber getKafkaStreamSubscriber()