【问题标题】:Flink: java.lang.UnsupportedOperationException: Cannot override partitioning for KeyedStreamFlink:java.lang.UnsupportedOperationException:无法覆盖 KeyedStream 的分区
【发布时间】:2020-02-12 00:55:00
【问题描述】:

我在运行我的小型 flink 程序时遇到以下异常。该应用程序有两个来自同一模拟源的数据流。它具有广播状态。我写这个是为了做一些性能测试,但给了我例外

Caused by: java.lang.UnsupportedOperationException: Cannot override partitioning for KeyedStream.
    at org.apache.flink.streaming.api.datastream.KeyedStream.setConnectionType(KeyedStream.java:251)
    at org.apache.flink.streaming.api.datastream.DataStream.broadcast(DataStream.java:429)
    at org.apache.flink.streaming.api.scala.DataStream.broadcast(DataStream.scala:495)

我的代码:

val testStream: DataStream[Tuple2[String, String]] = env
    .addSource(
    new MockKafkaSource
    )
    .filter(x => !x._1.equals("x"))
    .map(x => x)
    .uid("test stream 1")

val testStream2: DataStream[Tuple2[String, String]] = env
    .addSource(
    new MockKafkaSource
    )
    .map(x => x)
    .keyBy(x => x._1)
    .uid("test stream 2")

lazy val testStateDescriptor =
    new MapStateDescriptor("testState", classOf[String], classOf[Tuple2[String, String]])

val testBroadcastStream = testStream.broadcast(testStateDescriptor)

val broadcastOutStream: DataStream[Tuple2[String, String]] =
    testStream2
    .connect(testBroadcastStream)
    .process(new StateProcess)

broadcastOutStream.print()

这一行发生的异常:

val testBroadcastStream = testStream.broadcast(testStateDescriptor)

【问题讨论】:

  • 我假设您的 StateProcess 函数是 KeyedBroadcastProcessFunction,而不是 BroadcastProcessFunction,对吧?
  • 是的,它扩展了 KeyedBroadcastProcessFunction

标签: scala apache-flink flink-streaming


【解决方案1】:

我的问题是我在testStream2 的键控流上调用了uid 方法。我不得不将uid 移动到地图之后,然后键入流。

【讨论】:

    猜你喜欢
    • 1970-01-01
    • 2018-11-15
    • 2018-11-22
    • 1970-01-01
    • 1970-01-01
    • 2020-05-31
    • 2016-06-21
    • 1970-01-01
    • 1970-01-01
    相关资源
    最近更新 更多