【问题标题】:apache pulsar : error while consuming messageapache pulsar:消费消息时出错
【发布时间】:2022-11-02 21:30:25
【问题描述】:

我正在尝试从脉冲星主题中消费。我的客户端代码如下所示。 但是,我面对的是NoClassDefFoundError。请让我知道如何解决这个问题。

pulsar 客户端版本:2.10,pulsar 服务器版本:2.10, 安装类型:独立(不是 docker),操作系统:Ubuntu 20.04.4 LTS

我的脉冲星客户端代码:

private static void consumeFromPulsarAsync() throws Exception {

    logger.info("consumeFromPulsarAsync()");

    PulsarClient client = PulsarClient.
        builder()
        .serviceUrl("pulsar://localhost:6650")
        .build();

    logger.info("consumeFromPulsarAsync() client");

    Consumer<String> consumer = client.newConsumer(Schema.STRING)
        .topic("persistent://public/default/ack-2")
        .consumerName("pulsar-consumer-id-" + Math.random())
        .subscriptionName("pulsar-subscription-id-" + Math.random())
        .subscriptionType(SubscriptionType.Shared).subscribe();

    logger.info("consumeFromPulsarAsync() consumer");

    consumer.receiveAsync().thenCompose((msg) -> {
      logger.info("consumeFromPulsarAsync() consumed msg :: {}", msg.getValue());
      try {
        consumer.acknowledge(msg);
      } catch (PulsarClientException e) {
        throw new RuntimeException(e);
      }
      return null;
    });
  }

我的 pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.company</groupId>
  <artifactId>my.work.manager</artifactId>
  <version>release-1.0</version>
  <description>my Worker Manager</description>

  <properties>
    <maven.compiler.source>1.8</maven.compiler.source>
    <maven.compiler.target>1.8</maven.compiler.target>
    <flink.version>1.14.3</flink.version>
    <pulsar.version>2.10.0</pulsar.version>
  </properties>

  <dependencies>
    <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-clients_2.12</artifactId>
      <version>${flink.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-connector-kafka_2.12</artifactId>
      <version>${flink.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-connector-pulsar</artifactId>
      <version>1.15.0</version>
    </dependency>
    <dependency>
      <groupId>com.google.code.gson</groupId>
      <artifactId>gson</artifactId>
      <version>2.8.9</version>
    </dependency>
    <dependency>
      <groupId>org.projectlombok</groupId>
      <artifactId>lombok</artifactId>
      <version>1.18.22</version>
      <scope>provided</scope>
    </dependency>
    <dependency>
      <groupId>org.json</groupId>
      <artifactId>json</artifactId>
      <version>20210307</version>
    </dependency>
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-api</artifactId>
      <version>1.7.32</version>
    </dependency>
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-simple</artifactId>
      <version>1.7.30</version>
    </dependency>
    <dependency>
      <groupId>org.yaml</groupId>
      <artifactId>snakeyaml</artifactId>
      <version>1.30</version>
    </dependency>
    <dependency>
      <groupId>org.apache.httpcomponents</groupId>
      <artifactId>httpclient</artifactId>
      <version>4.5.13</version>
    </dependency>
    <dependency>
      <groupId>org.apache.pulsar</groupId>
      <artifactId>pulsar-client</artifactId>
      <version>${pulsar.version}</version>
    </dependency>
  </dependencies>

  <build>
    <plugins>
      <plugin>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.8.1</version>
        <configuration>
          <source>1.8</source>
          <target>1.8</target>
          <compilerArgs>
            <arg>-Xlint:all,-options,-path</arg>
          </compilerArgs>
        </configuration>
      </plugin>

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-jar-plugin</artifactId>
        <version>2.5</version>
        <configuration>
          <archive>
            <manifest>
              <mainClass>com.company.my.manager.flink.myWorkManager</mainClass>
            </manifest>
          </archive>
        </configuration>
      </plugin>

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-assembly-plugin</artifactId>
        <version>3.3.0</version>
        <configuration>
          <descriptorRefs>
            <descriptorRef>jar-with-dependencies</descriptorRef>
          </descriptorRefs>
        </configuration>
        <executions>
          <execution>
            <id>make-assembly</id>
            <phase>package</phase>
            <goals>
              <goal>single</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
    <resources>
      <resource>
        <directory>src/main/resources</directory>
        <targetPath>${project.build.outputDirectory}</targetPath>
        <includes>
          <include>**/*.yml</include>
        </includes>
      </resource>
    </resources>
  </build>

</project>

它显示的错误:

2022-05-10 17:52:52,549 警告 org.apache.pulsar.client.impl.MultiTopicsConsumerImpl [] - 多主题的分区自动更新定时器任务遇到错误 消费者。将安排另一个任务。 java.lang.NoClassDefFoundError: org/apache/pulsar/shade/com/google/common/primitives/Ints 在 org.apache.pulsar.shade.com.google.common.collect.Lists.computeArrayListCapacity(Lists.java:152) ~[e8fe35f4-c20c-4c3e-9b81-a3ff1a0ebaec_my.work.manager-release-1.0-jar-with-dependencies.jar:?] 在 org.apache.pulsar.shade.com.google.common.collect.Lists.newArrayListWithExpectedSize(Lists.java:192) ~[e8fe35f4-c20c-4c3e-9b81-a3ff1a0ebaec_my.work.manager-release-1.0-jar-with-dependencies.jar:?] 在 org.apache.pulsar.client.impl.MultiTopicsConsumerImpl$TopicsPartitionChangedListener.onTopicsExtended(MultiTopicsConsumerImpl.java:1238) ~[e8fe35f4-c20c-4c3e-9b81-a3ff1a0ebaec_my.work.manager-release-1.0-jar-with-dependencies.jar:?] 在 org.apache.pulsar.client.impl.MultiTopicsConsumerImpl$1.run(MultiTopicsConsumerImpl.java:1350) [e8fe35f4-c20c-4c3e-9b81-a3ff1a0ebaec_my.work.manager-release-1.0-jar-with-dependencies.jar:?] 在 org.apache.pulsar.shade.io.netty.util.HashedWheelTimer$HashedWheelTimeout.run(HashedWheelTimer.java:715) [e8fe35f4-c20c-4c3e-9b81-a3ff1a0ebaec_my.work.manager-release-1.0-jar-with-dependencies.jar:?] 在 org.apache.pulsar.shade.io.netty.util.concurrent.ImmediateExecutor.execute(ImmediateExecutor.java:34) [e8fe35f4-c20c-4c3e-9b81-a3ff1a0ebaec_my.work.manager-release-1.0-jar-with-dependencies.jar:?] 在 org.apache.pulsar.shade.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:703) [e8fe35f4-c20c-4c3e-9b81-a3ff1a0ebaec_my.work.manager-release-1.0-jar-with-dependencies.jar:?] 在 org.apache.pulsar.shade.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:790) [e8fe35f4-c20c-4c3e-9b81-a3ff1a0ebaec_my.work.manager-release-1.0-jar-with-dependencies.jar:?] 在 org.apache.pulsar.shade.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:503) [e8fe35f4-c20c-4c3e-9b81-a3ff1a0ebaec_my.work.manager-release-1.0-jar-with-dependencies.jar:?] 在 org.apache.pulsar.shade.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [e8fe35f4-c20c-4c3e-9b81-a3ff1a0ebaec_my.work.manager-release-1.0-jar-with-dependencies.jar:?] 在 java.lang.Thread.run(Thread.java:829) [?:?]

【问题讨论】:

    标签: apache-pulsar


    【解决方案1】:

    什么类型的集群?码头工人,独立,K8,... Pulsar 客户端和服务器都是 2.10 吗? 服务器用什么JDK?为客户? 服务器有错误吗? 什么服务器操作系统? 什么客户端操作系统?

    它说 WARN,它是在缺少的类中运行还是崩溃?有时在 MAC 上,您会收到关于缺少 SSL 的警告,并且它会与警告一起使用。

    可能和这个https://github.com/apache/pulsar/issues/9585有关

    调试步骤:

    创建一个新主题。 再试一次。 尝试使用命令行客户端进行生产和消费。

    创建一个单独的分区主题并使用它进行测试。

    我用这个测试运行了你的例子

    bin/pulsar-client produce --key "test1" "persistent://public/default/ack-2" -m "Test this thing 4" -n 25
    

    --

    public void consumeFromPulsarAsync() throws Exception {
    
            PulsarClient client = PulsarClient.
                    builder()
                    .serviceUrl("pulsar://localhost:6650")
                    .build();
    
            Consumer<String> consumer = client.newConsumer(Schema.STRING)
                    .topic("persistent://public/default/ack-2")
                    .consumerName("pulsar-consumer-id-" + Math.random())
                    .subscriptionName("pulsar-subscription-id-" + Math.random())
                    .subscriptionType(SubscriptionType.Shared).subscribe();
            
            consumer.receiveAsync().thenCompose((msg) -> {
                log.info("consumeFromPulsarAsync() consumed msg :: {}", msg.getValue());
                try {
                    consumer.acknowledge(msg);
                } catch (PulsarClientException e) {
                    e.printStackTrace();
                    throw new RuntimeException(e);
                }
                return null;
            });
        }
    

    --

    2022-05-11 10:00:47.321  INFO 28575 --- [r-client-io-6-1] o.a.pulsar.client.impl.ConnectionPool    : [[id: 0x2005ff4b, L:/192.168.1.63:49386 - R:pulsar1.fios-router.home/192.168.1.230:6650]] Connected to server
    2022-05-11 10:00:47.321  INFO 28575 --- [r-client-io-6-1] org.apache.pulsar.client.impl.ClientCnx  : [id: 0x2005ff4b, L:/192.168.1.63:49386 - R:pulsar1.fios-router.home/192.168.1.230:6650] Connected through proxy to target broker at 127.0.0.1:6650
    2022-05-11 10:00:47.323  INFO 28575 --- [r-client-io-7-1] o.a.pulsar.client.impl.ConsumerImpl      : [persistent://public/default/ack-2-partition-2][pulsar-subscription-id-0.6345962322446594] Subscribing to topic on cnx [id: 0xf5af925d, L:/192.168.1.63:49385 - R:pulsar1.fios-router.home/192.168.1.230:6650], consumerId 2
    2022-05-11 10:00:47.324  INFO 28575 --- [r-client-io-6-1] o.a.pulsar.client.impl.ConsumerImpl      : [persistent://public/default/ack-2-partition-2][pulsar-subscription-id-0.6808890118894377] Subscribing to topic on cnx [id: 0x2005ff4b, L:/192.168.1.63:49386 - R:pulsar1.fios-router.home/192.168.1.230:6650], consumerId 2
    2022-05-11 10:00:47.324  INFO 28575 --- [r-client-io-7-1] o.a.pulsar.client.impl.ConsumerImpl      : [persistent://public/default/ack-2-partition-1][pulsar-subscription-id-0.6345962322446594] Subscribing to topic on cnx [id: 0xf5af925d, L:/192.168.1.63:49385 - R:pulsar1.fios-router.home/192.168.1.230:6650], consumerId 1
    2022-05-11 10:00:47.324  INFO 28575 --- [r-client-io-6-1] o.a.pulsar.client.impl.ConsumerImpl      : [persistent://public/default/ack-2-partition-1][pulsar-subscription-id-0.6808890118894377] Subscribing to topic on cnx [id: 0x2005ff4b, L:/192.168.1.63:49386 - R:pulsar1.fios-router.home/192.168.1.230:6650], consumerId 1
    2022-05-11 10:00:47.324  INFO 28575 --- [r-client-io-7-1] o.a.pulsar.client.impl.ConsumerImpl      : [persistent://public/default/ack-2-partition-0][pulsar-subscription-id-0.6345962322446594] Subscribing to topic on cnx [id: 0xf5af925d, L:/192.168.1.63:49385 - R:pulsar1.fios-router.home/192.168.1.230:6650], consumerId 0
    2022-05-11 10:00:47.324  INFO 28575 --- [r-client-io-6-1] o.a.pulsar.client.impl.ConsumerImpl      : [persistent://public/default/ack-2-partition-0][pulsar-subscription-id-0.6808890118894377] Subscribing to topic on cnx [id: 0x2005ff4b, L:/192.168.1.63:49386 - R:pulsar1.fios-router.home/192.168.1.230:6650], consumerId 0
    2022-05-11 10:00:47.357  INFO 28575 --- [r-client-io-6-1] o.a.pulsar.client.impl.ConsumerImpl      : [persistent://public/default/ack-2-partition-0][pulsar-subscription-id-0.6808890118894377] Subscribed to topic on pulsar1.fios-router.home/192.168.1.230:6650 -- consumer: 0
    2022-05-11 10:00:47.370  INFO 28575 --- [r-client-io-6-1] o.a.pulsar.client.impl.ConsumerImpl      : [persistent://public/default/ack-2-partition-1][pulsar-subscription-id-0.6808890118894377] Subscribed to topic on pulsar1.fios-router.home/192.168.1.230:6650 -- consumer: 1
    2022-05-11 10:00:47.370  INFO 28575 --- [r-client-io-7-1] o.a.pulsar.client.impl.ConsumerImpl      : [persistent://public/default/ack-2-partition-1][pulsar-subscription-id-0.6345962322446594] Subscribed to topic on pulsar1.fios-router.home/192.168.1.230:6650 -- consumer: 1
    2022-05-11 10:00:47.371  INFO 28575 --- [r-client-io-6-1] o.a.pulsar.client.impl.ConsumerImpl      : [persistent://public/default/ack-2-partition-2][pulsar-subscription-id-0.6808890118894377] Subscribed to topic on pulsar1.fios-router.home/192.168.1.230:6650 -- consumer: 2
    2022-05-11 10:00:47.372  INFO 28575 --- [r-client-io-6-1] o.a.p.c.impl.MultiTopicsConsumerImpl     : [persistent://public/default/ack-2] [pulsar-subscription-id-0.6808890118894377] Success subscribe new topic persistent://public/default/ack-2 in topics consumer, partitions: 3, allTopicPartitionsNumber: 3
    2022-05-11 10:00:47.394  INFO 28575 --- [r-client-io-7-1] o.a.pulsar.client.impl.ConsumerImpl      : [persistent://public/default/ack-2-partition-0][pulsar-subscription-id-0.6345962322446594] Subscribed to topic on pulsar1.fios-router.home/192.168.1.230:6650 -- consumer: 0
    2022-05-11 10:00:47.394  INFO 28575 --- [r-client-io-7-1] o.a.pulsar.client.impl.ConsumerImpl      : [persistent://public/default/ack-2-partition-2][pulsar-subscription-id-0.6345962322446594] Subscribed to topic on pulsar1.fios-router.home/192.168.1.230:6650 -- consumer: 2
    2022-05-11 10:00:47.395  INFO 28575 --- [r-client-io-7-1] o.a.p.c.impl.MultiTopicsConsumerImpl     : [persistent://public/default/ack-2] [pulsar-subscription-id-0.6345962322446594] Success subscribe new topic persistent://public/default/ack-2 in topics consumer, partitions: 3, allTopicPartitionsNumber: 3
    2022-05-11 10:00:52.529  INFO 28575 --- [t-internal-11-1] a.f.a.consumer.AirQualityConsumerApp     : consumeFromPulsarAsync() consumed msg :: Test this thing 4
    2022-05-11 10:00:52.529  INFO 28575 --- [nt-internal-9-1] a.f.a.consumer.AirQualityConsumerApp     : consumeFromPulsarAsync() consumed msg :: Test this thing 4
    ^C2022-05-11 10:00:54.823  INFO 28575 --- [r-client-io-1-1] o.a.pulsar.client.impl.ConsumerImpl      : [persistent://public/default/airquality] [airqualitysj1] Closed consumer
    2022-05-11 10:00:54.824  INFO 28575 --- [ionShutdownHook] o.a.pulsar.client.impl.PulsarClientImpl  : Client closing. URL: pulsar://pulsar1:6650
    2022-05-11 10:00:54.826  INFO 28575 --- [r-client-io-1-1] org.apache.pulsar.client.impl.ClientCnx  : [id: 0x4d247f3d, L:/192.168.1.63:49381 ! R:pulsar1.fios-router.home/192.168.1.230:6650] Disconnected
    2022-05-11 10:00:54.829  INFO 28575 --- [r-client-io-1-1] org.apache.pulsar.client.impl.ClientCnx  : [id: 0x5fa81a8b, L:/192.168.1.63:49382 ! R:pulsar1.fios-router.home/192.168.1.230:6650] Disconnected
    

    【讨论】:

    • 是的,客户端和服务器都是 2.10。操作系统是 Ubuntu 20.04.4 它没有运行。它崩溃了。我创建了新主题,为其设置架构。但同样的错误再次发生。
    • 你能分享一下架构吗?这很奇怪。让我测试一下。
    • 我运行了利用它发送消息的代码。也许您需要使用域名或 IP 地址而不是 localhost?我使用了 JDK 11
    • 您可以发布您的 POM.XML 或 build.gradle。
    • 我已经在问题中发布(添加)了我的 pom.xml
    猜你喜欢
    • 2020-08-14
    • 2020-09-22
    • 2022-07-14
    • 2019-11-19
    • 2020-03-12
    • 2022-01-06
    • 1970-01-01
    • 1970-01-01
    • 2014-02-13
    相关资源
    最近更新 更多