StreamSets 处理器找不到 Redis 库

Question

我有两个带有 Redis 和 StreamSets 的容器。我想用 Java 编写自定义处理器并放入管道。但是当我将代码从教程添加到处理器时，将 jar 发送到 lib 并尝试启动时出现异常。请问你能帮帮我吗？

处理器教程 - https://github.com/streamsets/tutorials/blob/master/tutorial-processor/readme.md

Redis 教程-https://dzone.com/articles/quickstart-how-to-use-redis-on-java

  redis:
    image: redis
    ports:
      - 6379:6379
    command: redis-server


  streamsets:
    build:
      context: "."
      args:
        SDC_LIBS: streamsets-datacollector-apache-kafka_2_0-lib                ports:
      - 18630:18630
    volumes:
      - ./user-libs:/opt/streamsets-datacollector-3.9.0/user-libs

在样本处理器中启动：

  @Override
  protected List<ConfigIssue> init() {
      ....
      Config config = new Config();
      config.useSingleServer().setAddress("redis://redis:6379");
      redisson = Redisson.create(config);
      ... }



  <dependency>
      <groupId>org.redisson</groupId>
      <artifactId>redisson</artifactId>
      <version>3.10.4</version>
      <scope>provided</scope>
  </dependency>

处理器代码测试成功。

日志

streamsets_1  | 2020-03-21 14:00:19,954 [user:] [pipeline:] [runner:] [thread:main] [stage:] ERROR Main - Abnormal exit: java.lang.NoClassDefFoundError: Lorg/redisson/api/RedissonClient;

streamsets_1  | java.lang.NoClassDefFoundError: Lorg/redisson/api/RedissonClient;
streamsets_1  |         at java.lang.Class.getDeclaredFields0(Native Method)
streamsets_1  |         at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
streamsets_1  |         at java.lang.Class.privateGetPublicFields(Class.java:2614)
streamsets_1  |         at java.lang.Class.privateGetPublicFields(Class.java:2629)
streamsets_1  |         at java.lang.Class.getFields(Class.java:1557)
streamsets_1  |         at com.streamsets.datacollector.definition.ConfigDefinitionExtractor.validate(ConfigDefinitionExtractor.java:82)
streamsets_1  |         at com.streamsets.datacollector.definition.ConfigDefinitionExtractor.validate(ConfigDefinitionExtractor.java:65)
streamsets_1  |         at com.streamsets.datacollector.definition.StageDefinitionExtractor.validate(StageDefinitionExtractor.java:141)
streamsets_1  |         at com.streamsets.datacollector.definition.StageDefinitionExtractor.extract(StageDefinitionExtractor.java:197)
streamsets_1  |         at com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask.loadStages(ClassLoaderStageLibraryTask.java:557)
streamsets_1  |         at com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask.initTask(ClassLoaderStageLibraryTask.java:278)
streamsets_1  |         at com.streamsets.datacollector.task.AbstractTask.init(AbstractTask.java:62)
streamsets_1  |         at com.streamsets.datacollector.task.CompositeTask.initTask(CompositeTask.java:44)
streamsets_1  |         at com.streamsets.datacollector.task.AbstractTask.init(AbstractTask.java:62)
streamsets_1  |         at com.streamsets.datacollector.task.TaskWrapper.init(TaskWrapper.java:40)
streamsets_1  |         at com.streamsets.datacollector.main.Main.lambda$doMain$1(Main.java:123)
streamsets_1  |         at java.security.AccessController.doPrivileged(Native Method)
streamsets_1  |         at com.streamsets.datacollector.security.SecurityUtil.doAs(SecurityUtil.java:92)
streamsets_1  |         at com.streamsets.datacollector.main.Main.doMain(Main.java:158)
streamsets_1  |         at com.streamsets.datacollector.main.DataCollectorMain.main(DataCollectorMain.java:53)
streamsets_1  |         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
streamsets_1  |         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
streamsets_1  |         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
streamsets_1  |         at java.lang.reflect.Method.invoke(Method.java:498)
streamsets_1  |         at com.streamsets.pipeline.BootstrapMain.bootstrap(BootstrapMain.java:351)
streamsets_1  |         at com.streamsets.pipeline.BootstrapMain.main(BootstrapMain.java:110)
streamsets_1  | Caused by: java.lang.ClassNotFoundException: org.redisson.api.RedissonClient
streamsets_1  |         at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
streamsets_1  |         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
streamsets_1  |         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
streamsets_1  |         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
streamsets_1  |         at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:353)
streamsets_1  |         at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:316)
streamsets_1  |         at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:353)
streamsets_1  |         at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:316)
streamsets_1  |         ... 26 more
streamsets_1  |
streamsets_1  | Abnormal exit: java.lang.NoClassDefFoundError: Lorg/redisson/api/RedissonClient;Check STDERR for more details
streamsets_1  |
streamsets_1  | java.lang.NoClassDefFoundError: Lorg/redisson/api/RedissonClient;
streamsets_1  |         at java.lang.Class.getDeclaredFields0(Native Method)
streamsets_1  |         at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
streamsets_1  |         at java.lang.Class.privateGetPublicFields(Class.java:2614)
streamsets_1  |         at java.lang.Class.privateGetPublicFields(Class.java:2629)
streamsets_1  |         at java.lang.Class.getFields(Class.java:1557)
streamsets_1  |         at com.streamsets.datacollector.definition.ConfigDefinitionExtractor.validate(ConfigDefinitionExtractor.java:82)
streamsets_1  |         at com.streamsets.datacollector.definition.ConfigDefinitionExtractor.validate(ConfigDefinitionExtractor.java:65)
streamsets_1  |         at com.streamsets.datacollector.definition.StageDefinitionExtractor.validate(StageDefinitionExtractor.java:141)
streamsets_1  |         at com.streamsets.datacollector.definition.StageDefinitionExtractor.extract(StageDefinitionExtractor.java:197)
streamsets_1  |         at com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask.loadStages(ClassLoaderStageLibraryTask.java:557)
streamsets_1  |         at com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask.initTask(ClassLoaderStageLibraryTask.java:278)
streamsets_1  |         at com.streamsets.datacollector.task.AbstractTask.init(AbstractTask.java:62)
streamsets_1  |         at com.streamsets.datacollector.task.CompositeTask.initTask(CompositeTask.java:44)
streamsets_1  |         at com.streamsets.datacollector.task.AbstractTask.init(AbstractTask.java:62)
streamsets_1  |         at com.streamsets.datacollector.task.TaskWrapper.init(TaskWrapper.java:40)
streamsets_1  |         at com.streamsets.datacollector.main.Main.lambda$doMain$1(Main.java:123)
streamsets_1  |         at java.security.AccessController.doPrivileged(Native Method)
streamsets_1  |         at com.streamsets.datacollector.security.SecurityUtil.doAs(SecurityUtil.java:92)
streamsets_1  |         at com.streamsets.datacollector.main.Main.doMain(Main.java:158)
streamsets_1  |         at com.streamsets.datacollector.main.DataCollectorMain.main(DataCollectorMain.java:53)
streamsets_1  |         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
streamsets_1  |         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
streamsets_1  |         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
streamsets_1  |         at java.lang.reflect.Method.invoke(Method.java:498)
streamsets_1  |         at com.streamsets.pipeline.BootstrapMain.bootstrap(BootstrapMain.java:351)
streamsets_1  |         at com.streamsets.pipeline.BootstrapMain.main(BootstrapMain.java:110)
streamsets_1  | Caused by: java.lang.ClassNotFoundException: org.redisson.api.RedissonClient
streamsets_1  |         at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
streamsets_1  |         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
streamsets_1  |         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
streamsets_1  |         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
streamsets_1  |         at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:353)
streamsets_1  |         at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:316)
streamsets_1  |         at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:353)
streamsets_1  |         at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:316)
streamsets_1  |         ... 26 more

Answer 1

问题可能出在您的dependency scope 上。您已指定provided，其定义为：

这很像compile，但表示您希望 JDK 或容器在运行时提供依赖项。例如，在为 Java 企业版构建 Web 应用程序时，您可以将 Servlet API 和相关 Java EE API 的依赖设置为范围 provided，因为 Web 容器提供了这些类。此范围仅在编译和测试类路径上可用，并且不可传递。

您应该指定compile 范围。