GKE 升级后，Google Kubernetes 部署上的 DSpace jspui 失败答案

【问题标题】：DSpace jspui on Google Kubernetes deployment fails after GKE upgradeGKE 升级后，Google Kubernetes 部署上的 DSpace jspui 失败
【发布时间】：2023-03-10 20:58:01
【问题描述】：

我们已将 DSpace 6.3 部署到 Google Kubernetes Engine (GKE)，它一直运行良好。但是，当我们将 GKE 从 v1.12.7-gke.24 升级到 1.14.10-gke.50 时，容器突然出现故障。对 k8s 版本的更改是工作和失败的 k8s 节点之间的唯一区别。本地构建的 docker 容器工作正常。我们正在将其他 DSpace 模块部署在单独的容器（例如 solr）中，它们工作正常，只有 jspui 模块出现故障。

DSpace 分支“dspace-6_x”标签“dspace-6.3”

Docker 镜像：tomcat:8-alpine

通过 gitlab CI/CD 管道部署

失败是由于Spring Loader在调用各种DSpace工厂服务单例模式bean的早期加载时摔倒造成的。这会在加载网站时导致 404 错误，因为 Web 应用无法初始化。

/usr/local/tomcat/log/localhost.YYYY-MM-dd.log 中的错误消息：

28-Oct-2020 23:47:18.668 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart 
Exception sending context initialized event to listener instance of class 
[org.dspace.servicemanager.servlet.DSpaceKernelServletContextListener]
        java.lang.RuntimeException: Failure during filter init: Failed to startup the DSpace Service 
Manager: failure starting up spring service manager: Error creating bean with name 
'org.dspace.app.sherpa.submit.SHERPASubmitService' defined in URL 
[jar:file:/dspace/webapps/jspui/WEB-INF/lib/dspace-api-6.3.jar!/spring/spring-dspace-addon-sherpa-services.xml]: 
Cannot resolve reference to bean 'org.dspace.app.sherpa.submit.SHERPASubmitConfigurationService' while setting 
bean property'configuration'; nested exception is org.springframework.beans.factory.BeanCreationException: 
Error creating bean with name 'org.dspace.app.sherpa.submit.SHERPASubmitConfigurationService' defined in 
file [/dspace/config/spring/api/sherpa.xml]: Cannot create inner bean 
'org.dspace.app.sherpa.submit.MetadataValueISSNExtractor#1b511285' of type 
[org.dspace.app.sherpa.submit.MetadataValueISSNExtractor] while setting bean property 
'issnItemExtractors' with key [0]; nestedexception is 
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 
'org.dspace.app.sherpa.submit.MetadataValueISSNExtractor#1b511285': Injection of autowired dependencies 
failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire 
field: public org.dspace.content.service.ItemService 
org.dspace.app.sherpa.submit.MetadataValueISSNExtractor.itemService; nested exception is 
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 
'org.dspace.content.ItemServiceImpl#0': Injection of autowireddependencies failed; nested exception is
 org.springframework.beans.factory.BeanCreationException: Could not autowire field: protected 
org.dspace.handle.service.HandleService org.dspace.content.DSpaceObjectServiceImpl.handleService; ...

“启动 spring 服务管理器失败”错误消息出现在：

org.dspace.servicemanager.DSpaceServiceManager （\dspace-services\src\main\java\org\dspace\servicemanager\DSpaceServiceManager.java 第 215 行）

在第 212 行的 catch 语句中调用：

org.dspace.servicemanager.spring.SpringServiceManager.startup() （\dspace-services\src\main\java\org\dspace\servicemanager\spring\SpringServiceManager.java 第 177 行）

它使用 Spring 框架来提前加载工厂 bean。

我们的第一个想法是新的 k8s 版本可能需要更多的内存。所以我们将 Tomcat 内存从 1.5GB 增加到 4GB。这并没有解决问题。

我们研究了升级之间的 GKE 中间版本的发行说明，但没有发现任何帮助。

我们尝试过使用其他 Tomcat docker 镜像，但无济于事。因此，我们认为这不是操作系统的问题。

远程调试连接到 Tomcat 的速度不够快，无法捕获异常。我们尝试了适用于 Java 的 Google Cloud Debugger，但 Alpine Linux 缺少一些必需的库。无论如何，我不相信我们会发现比正在记录的错误消息更有帮助的东西。

如果有人有任何想法，我们将不胜感激。

我们的生产 k8s 配置 yaml 文件：

ingress:
  hosts:
    - our.url.uts.edu.au

database:
  secret: our_password
  name: our_db_name
  host: "our.db.instance.url"
  port: "5432"

dspace:
  env:
    - name: DSPACE_HOSTNAME
      value: our.url.uts.edu.au
    - name: SOLR_PORT
      value: "8080"
      # Include colon if port is specified
    - name: DSPACE_PORT
      value: ""
    - name: MAX_DB_CONNECTIONS
      value: "50"
    - name: "MAX_IDLE_DB_CONNECTIONS"
      value: "30"
    - name: INITIAL_DB_CONNECTIONS
      value: "20"
    - name: S3_ASSETSTORE_SUBFOLDER
      value: "our_folder"
    - name: S3_CONNECTION_TTL
      value: "120000"
    - name: S3_MAX_CONNECTIONS
      value: "50"
    - name: REST_EVENT_WEBHOOK_URL
      value: http://our.rest.service.url/dspace/v2/webhook
    - name: UTSLIB_FRAMEWORK_DSPACE_TOKEN
      value: OUR_TOKEN
    - name: CATALINA_OPTS
      value: "-Xms1512m -Xmx1512m"

  resources:
    requests:
      memory: "1640Mi"
      cpu: 100m
    limits:
      memory: "1896Mi"
      cpu: "450m"

solr:
  pvc:
    accessModes:
      - ReadWriteOnce
    annotations: {}
    size: 35Gi

  env:
    - name: CATALINA_OPTS
      value: "-Xms3904m -Xmx3904m -XX:+UseG1GC"

  resources:
    requests:
      memory: "4032Mi"
      cpu: 50m
    limits:
      memory: "4096Mi"
      cpu: "800m"

cron:
  env:
    - name: SOLR_PORT
      value: "8080"
    - name: MAX_DB_CONNECTIONS
      value: "3"
    - name: MAX_IDLE_DB_CONNECTIONS
      value: "1"
    - name: INITIAL_DB_CONNECTIONS
      value: "0"
    - name: S3_ASSETSTORE_SUBFOLDER
      value: "our_folder"
    - name: S3_CONNECTION_TTL
      value: "120000"
    - name: S3_MAX_CONNECTIONS
      value: "50"
    - name: JAVA_OPTS
      value: "-Xms32m -Xmx384m"
    - name: REST_EVENT_WEBHOOK_URL
      value: http://our.rest.service.url/dspace/v2/webhook
    - name: UTSLIB_FRAMEWORK_DSPACE_TOKEN
      value: OUR_TOKEN

我们的 Dockerfile 分为构建和运行时进程。 Dockerfile.build

FROM maven:3-jdk-8

# Modules that should be excluded from depdendency resolution
ARG EXCLUDE_MODULES=!dspace-rdf,!dspace-sword,!dspace-xmlui,!dspace-xmlui-mirage2

ENV DSPACE_VERSION=6.3 \
    DSPACE_SHA1=e60db8dee2726933fcc7b7949c16757a510a79c5

ENV ANT_VERSION=1.10.8
ENV ANT_HOME=/opt/ant-$ANT_VERSION
ENV PATH=$ANT_HOME/bin:$PATH \
    ANT_SHA1=20658b765bed8a7c3d18daa71a108e15d1937da2

WORKDIR /dspace-src

# Download DSpace source and install Ant
RUN curl -fSL "https://github.com/DSpace/DSpace/releases/download/dspace-${DSPACE_VERSION}/dspace-${DSPACE_VERSION}-src-release.tar.gz" -o dspace.tar.gz && \
    echo "${DSPACE_SHA1} *dspace.tar.gz" | sha1sum -c - && \
    tar -xz -f dspace.tar.gz --strip-components=1 && \
    rm -f dspace.tar.gz && \
    curl -fSL "https://archive.apache.org/dist/ant/binaries/apache-ant-${ANT_VERSION}-bin.tar.gz" -o ant.tar.gz && \
    echo "${ANT_SHA1} *ant.tar.gz" | sha1sum -c - && \
    mkdir ${ANT_HOME} && \
    tar -xz -f ant.tar.gz -C ${ANT_HOME} --strip-components=1 && \
    rm -rf ant.tar.gz

# Copy in custom artifacts
COPY ./src/artifacts/ ./artifacts

# Copy in pom.xml files
COPY ./src/dspace/pom.xml                          ./dspace/
COPY ./src/dspace/modules/pom.xml                  ./dspace/modules/
COPY ./src/dspace/modules/jspui/pom.xml            ./dspace/modules/jspui/
COPY ./src/dspace/modules/utslib-copyright/pom.xml ./dspace/modules/utslib-copyright/
COPY ./src/dspace/modules/utslib-taglib/pom.xml    ./dspace/modules/utslib-taglib/

# Install custom artifacts and prime the Maven repository 
RUN mvn clean install --batch-mode --fail-never -f ./artifacts/JRis-master && \
    mvn install -P ${EXCLUDE_MODULES} --batch-mode --fail-never -T 5

Dockerfile.runtime：

ARG BUILD_IMAGE=our.git.url/dspace/build:latest

FROM ${BUILD_IMAGE} as build

# Copy in our source changes
COPY ./src/dspace ./dspace

# We don't use these modules, but they'll be built anyway if not excluded
ARG EXCLUDE_MODULES=!dspace-rdf,!dspace-xmlui,!dspace-sword

# Unzip the MaxMind GeoLite database (IP location stuff for Solr).
# (MaxMind changed their privacy policy so you now have to login to download,
# which makes it fail for the standard DSpace installation)
# Build dspace with our source changes and move it to the installation directory
# Build only our customisations (skip building the specified modules)
# Could multithread the maven build, but there's dependency resolution problems
RUN tar -zxf ./dspace/config/GeoLite2-City_20191224.tar.gz --strip-components=1 -C ./dspace/config && \
    rm ./dspace/config/GeoLite2-City_20191224.tar.gz && \
    mvn package --batch-mode -P ${EXCLUDE_MODULES} -f ./dspace/pom.xml && \
    cd ./dspace/target/dspace-installer && \
    ant copy_webapps install_code

FROM tomcat:8-alpine
#FROM tomcat:8-jre8

ARG DSPACE_INSTALL_DIR=/dspace

ENV DSPACE_HOME=${DSPACE_INSTALL_DIR}

# Copy built source into this image
COPY --from=build ${DSPACE_INSTALL_DIR} ${DSPACE_INSTALL_DIR}

# Copy in our config overrides
# (These are not used in compilation, but are applied at runtime)
COPY ./src/local.cfg ${DSPACE_INSTALL_DIR}/config/

# Symlink all webapps and create temp upload directory
RUN ln -s ${DSPACE_INSTALL_DIR}/webapps/* ./webapps/

【问题讨论】：

是否可以将此集群升级到当前默认版本的 GKE (1.16.13-gke.401)？您能否提供重现此问题的配置文件和步骤？
感谢您回复我们。我们不愿进一步升级，因为您一次只能升级一个主要版本 - 所以我们将执行两次升级，这非常耗时。此外，我们还有其他几个应用程序在该集群的节点中运行，我们认为破坏它们的风险太大，并且我们使用的一些东西已被弃用，这涉及重新调整我们的 gitlab CI/CD 部署过程的大量工作。我将在下面发布配置文件。
将配置文件添加到原始问题。

标签： spring-boot tomcat google-kubernetes-engine dspace

【解决方案1】：

在 DSpace 和 Tomcat 中实现最详细的日志记录级别后，可以获得有关 Spring 错误来源的更多信息。

问题在于我们的自定义工厂类之一。错误日志提取：

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 
'org.dspace.storage.bitstore.BitstreamStorageService' defined in file 
[/dspace/config/spring/api/bitstore.xml]: Cannot resolve reference to bean 's3Store' while 
setting bean property 'stores' with key [TypedStringValue: value [0], target type [null]]; 
nested exception is org.springframework.beans.factory.BeanCreationException: Error 
creating bean with name 's3Store' defined in file 
[/dspace/config/spring/api/bitstore.xml]: Error setting property values; nested exception 
is org.springframework.beans.NotWritablePropertyException: Invalid property 
's3ConnectionTTL' of bean class [org.dspace.storage.bitstore.S3BitStoreService]: Bean 
property 's3ConnectionTTL' is not writable or has an invalid setter method. Does the 
parameter type of the setter match the return type of the getter?

有问题的属性是可写的，具有有效的 getter 和 setter，并且 getter 和 setter 都是长类型。我删除了属性集代码，让它保持默认。部署工作。

简单地升级 k8s 版本会导致这个错误是我们无法理解的。在具有先前 GKE 版本的 pod 中，完全相同的代码可以正常执行。

【讨论】：