java.sql.SQLException：排序内存不足，请考虑增加服务器排序缓冲区大小

【问题标题】：java.sql.SQLException: Out of sort memory, consider increasing server sort buffer sizejava.sql.SQLException：排序内存不足，请考虑增加服务器排序缓冲区大小
【发布时间】：2020-12-25 19:22:35
【问题描述】：

我正在尝试使用带有 JDBC 的 java 运行此查询：

SELECT * FROM jobs WHERE 0=0 # find all occurrences of the search string + any component of the search string
        OR job_title COLLATE utf8mb4_0900_ai_ci LIKE CONCAT('%', 'software', '%')
        OR job_description COLLATE utf8mb4_0900_ai_ci LIKE CONCAT('%', 'software', '%')
        OR job_title COLLATE utf8mb4_0900_ai_ci LIKE CONCAT('%', 'engineer', '%')
        OR job_description COLLATE utf8mb4_0900_ai_ci LIKE CONCAT('%', 'engineer', '%')
        AND date_posted >= now() - INTERVAL 30 DAY # ensure results are from the last month
ORDER BY
        # order first by results that begin with the search string
        job_title COLLATE utf8mb4_0900_ai_ci LIKE CONCAT('software engineer', '%') DESC,
        # order second by results that contain the search string
        IFNULL(NULLIF(INSTR(job_title COLLATE utf8mb4_0900_ai_ci, 'software engineer'), 0), 99999) DESC,
        # order third by results that contain the first component of the search string
        IFNULL(NULLIF(INSTR(job_title COLLATE utf8mb4_0900_ai_ci, 'software'), 0), 99999) DESC,
        # order forth by results that contain the second component of the search string
        IFNULL(NULLIF(INSTR(job_title COLLATE utf8mb4_0900_ai_ci, 'engineer'), 0), 99999) DESC,
        # order alphabetically
        job_title,
        IFNULL(NULLIF(INSTR(job_description COLLATE utf8mb4_0900_ai_ci, 'software engineer'), 0), 99999);

当我在 MySQL 终端中运行此查询时，它运行良好。但是，当我通过 JDBC 运行它时，searchTokens = ['software', 'engineer']

var searchTokens = searchValue.split(" ");

String query = "SELECT * FROM jobs WHERE 0=0 \n";

for (String token : searchTokens) {
    query += String.format("    OR job_title COLLATE utf8mb4_0900_ai_ci LIKE CONCAT('%%', '%s', '%%')\n", token);
    query += String.format("    OR job_description COLLATE utf8mb4_0900_ai_ci LIKE CONCAT('%%', '%s', '%%')\n", token);
}

query += "    AND date_posted >= now() - INTERVAL 30 DAY\n ORDER BY\n";
query += String.format("    job_title COLLATE utf8mb4_0900_ai_ci LIKE CONCAT('%s', '%%') DESC,\n", searchValue);
query +=
    String.format("    IFNULL(NULLIF(INSTR(job_title COLLATE utf8mb4_0900_ai_ci, '%s'), 0), 99999),\n", searchValue);

for (String token : searchTokens) {
    query +=
        String.format("    IFNULL(NULLIF(INSTR(job_title COLLATE utf8mb4_0900_ai_ci, '%s'), 0), 99999),\n", token);
}

query += String.format("    job_title,\n" +
        "    IFNULL(NULLIF(INSTR(job_description COLLATE utf8mb4_0900_ai_ci, '%s'), 0), 99999);", searchValue);

List<Map<String, Object>> result = jdbcTemplate.queryForList(query);

我收到此错误：

nested exception is org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [SELECT * FROM jobs WHERE 0=0 
    OR job_title COLLATE utf8mb4_0900_ai_ci LIKE CONCAT('%', 'software', '%')
    OR job_description COLLATE utf8mb4_0900_ai_ci LIKE CONCAT('%', 'software', '%')
    OR job_title COLLATE utf8mb4_0900_ai_ci LIKE CONCAT('%', 'engineer', '%')
    OR job_description COLLATE utf8mb4_0900_ai_ci LIKE CONCAT('%', 'engineer', '%')
    AND date_posted >= now() - INTERVAL 30 DAY
 ORDER BY
    job_title COLLATE utf8mb4_0900_ai_ci LIKE CONCAT('software engineer', '%') DESC,
    IFNULL(NULLIF(INSTR(job_title COLLATE utf8mb4_0900_ai_ci, 'software engineer'), 0), 99999),
    IFNULL(NULLIF(INSTR(job_title COLLATE utf8mb4_0900_ai_ci, 'software'), 0), 99999),
    IFNULL(NULLIF(INSTR(job_title COLLATE utf8mb4_0900_ai_ci, 'engineer'), 0), 99999),
    job_title,
    IFNULL(NULLIF(INSTR(job_description COLLATE utf8mb4_0900_ai_ci, 'software engineer'), 0), 99999);]; SQL state [HY001]; error code [1038]; Out of sort memory, consider increasing server sort buffer size; nested exception is java.sql.SQLException: Out of sort memory, consider increasing server sort buffer size] with root cause

java.sql.SQLException: Out of sort memory, consider increasing server sort buffer size
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129) ~[mysql-connector-java-8.0.17.jar:8.0.17]
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97) ~[mysql-connector-java-8.0.17.jar:8.0.17]
    at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) ~[mysql-connector-java-8.0.17.jar:8.0.17]
    at com.mysql.cj.jdbc.StatementImpl.executeQuery(StatementImpl.java:1200) ~[mysql-connector-java-8.0.17.jar:8.0.17]
    at org.springframework.jdbc.core.JdbcTemplate$1QueryStatementCallback.doInStatement(JdbcTemplate.java:439) ~[spring-jdbc-5.2.8.RELEASE.jar:5.2.8.RELEASE]
    at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:376) ~[spring-jdbc-5.2.8.RELEASE.jar:5.2.8.RELEASE]
    at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:452) ~[spring-jdbc-5.2.8.RELEASE.jar:5.2.8.RELEASE]
    at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:462) ~[spring-jdbc-5.2.8.RELEASE.jar:5.2.8.RELEASE]
    at org.springframework.jdbc.core.JdbcTemplate.queryForList(JdbcTemplate.java:490) ~[spring-jdbc-5.2.8.RELEASE.jar:5.2.8.RELEASE]
    at com.jobtech.controller.JobroleSearchController.submit(JobroleSearchController.java:49) ~[classes/:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
    at java.base/java.lang.reflect.Method.invoke(Method.java:567) ~[na:na]
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105) ~[spring-webmvc-5.2.8.RELEASE.jar:5.2.8.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878) ~[spring-webmvc-5.2.8.RELEASE.jar:5.2.8.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792) ~[spring-webmvc-5.2.8.RELEASE.jar:5.2.8.RELEASE]
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.2.8.RELEASE.jar:5.2.8.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) ~[spring-webmvc-5.2.8.RELEASE.jar:5.2.8.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) ~[spring-webmvc-5.2.8.RELEASE.jar:5.2.8.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.2.8.RELEASE.jar:5.2.8.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) ~[spring-webmvc-5.2.8.RELEASE.jar:5.2.8.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:652) ~[tomcat-embed-core-9.0.37.jar:4.0.FR]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.2.8.RELEASE.jar:5.2.8.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:733) ~[tomcat-embed-core-9.0.37.jar:4.0.FR]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.37.jar:9.0.37]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1589) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.37.jar:9.0.37]
    at java.base/java.lang.Thread.run(Thread.java:830) ~[na:na]

我已尝试将 sort_buffer_size 设置为 256K，但此错误仍然存​​在。但是，我读过将排序缓冲区大小设置得太大是不好的。当它在 MySQL 终端中运行得很好时，是什么导致我在这个 SQL 语句中出错？

【问题讨论】：

  • 仅供参考： 运算符优先级意味着WHERE a OR b OR c AND d 意味着WHERE a OR b OR (c AND d)，并且由于第一个表达式是0=0，因此您的查询运行就像有根本没有 WHERE 子句，即它正在处理整个表。难怪排序缓冲区已满。

标签： java mysql sql spring jdbc


【解决方案1】：

这是因为数据库的内部缓冲区对于查询来说太小了。

在 MySQL 命令中运行以下查询以增加内存

SET GLOBAL sort_buffer_size = 512000000; // It'll reset after server restart

要设置永久，编辑以下文件并添加：

sudo nano /etc/mysql/mysql.conf.d/mysqld.cnf

sort_buffer_size = 512000000;

sudo service mysql restart

请注意，1073741824 等于字节，将分配 1 GB 内存。

另外，有可能是其他原因导致问题，您可以检查 CPU、内存和慢查询日志

SHOW ENGINE INNODB STATUS

然后您可以根据当前缓冲池大小分析输出。

BUFFER POOL AND MEMORY
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 412090368
Dictionary memory allocated 380237
Buffer pool size   24576
Free buffers       23618
Database pages     954
Old database pages 372
Modified db pages  0
Pending reads      0

更多详情，请参考 MySQL 文档：MySQL Optimization

【讨论】：

    【解决方案2】：

    您可以通过修复where 子句来解决内存不足问题。日期过滤只适用于最后一个条件，因为括号是错误的。

    所以：

    WHERE (job_title COLLATE utf8mb4_0900_ai_ci LIKE CONCAT('%', 'software', '%') OR
       job_description COLLATE utf8mb4_0900_ai_ci LIKE CONCAT('%', 'software', '%') OR
       job_title COLLATE utf8mb4_0900_ai_ci LIKE CONCAT('%', 'engineer', '%') OR
       ob_description COLLATE utf8mb4_0900_ai_ci LIKE CONCAT('%', 'engineer', '%')
      ) AND
      date_posted >= now() - INTERVAL 30 DAY

    注意：为了提高性能，您可能需要考虑在此表上使用全文索引。此外，在查询中使用collate 表明存在数据建模问题。如果您想要不区分大小写的搜索。 . .好吧，无论如何，这都是 MySQL 中的正常默认值。

    【讨论】：

    • 如何在这种情况下使用全文提高性能？
    • @JimmyNesbitt 。 . .使用具有正确排序规则的全文索引应该可以提高性能。您可以在文档中了解它们。
    【解决方案3】：

    这是因为数据库的内部缓冲区对于查询来说太小了。

    您可以在以下配置文件/etc/mysql/my.cnf中增加缓冲区：

    sort_buffer_size to 512K

    【讨论】：

    • 但是如果是这样，为什么这个查询可以直接通过mysql运行呢？
    • 哦，我没看到 0=0。对不起！这意味着该子句总是正确的，因为其他人已经写了。我会说直接 SQL 查询使用排序缓冲区比 Java 语句更有效，这只会在 Java 下导致错误。
    猜你喜欢
    • 2015-06-17
    • 1970-01-01
    • 1970-01-01
    • 2023-03-29
    • 1970-01-01
    • 2019-08-06
    • 1970-01-01
    • 2012-06-16
    • 1970-01-01
    相关资源
    最近更新 更多
    热门标签
    Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode