Spring Boot REST API 服务器连接有时超时

Question

我正在使用 Apache JMeter 测试我的 Spring Boot REST API 服务器。

“连接时间”的大部分时间为 10 毫秒，但有时会发生连接超时。

JMeter 的设置

- Thread
Number of Threads : 500
Ramp-Up Period : 50

- HTTP Request
Connect timeout : 20000ms
Response timeout : 40000ms

- Constant timer
Thread delay : 1000ms

CentOS7 设置

open files                      (-n) 65535
max user processes              (-u) 65535

Java 执行设置

-Xmx8g -Xms4g -Xmn2g -XX:PermSize=512m -XX:SurvivorRatio=16

Spring Boot application.properties Tomcat 设置

server.tomcat.connection-timeout=20000
server.tomcat.max-connections=100000
server.tomcat.threads.max=10000
server.tomcat.threads.min-spare=2500
server.tomcat.accept-count=10000

JMeter 正常工作最多 10,000~20,000 个请求，但之后会出现连接超时。

Starting standalone test @ Fri Apr 16 09:30:29 KST 2021 (1618533029412)
Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445
summary +      1 in 00:00:03 =    0.4/s Avg:  1319 Min:  1319 Max:  1319 Err:     0 (0.00%) Active: 127 Started: 127 Finished: 0
summary +   8771 in 00:00:28 =  318.5/s Avg:   398 Min:   236 Max:  4191 Err:     0 (0.00%) Active: 500 Started: 500 Finished: 0
summary =   8772 in 00:00:30 =  290.3/s Avg:   398 Min:   236 Max:  4191 Err:     0 (0.00%)
summary +   9349 in 00:00:30 =  311.6/s Avg:   534 Min:   237 Max:  3582 Err:     0 (0.00%) Active: 500 Started: 500 Finished: 0
summary =  18121 in 00:01:00 =  300.9/s Avg:   468 Min:   236 Max:  4191 Err:     0 (0.00%)
summary +   5872 in 00:00:30 =  195.7/s Avg:  1494 Min:   235 Max: 20855 Err:    10 (0.17%) Active: 500 Started: 500 Finished: 0
summary =  23993 in 00:01:30 =  265.9/s Avg:   719 Min:   235 Max: 20855 Err:    10 (0.04%)

org.apache.http.conn.HttpHostConnectException: Connect to <IP Address>:<Port> [/112.220.184.107] failed: Connection timed out: connect
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156)
    at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl$JMeterDefaultHttpClientConnectionOperator.connect(HTTPHC4Impl.java:336)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374)
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
    at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.executeRequest(HTTPHC4Impl.java:843)
    at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:574)
    at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:67)
    at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1231)
    at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1220)
    at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:622)
    at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:546)
    at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:486)
    at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:253)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection timed out: connect
    at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
    at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:607)
    at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75)
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
    ... 19 more

你能告诉我为什么会这样吗？

Answer 1

summary +   5872 in 00:00:30 =  195.7/s Avg:  1494 Min:   235 Max: 20855 Err:    10 (0.17%) Active: 500 Started: 500 Finished: 0
                                                                  ^^^^^^^

您的响应时间增加到 20 秒以上，因此 Tomcat 拒绝连接（很可能该设置充当slow HTTP attacks 的保护措施

要么将其设置为 -1 以禁用它，要么使用 profiler tool 查看导致应用程序/tomcat 端出现问题的原因。

此外，JMeter 命令行输出并不能说明全部情况，我建议从 .jtl 结果文件中生成一些图表，例如 Response Times Over Time、Active Threads Over Time、Transactions per Second 等，这样您就可以关联各种性能指标。监控服务器端操作系统健康指标（如 CPU、RAM、网络、交换、磁盘使用情况以及 JVM JMX 指标）也是一个好主意，可以使用 JMeter PerfMon Plugin