LWP::Simple & LWP::UserAgent 不返回完整的网页内容

Question

我有一台内部本地机器，用于托管我需要处理的分析。如果您有 URL，分析将以 JSON 格式返回。例如，

http://1.2.3.4:8081/analytics/mydata/myNodeData?flat

它会给出 JSON 文件。现在我通过 LWP::Simple（也尝试过 LWP::UserAgent）将 JSON 文件获取到一个变量中，然后对其进行解析。它适用于我的大多数 URL。问题在于，对于某些 URL，它不会将整个数据读入我的变量，而只是将部分数据读入。

use LWP::Simple;                # From CPAN
use LWP::UserAgent;
use JSON qw( decode_json );     # From CPAN
use Data::Dumper;               # Perl core module
use Data::Diver qw{ Dive };         # for diving in the Hash
use strict;                     # Good practice
use warnings;                   # Good practice

#using LWP Simple    
my $trendsurl = 'http://1.2.3.4:8081/analytics/mydata/myNodeData?flat';
my $json = get( $trendsurl );
die "Could not get $trendsurl!" unless defined $json;
my $decoded_json = decode_json( $json );

#using LWP::UserAgent
my $ua = LWP::UserAgent->new();
my $req = new HTTP::Request GET => $trendsurl;
my $res = $ua->request($req);
my $content = $res->content;

如果它在这里有用的是显示内容的调试信息（decoded_json）是相同的。

  DB<1> p $content
{"NodeStatus": {"deleted": false, "disk_usage_info": [{"partition_space_availabl
e_1k": 791475728, "partition_space_used_1k": 171611096, "partition_name": "/dev/
mapper/os-root", "partition_type": "ext4"}, {"partition_space_available_1k": 151
200, "partition_space_used_1k": 39244, "partition_name": "/dev/vda3", "partition
_type": "ext2"}], "process_info": [{"process_name": "XX-api:0", "process_s
tate": "PROCESS_STATE_RUNNING", "last_stop_time": null, "start_count": 2, "core_
file_list": [], "last_start_time": "1461822985334246", "stop_count": 0, "last_ex
it_time": null, "exit_count": 0}, {"process_name": "XX-config-nodemgr", "p
rocess_state": "PROCESS_STATE_RUNNING", "last_stop_time": null, "start_count": 2
, "core_file_list": [], "last_start_time": "1461822979324868", "stop_count": 0,
"last_exit_time": null, "exit_count": 0}, {"process_name": "XX-discovery:0
", "process_state": "PROCESS_STATE_RUNNING", "last_stop_time": null, "start_coun
t": 2, "core_file_list": [], "last_start_time": "1461822983332516", "stop_count"
: 0, "last_exit_time": null, "exit_count": 0}, {"process_name": "XX-svc-m

  DB<2>

你看到日志文件被突然终止了..如果我通过浏览器得到这个，它就完成了..

编辑：

我在获取 URL 然后解码 JSON 时添加了以下 eval 语句。检索没有错误，但 JSON 解码失败，因为它不是完整的 JSON。

eval { $decoded_json = parse_json( $json ) };
if ($@) {
    warn "func raised an exception: $@";
}
func raised an exception: JSON error at line 1: Unexpected end of input parsing 
string starting from byte 1105 at Tester.pl line 28.

编辑 2： 我只是尝试了另一种方式...

#!/usr/bin/perl  

 use LWP::UserAgent;  
 use HTTP::Request;  

 my $URL = 'http://1.2.3.4:8081/analytics/mydata/myNodeData?flat';  

 my $ua = LWP::UserAgent->new();  
 my $header = HTTP::Request->new(GET => $URL);  
 my $request = HTTP::Request->new('GET', $URL, $header);  
 my $response = $ua->request($request);  

 if ($response->is_success){  
     print "URL:$URL\nHeaders:\n";  
     print $response->headers_as_string;  
 }elsif ($response->is_error){  
     print "Error:$URL\n";  
     print $response->error_as_HTML;  
 } 

调试显示选择了 $response->is_success 块，这是输出。

URL:http://1.2.3.4:8081/analytics/mydata/myNodeData?flat
Headers:
Connection: close
Date: Tue, 10 May 2016 19:49:40 GMT
Content-Type: application/json
Client-Aborted: die
Client-Date: Tue, 10 May 2016 19:49:38 GMT
Client-Peer: 1.2.3.4:8081
Client-Response-Num: 1
Client-Transfer-Encoding: chunked
X-Died: read failed: An existing connection was forcibly closed by the remote ho
st. at C:/Perl64/lib/LWP/Protocol/http.pm line 465.

Answer 1

我使用这个代码或类似的东西，这个代码使用 POST 作为参数，我得到一个大的 JSON 并且它工作得很好

use LWP::UserAgent;
use Data::Dumper;
use JSON;
use JSON::Parse;

my $params = 
{(
'flat' => 1,
)};

    my $server_endpoint = "http://1.2.3.4:8081/analytics/mydata/myNodeData";
    my $ua = LWP::UserAgent->new;
    my $req = HTTP::Request->new();
    my $response = $ua->post( $server_endpoint, $params );
    my $result  = $response->decoded_content();

    $result = from_json($result);
    print Dumper $result;