具有代理和自定义用户代理的 PERL LWP::Parallel::UserAgent

Question

我无法更改用户代理并通过代理进行并行连接。这是我的简单脚本：

use HTTP::Request; 
use LWP::ConnCache;
use LWP::Parallel::UserAgent;

$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0; 


my $reqs = [  
     HTTP::Request->new('GET', 'https://website1.com'),
     HTTP::Request->new('GET', 'https://website2.com'),
     HTTP::Request->new('GET', 'https://website3.com'),
     HTTP::Request->new('GET', 'https://website4.com'),
     HTTP::Request->new('GET', 'https://website5.com'),
     HTTP::Request->new('GET', 'https://website6.com'),
];



my ($req,$res);

# register requests
foreach $req (@$reqs) {
    print "Registering '".$req->url."'\n";

    $ua = LWP::Parallel::UserAgent->new();
    $ua->duplicates(0);
    $ua->timeout(30);        
    $ua->redirect(1);
    $ua->agent("Test Service v1");  #this doesn't work

    my $proxy_server = '192.168.10.10:8080';
    $ua->proxy(['https', 'http', 'ftp'] => $proxy_server);

    $ua->register ($req , \&handle_answer);
}

my $entries = $ua->wait();




sub handle_answer {
    my ($content, $response, $protocol, $entry) = @_;

    print "Handling answer from '",$response->request->url,": ",
          length($content), " bytes, Code ",
          $response->code, ", ", $response->message,"\n";

    if (length ($content) ) {
        $response->add_content($content);
    } else {

    }


    return undef;
}

我发现更改用户代理的唯一方法是通过模块 HTTP::Headers 并更改 http 请求。

use HTTP::Headers;
....
...
my $headers = new HTTP::Headers(
    'User-Agent' => "Test Service v1",
); 

my $reqs = [  
     HTTP::Request->new('GET', 'https://website1.com', $headers),
     ...
];

...

但我无法通过代理发出请求... 感谢您的帮助

Answer 1

也许有更好的解决方案，但我会使用线程：

#!/usr/bin/perl

use threads;
use LWP::UserAgent;

my $ua = LWP::UserAgent->new();
   $ua->agent("MY USER AGENT");
   $ua->proxy(['https', 'http', 'ftp'] => "http://192.168.10.10:8080");

my @urls = (
 'https://website1.com',
 'https://website2.com',
 'https://website3.com'
);

my @threads;
for (@urls) {
   push @threads, async { $ua->get($_) };
}

for my $thread (@threads) {
   my $response = $thread->join;
   if ($response->is_success) {
       print $response->status_line, "\n";
   }
}