PERL LWP :: Parallel :: UserAgent使用代理和自定义用户代理

Question

我无法建立并行连接来更改用户代理并通过代理。这是我的简单脚本：

use HTTP::Request; 
use LWP::ConnCache; 
use LWP::Parallel::UserAgent; 

$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0; 


my $reqs = [ 
    HTTP::Request->new('GET', 'https://website1.com'), 
    HTTP::Request->new('GET', 'https://website2.com'), 
    HTTP::Request->new('GET', 'https://website3.com'), 
    HTTP::Request->new('GET', 'https://website4.com'), 
    HTTP::Request->new('GET', 'https://website5.com'), 
    HTTP::Request->new('GET', 'https://website6.com'), 
]; 



my ($req,$res); 

# register requests 
foreach $req (@$reqs) { 
    print "Registering '".$req->url."'\n"; 

    $ua = LWP::Parallel::UserAgent->new(); 
    $ua->duplicates(0); 
    $ua->timeout(30);   
    $ua->redirect(1); 
    $ua->agent("Test Service v1"); #this doesn't work 

    my $proxy_server = '192.168.10.10:8080'; 
    $ua->proxy(['https', 'http', 'ftp'] => $proxy_server); 

    $ua->register ($req , \&handle_answer); 
} 

my $entries = $ua->wait(); 




sub handle_answer { 
    my ($content, $response, $protocol, $entry) = @_; 

    print "Handling answer from '",$response->request->url,": ", 
      length($content), " bytes, Code ", 
      $response->code, ", ", $response->message,"\n"; 

    if (length ($content)) { 
     $response->add_content($content); 
    } else { 

    } 


    return undef; 
} 

我发现更改用户代理的唯一方法是通过模块HTTP :: Headers并更改http请求。

use HTTP::Headers; 
.... 
... 
my $headers = new HTTP::Headers(
    'User-Agent' => "Test Service v1", 
); 

my $reqs = [ 
    HTTP::Request->new('GET', 'https://website1.com', $headers), 
    ... 
]; 

... 

但我不能让通过代理的请求...... 感谢您的帮助

Answer 1

也许还有更好的解决方案，但我会用螺纹：

#!/usr/bin/perl 

use threads; 
use LWP::UserAgent; 

my $ua = LWP::UserAgent->new(); 
    $ua->agent("MY USER AGENT"); 
    $ua->proxy(['https', 'http', 'ftp'] => "http://192.168.10.10:8080"); 

my @urls = (
'https://website1.com', 
'https://website2.com', 
'https://website3.com' 
); 

my @threads; 
for (@urls) { 
    push @threads, async { $ua->get($_) }; 
} 

for my $thread (@threads) { 
    my $response = $thread->join; 
    if ($response->is_success) { 
     print $response->status_line, "\n"; 
    } 
}