You are asking too abstract question and trying to get too specific answer.
If you want an example that works for you, provide a details: what sites
do you want to parse, where they come from, why do you need to throttle
download queue, what exactly r u trying to limit (max connections at a
time, max kicks per a time unit) and how (per link, per host, per ip,
per subnet).
On 11/16/2016 01:04 AM, Alex Povolotsky wrote:
I've found the exact example I needed. Unfortunately, it is commented
and described in Russian. I did not need nothing like minion
On Mon, Nov 14, 2016 at 6:46 PM, Scott Wiersdorf
<[email protected] <mailto:[email protected]>> wrote:
I agree with Alexander Karelas: your question is a textbook job
queue application.
It is possible to solve this using only non-blocking user agents,
but in either case you'll have to write some kind of controller
that keeps track of "in-flight" requests, a queue of pending
requests, and something to manage the queue. It's not a trivial
application to do it well—you're essentially implementing Minion.
This is likely not the right forum to ask for someone to write
this for you.
Scott
On Monday, November 14, 2016 at 5:53:50 AM UTC-7, Alex Povolotsky
wrote:
I do not need an extra job manager with forks, I need to
employ Mojo's event-based nonblocking I/O
On Mon, Nov 14, 2016 at 3:46 PM, Alexander Karelas
<[email protected]> wrote:
I'm not an expert, but what you're asking for looks like a
job for Minion: https://metacpan.org/pod/Minion
<https://metacpan.org/pod/Minion>
On 14/11/16 14:41, Alex Povolotsky wrote:
Looks like I was unclear. I do not need only extract all
links, I must continue fetching and parsing them, not
more than N at a time and if possible not less. I need
some king of mirroring crawler.
On Sat, Nov 12, 2016 at 6:25 PM, Scott Wiersdorf
<[email protected]> wrote:
Here is a complete working example you can run:
#!/usr/bin/env perl
use Mojolicious::Lite;
get '/random-urls' => sub {
my $c = shift;
$c->render_later;
$c->delay(
sub { ## first step
my $delay = shift;
$c->ua->get('https://www.random.org/bytes/
<https://www.random.org/bytes/>', $delay->begin);
$c->ua->get('https://www.random.org/integer-sets/
<https://www.random.org/integer-sets/>',
$delay->begin);
$c->ua->get('https://www.random.org/strings/
<https://www.random.org/strings/>', $delay->begin);
$c->ua->get('https://www.random.org/audio-noise/
<https://www.random.org/audio-noise/>',
$delay->begin);
},
sub { ## second step
my $delay = shift;
for my $dom (map { $_->res->dom } @_) {
say STDERR $dom->at('title')->text;
}
$c->render(text => "Got all the links");
}
);
};
app->start;
When the results finally all come back from the first
step, the second step will print out the page titles
to STDERR, and return to the client "Got all the links".
Scott
On Saturday, November 12, 2016 at 2:47:57 AM UTC-7,
Alex Povolotsky wrote:
Hello
Examples of Mojo::UserAgent are limited to
fetching a set of files.
But I need a bit more complex thing: I need to
parse a site, reading pages, parsing it and
reading links, using non-blocking UA with, say, 4
downloads at a time, no more and if possible no less.
Can someone give me a good example?
Alex
--
You received this message because you are subscribed
to a topic in the Google Groups "Mojolicious" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/mojolicious/wdp_pgd4e0k/unsubscribe
<https://groups.google.com/d/topic/mojolicious/wdp_pgd4e0k/unsubscribe>.
To unsubscribe from this group and all its topics,
send an email to [email protected].
To post to this group, send email to
[email protected].
Visit this group at
https://groups.google.com/group/mojolicious
<https://groups.google.com/group/mojolicious>.
For more options, visit
https://groups.google.com/d/optout
<https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to
the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails
from it, send an email to [email protected].
To post to this group, send email to
[email protected].
Visit this group at
https://groups.google.com/group/mojolicious
<https://groups.google.com/group/mojolicious>.
For more options, visit
https://groups.google.com/d/optout
<https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to a
topic in the Google Groups "Mojolicious" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/mojolicious/wdp_pgd4e0k/unsubscribe
<https://groups.google.com/d/topic/mojolicious/wdp_pgd4e0k/unsubscribe>.
To unsubscribe from this group and all its topics, send an
email to [email protected].
To post to this group, send email to
[email protected].
Visit this group at
https://groups.google.com/group/mojolicious
<https://groups.google.com/group/mojolicious>.
For more options, visit https://groups.google.com/d/optout
<https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to a topic in
the Google Groups "Mojolicious" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/mojolicious/wdp_pgd4e0k/unsubscribe
<https://groups.google.com/d/topic/mojolicious/wdp_pgd4e0k/unsubscribe>.
To unsubscribe from this group and all its topics, send an email
to [email protected]
<mailto:[email protected]>.
To post to this group, send email to [email protected]
<mailto:[email protected]>.
Visit this group at https://groups.google.com/group/mojolicious
<https://groups.google.com/group/mojolicious>.
For more options, visit https://groups.google.com/d/optout
<https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to the Google
Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [email protected]
<mailto:[email protected]>.
To post to this group, send email to [email protected]
<mailto:[email protected]>.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
