In article <7xipgj8vxh@ruckus.brouhaha.com>,
Paul Rubin wrote:
> Roy Smith writes:
> > I agree that application-level name cacheing is "wrong", but sometimes
> > doing it the wrong way just makes sense. I could whip up a simple
> > cacheing wrapper around getaddrinfo() in 5 minutes. Dep
On 4/28/2012 1:04 PM, Paul Rubin wrote:
Roy Smith writes:
I agree that application-level name cacheing is "wrong", but sometimes
doing it the wrong way just makes sense. I could whip up a simple
cacheing wrapper around getaddrinfo() in 5 minutes. Depending on the
environment (both technology
Roy Smith writes:
> I agree that application-level name cacheing is "wrong", but sometimes
> doing it the wrong way just makes sense. I could whip up a simple
> cacheing wrapper around getaddrinfo() in 5 minutes. Depending on the
> environment (both technology and bureaucracy), getting a cach
On Sun, Apr 29, 2012 at 12:27 AM, Danyel Lawson wrote:
> I'm glad I thought of it. ;) But the trick is to use port 5353 and set
> a really short timeout on responses in the config for the DNS cache.
I don't think false timeouts are any better than true ones, if you
actually know the true ones. Bu
I'm glad I thought of it. ;) But the trick is to use port 5353 and set
a really short timeout on responses in the config for the DNS cache.
On Sat, Apr 28, 2012 at 10:15 AM, Chris Angelico wrote:
> On Sat, Apr 28, 2012 at 11:46 PM, Danyel Lawson
> wrote:
>> The DNS lookup is one of those things
On Sat, Apr 28, 2012 at 11:46 PM, Danyel Lawson wrote:
> The DNS lookup is one of those things that may make sense to run as a
> separate daemon process that listens on a socket.
Yeah, it does. One that listens on port 53, TCP and UDP, perhaps. :)
You've just recommended installing a separate ca
Sprinkle time.sleep(0) liberally throughout your code where you think
natural processing breaks should be. Even in while loops. It's lame
but is the only way to make Python multithreading task switch fairly.
Your compute intensive tasks need a time.sleep(0) in their loops. This
prevents starvation
In article <7xy5pgqwto@ruckus.brouhaha.com>,
Paul Rubin wrote:
> John Nagle writes:
> >I may do that to prevent the stall. But the real problem was all
> > those DNS requests. Parallizing them wouldn't help much when it took
> > hours to grind through them all.
>
> True dat. But bui
On 4/27/2012 9:55 PM, Paul Rubin wrote:
John Nagle writes:
I may do that to prevent the stall. But the real problem was all
those DNS requests. Parallizing them wouldn't help much when it took
hours to grind through them all.
True dat. But building a DNS cache into the application seem
John Nagle writes:
>I may do that to prevent the stall. But the real problem was all
> those DNS requests. Parallizing them wouldn't help much when it took
> hours to grind through them all.
True dat. But building a DNS cache into the application seems like a
kludge. Unless the number of
On 4/27/2012 9:20 PM, Paul Rubin wrote:
John Nagle writes:
The code that stored them looked them up with "getaddrinfo()", and
did this while a lock was set.
Don't do that!!
Added a local cache in the program to prevent this.
Performance much improved.
Better to release the lock while
John Nagle writes:
> The code that stored them looked them up with "getaddrinfo()", and
> did this while a lock was set.
Don't do that!!
>Added a local cache in the program to prevent this.
> Performance much improved.
Better to release the lock while the getaddrinfo is running, if you can
On Sat, Apr 28, 2012 at 1:35 PM, John Nagle wrote:
> On CentOS, "getaddrinfo()" at the
> glibc level doesn't always cache locally (ref
> https://bugzilla.redhat.com/show_bug.cgi?id=576801). Python
> doesn't cache either.
How do you manage your local cache? The Python getaddrinfo function
doesn't
On 4/27/2012 6:25 PM, Adam Skutt wrote:
On Apr 27, 2:54 pm, John Nagle wrote:
I have a multi-threaded CPython program, which has up to four
threads. One thread is simply a wait loop monitoring the other
three and waiting for them to finish, so it can give them more
work to do. When the
On Apr 27, 2:54 pm, John Nagle wrote:
> I have a multi-threaded CPython program, which has up to four
> threads. One thread is simply a wait loop monitoring the other
> three and waiting for them to finish, so it can give them more
> work to do. When the work threads, which read web pages a
On 27/04/2012 23:30, Dennis Lee Bieber wrote:
Oh, continuation thought...
If the workers are calling into C-language operations, unless those
operations release the GIL, it doesn't matter what the OS or Python
thread switch timings are. The OS may interrupt the thread (running
C
John Nagle writes:
>I know that the CPython thread dispatcher sucks, but I didn't
> realize it sucked that bad. Is there a preference for running
> threads at the head of the list (like UNIX, circa 1979) or
> something like that?
I think it's left up to the OS thread scheduler, Windows in yo
On 4/27/2012 20:54, John Nagle wrote:
I have a multi-threaded CPython program, which has up to four
threads. One thread is simply a wait loop monitoring the other
three and waiting for them to finish, so it can give them more
work to do. When the work threads, which read web pages and
then parse
I have a multi-threaded CPython program, which has up to four
threads. One thread is simply a wait loop monitoring the other
three and waiting for them to finish, so it can give them more
work to do. When the work threads, which read web pages and
then parse them, are compute-bound, I've had
19 matches
Mail list logo
