usually python webservers execute your functions in their own threads, and starting your own (threads) is always not recommended. For things like that you'd have to resort to an external process executing your code, and collecting the results somewhere where the app can fetch it (in an async way). Some queue implementations exists, you can try web2py's scheduler for the job (and you can probably simplify the implementation details).
PS: no webapp (and probably server) would scale 1000 concurrent ssh connections in 1000 different threads. I see a pool of 20 and that would mean 50 batches to elaborate a single requirement.....that would probably go into timeout before returning meaningful results if executed inside a webserver (even if you managed to get your threads running correctly). Il giorno domenica 29 luglio 2012 23:28:48 UTC+2, Tito Garrido ha scritto: > > Hi Folks! > > I'd like to create a web application that will basically send a command to > 1000 servers and grab some data using subprocess + ssh like: > > *p = sub.Popen(['ssh','-q', 'support@%s'%hostname, > command],stdout=sub.PIPE, stderr=sub.STDOUT) > out = p.stdout.read() > print out # it will be a db insert > p.poll() > rc=p.returncode > print rc # it will be a db insert* > > I can't execute all threads at the same time so I would need to pool then > and execute using Queue module. > > My first code was:* > class ThreadSSH(threading.Thread): > """Thread ssh command""" > def __init__(self, queue, name): > threading.Thread.__init__(self) > self.queue = queue > self.name = name > > def run(self): > #while not self.queue.empty(): > while True: > # print 'Starting %s' %self.name > #grabs host from queue > print 'Empty: %s' %self.queue.empty() > print 'Qsize: %s' %self.queue.qsize() > print '%s' %dir(self.queue) > host = self.queue.get() > > #core of the thread > time.sleep(10) > print 'Active count: %s' %threading.activeCount() > print "Exiting " + self.name > > #signals to queue job is done > self.queue.task_done() > > def test(): > hosts = ['host1', 'host2', 'host3', 'host4'] > # Queue of servers, we will fill it down there > queue = Queue.Queue() > print 'begin: %s' %threading.enumerate() > #spawn a pool of threads, and pass them queue instance > for i in range(20): > t = ThreadSSH(queue, 'test-%s' %i) > t.setDaemon(True) > t.start() > print 'thread generated: %s' %threading.enumerate() > > #populate queue with data > for host in hosts: > queue.put(host) > print 'SIZE: %s' %queue.qsize()* > > > Not sure if this is the right approach for this kind of application but > the problem is that those threads never ends... if I put in another > function: > *def threadInfo(): > output='enumerate: %s<br/>' %['%s Live(%s) Daemon(%s)' > %(t.name,t.isAlive(),t.isDaemon()) > for t in threading.enumerate()] > output+='active count: %s<br/>' %threading.activeCount() > output+='current thread: %s<br/>' %threading.currentThread().name > return output* > > I can see that even the thread ending it is showing on enumerate. > > Do you guys have another idea or know how to fix this issue? > > Thanks in advance, > > Tito > > -- > > Linux User #387870 > .........____ > .... _/_õ|__| > ..º[ .-.___.-._| . . . . > .__( o)__( o).:_______ > --
