[Twisted-Python] can QtWebKit fire a twisted deferred?
''' I know this is meant for pyqt but I've had little luck this that board. The short answer is yes, but it's a total hack...and I'm wondering if anybody has a more elegant solution. Specifically I would like to be able to fire a defer from the javascript instead of having to call alert. Below is an example of how I currently gain access...not necessarily a working version. ''' url = 'http://www.google.com' mo = """ var observer = new MutationObserver(function(mutations) { for (var x = 0; x < mutations.length; x++) { if (mutations[x].addedNodes.length > 0) { alert('add nodes') } } }); var config = { attributes: true, childList: true, characterData: false } var nti = document.querySelector('document.body') observer.observe(nti, config); """ page = QWebPage() def alarmAction(qframe, msg): print 'deferred action', qframe, msg def mutationHook(): d = defer.Deferred() d.addCallback(alarmAction) page.javaScriptAlert = d.callback page.mainFrame().documentElement().evaluateJavaScript(mo) page.loadFinished(mutationHook) page.mainFrame().setUrl(url) ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] can QtWebKit fire a twisted deferred?
That was pseudo code, I've been using qt5reactor to happy results. All the same good to know on deferreds working in non-twisted code... My challenge is having only 2 hooks into the DOM - javaScriptAlert and javaScriptConfirm that ultimately with QtWebEngine aren't available. thank for the feedback. On Wed, May 6, 2015 at 3:35 PM, Glyph wrote: > > On May 6, 2015, at 11:04 AM, Kevin Mcintyre wrote: > > ''' > I know this is meant for pyqt but I've had little luck this that board. > > The short answer is yes, but it's a total hack...and I'm wondering if > anybody has a more elegant solution. > > Specifically I would like to be able to fire a defer from the javascript > instead of having to call alert. > > Below is an example of how I currently gain access...not necessarily a > working version. > ''' > > > I don't see any import of qt4reactor or Twisted (other than the implied > import of Deferred) in this code. Firing a Deferred in this way should be > totally fine though; you don't actually need the reactor for Deferreds to > work (it's just important that if it's going to interact with any *other* > Twisted > code it needs to run on the main thread). > > Sorry I don't have a better suggestion, but I don't see much wrong with > what you're doing here :). > > -g > > url = 'http://www.google.com' > > mo = """ > var observer = new MutationObserver(function(mutations) { > for (var x = 0; x < mutations.length; x++) { > if (mutations[x].addedNodes.length > 0) { > alert('add nodes') > } > } > }); > var config = { attributes: true, childList: true, characterData: > false } > var nti = document.querySelector('document.body') > observer.observe(nti, config); > """ > > page = QWebPage() > def alarmAction(qframe, msg): > print 'deferred action', qframe, msg > def mutationHook(): > d = defer.Deferred() > d.addCallback(alarmAction) > page.javaScriptAlert = d.callback > page.mainFrame().documentElement().evaluateJavaScript(mo) > > > page.loadFinished(mutationHook) > page.mainFrame().setUrl(url) > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Dumb Logger API question
http://twistedmatrix.com/documents/current/core/howto/logger.html I'm looking at the new-ish logger api and wonder how to log to console as a first step to transitioning. Any pointers? Thanks. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Dumb Logger API question
yes, I stared at that section for a bit. Not to get off topic, but the readability of the api docs from back when is slightly better in my opinion. So the topic being I'm an idiot - thanks for both examples...that's a perfect example. I got confused by "from myobservers import PrintingObserver" cause I'm wondering if myobservers is a core package. On Tue, Nov 10, 2015 at 6:05 PM, Glyph Lefkowitz wrote: > > On Nov 9, 2015, at 2:29 PM, Kevin Mcintyre wrote: > > http://twistedmatrix.com/documents/current/core/howto/logger.html > > I'm looking at the new-ish logger api and wonder how to log to console as > a first step to transitioning. > > Any pointers? Thanks. > > > Do you mean "emit structured log text to stdout"? In that case, that is > covered in the document you link to; these two sections: > > > http://twistedmatrix.com/documents/current/core/howto/logger.html#avoid-mutable-event-keys > > http://twistedmatrix.com/documents/current/core/howto/logger.html#starting-the-global-log-publisher > > create a jsonFileLogObserver and start logging to it: > > import sys > from twisted.logger import jsonFileLogObserver, globalLogBeginner, Logger > globalLogBeginner.beginLoggingTo([jsonFileLogObserver(sys.stdout)]) > log = Logger() > log.info("Information.") > > > If you want unstructured log output for human reading (not a good choice > for an automated system, but perhaps useful for debugging), you can instead > go with textFileLogObserver, in almost exactly the same way: > > import sys > from twisted.logger import textFileLogObserver, globalLogBeginner, Logger > globalLogBeginner.beginLoggingTo([textFileLogObserver(sys.stdout)]) > log = Logger() > log.info("Information.") > > > Is this what you were looking for? > > -glyph > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] asynchronous response
Hey - I'm confused, so nothing new :) ...but I'm running at this example and I'm scratching my head. http://twistedmatrix.com/documents/current/web/howto/web-in-60/asynchronous-deferred.html I would've thought 2 requests could be served simultaneously, but when I fire off 2 requests the first received gets it's response after 5 seconds, while the second response takes 10 seconds. I think I understand what's happening, but I don't know why...and I would love an example where the subsequent request doesn't have to wait for the first request to finish. Thanks, Kevin ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] asynchronous response
Just a follow-up with my own sanity check example. http://twistedmatrix.com/documents/current/web/howto/web-in-60/asynchronous-deferred.html My conclusion is that deferLater is a blocking method - am I wrong? On Wed, Dec 16, 2015 at 4:11 PM, Kevin Mcintyre wrote: > Hey - I'm confused, so nothing new :) ...but I'm running at this example > and I'm scratching my head. > > > http://twistedmatrix.com/documents/current/web/howto/web-in-60/asynchronous-deferred.html > > I would've thought 2 requests could be served simultaneously, but when I > fire off 2 requests the first received gets it's response after 5 seconds, > while the second response takes 10 seconds. > > I think I understand what's happening, but I don't know why...and I would > love an example where the subsequent request doesn't have to wait for the > first request to finish. > > Thanks, > Kevin > > > > > > from twisted.internet.task import deferLater from twisted.web.resource import Resource from twisted.web.server import NOT_DONE_YET, Site from twisted.internet import reactor, defer, threads from twisted.web.xmlrpc import Proxy from twisted.web.xmlrpc import XMLRPC from twisted.web.client import getPage import time import xmlrpclib class TestResource(Resource): isLeaf = True def render_ERROR(self, err, request): print 'xmlrpc error:', self.__class__.__name__, err request.finish() def render_TIMESTAMP(self, ts, request): request.write(str(ts)) request.finish() def render_GET(self, request): print 'test:', int(time.time()) d = Proxy('http://localhost:8082').callRemote('timestamp') d.addCallback(self.render_TIMESTAMP, request) d.addErrback(self.render_ERROR, request) return NOT_DONE_YET class Test1Resource(TestResource): isLeaf = True def render_GET(self, request): print 'test1:', int(time.time()) d = threads.deferToThread(xmlrpclib.ServerProxy("http://localhost:8082";).timestamp) d.addCallback(self.render_TIMESTAMP, request) d.addErrback(self.render_ERROR, request) return NOT_DONE_YET class Test2Resource(XMLRPC): def delayed_html(self): time.sleep(5) return int(time.time()) def xmlrpc_timestamp(self): print 'timestamp:', int(time.time()) ''' blocking!! return self.delayed_html() ''' return threads.deferToThread(self.delayed_html) def done_or_error(ans = None): print 'done_or_error:', ans reactor.stop() def results(res, init_time): print init_time, res[0][1], res[1][1], res[2][1], res[3][1], int(time.time()) def quad_request(): init_time = int(time.time()) dl = defer.DeferredList([getPage('http://localhost:8080'), getPage('http://localhost:8080'), getPage('http://localhost:8081'), getPage('http://localhost:8081')]) dl.addCallback(results, init_time) dl.addBoth(done_or_error) if __name__ == '__main__': print 'start reactor' reactor.listenTCP(8080, Site(TestResource())) reactor.listenTCP(8081, Site(Test1Resource())) reactor.listenTCP(8082, Site(Test2Resource())) reactor.callWhenRunning(quad_request) reactor.run()___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] asynchronous response
holy moly - you're right! This fool will sleep so much better...a million thanks. On Thu, Dec 17, 2015 at 12:09 AM, Amber "Hawkie" Brown < hawk...@atleastfornow.net> wrote: > > > On 17 Dec 2015, at 08:11, Kevin Mcintyre wrote: > > > > Hey - I'm confused, so nothing new :) ...but I'm running at this example > and I'm scratching my head. > > > > > http://twistedmatrix.com/documents/current/web/howto/web-in-60/asynchronous-deferred.html > > > > I would've thought 2 requests could be served simultaneously, but when I > fire off 2 requests the first received gets it's response after 5 seconds, > while the second response takes 10 seconds. > > > > I think I understand what's happening, but I don't know why...and I > would love an example where the subsequent request doesn't have to wait for > the first request to finish. > > > > Thanks, > > Kevin > > > I've ran into this before -- browsers sometimes rate-limit requests, and > won't actually send the second request until the first is done, over the > same connection, rather than making a second TCP connection -- try using > cURL or wget, which has no such limitation, and see if it works any better. > > - Amber > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > from twisted.internet.task import deferLater from twisted.web.resource import Resource from twisted.web.server import NOT_DONE_YET, Site from twisted.internet import reactor, defer from twisted.web.client import getPage import time class DelayedResource(Resource): isLeaf = True def _delayedRender(self, request): request.write(str(int(time.time( request.finish() def render_GET(self, request): d = deferLater(reactor, 5, lambda: request) d.addCallback(self._delayedRender) return NOT_DONE_YET def results(res, init_time): print init_time, res[0][1], res[1][1], int(time.time()) def done_or_error(ans = None): print 'done_or_error:', ans reactor.stop() def double_request(): init_time = int(time.time()) dl = defer.DeferredList([getPage('http://localhost:8080'), getPage('http://localhost:8080')]) dl.addCallback(results, init_time) dl.addBoth(done_or_error) if __name__ == '__main__': reactor.listenTCP(8080, Site(DelayedResource())) reactor.callWhenRunning(double_request) print 'start reactor' reactor.run()___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] on template processing
1) Is template block processing done top to bottom sequentially? I've included example.py - it's interesting that by having a blocking call first the second request is blocked, but having the blocking call after a non-blocking call the second request is not. I would've thought it would behave as a deferred list, but looking at _flattenTree and guessing not. Maybe related to wait_for_it example? 2) Is it possible for a Resource to act as an Element too? I've included a non-working elementresource.py. I'm a total hack, but I would think that if an instance had a loader attr it could be processable. 3) Is it possible to include xmlns:t=" http://twistedmatrix.com/ns/twisted.web.template/0.1"; not in a tag itself? Or perhaps have a tag like 'render-block' that could be transparent-like? 4) Is it possible to have xmlns:t=" http://twistedmatrix.com/ns/twisted.web.template/0.1"; in multiple places in a template? I have some cases with inline scripts that bonk out with > or < characters. This is very much related to #3. 5) Is it possible for a render element to return something like "<~sometag t:render>..." and process recursively? 6) Is there any examples of connection keep-alive long polling? 7) Examples of request based scoping would be great. All the examples on http://twistedmatrix.com/documents/13.0.0/web/howto/twisted-templates.html have flatten(None... - I've included request_scope.py 8) The wait_for_it example, is that meant as a chunked transfer example? It would be cool to have an example I could open in a browser. Trying to wrap my head around this and subviews in the meantime. Happy holidays. from twisted.web import server, resource, template, client from twisted.internet import reactor, defer, threads import time import pprint blocking_first = """http://twistedmatrix.com/ns/twisted.web.template/0.1";> Blocking:Non-blocking:""" blocking_second = """http://twistedmatrix.com/ns/twisted.web.template/0.1";> Non-blocking:Blocking:""" class ExampleTemplate(template.Element): loader = template.XMLString(blocking_second) def non_blocking_call(self): print 'called non-blocking' d = defer.Deferred() d.addCallback(lambda ign: str(int(time.time( reactor.callLater(2, d.callback, None) return d def blocking_call(self): print 'called blocking' time.sleep(2) return str(int(time.time())) @template.renderer def blocking(self, request, tag): ''' switch to yield threads.deferToThread(self.blocking_call) to unblock ''' return self.blocking_call() @template.renderer def non_blocking(self, request, tag): yield self.non_blocking_call() class ExampleResource(resource.Resource): isLeaf = True def render_GET(self, request): print 'get:', int(time.time()) d = template.flattenString(request, ExampleTemplate()) d.addCallback(request.write) d.addCallback(lambda ign: request.finish()) return server.NOT_DONE_YET site = server.Site(ExampleResource()) def simulataneous(): dl = defer.DeferredList([client.getPage('http://localhost:6789'), client.getPage('http://localhost:6789')]) dl.addCallback(lambda res: [pprint.pprint(ans[1]) for ans in res ]) dl.addErrback(lambda err: pprint.pprint(err)) if __name__ == '__main__': reactor.listenTCP(6789, site) reactor.callWhenRunning(simulataneous) reactor.run()from twisted.web import server, resource, template, client from twisted.internet import reactor import pprint class ElementResource(resource.Resource, template.Element): isLeaf = True loader = template.XMLString("""http://twistedmatrix.com/ns/twisted.web.template/0.1";>""") @template.renderer def whatever(self, request, tag): return 'whatever' def render_GET(self, request): d = template.flattenString(request, self) d.addCallback(request.write) d.addCallback(lambda ign: request.finish()) return server.NOT_DONE_YET site = server.Site(ElementResource()) def test(): d = client.getPage('http://localhost:6789') d.addCallback(lambda res: pprint.pprint(res)) d.addErrback(lambda err: pprint.pprint(err.getErrorMessage())) if __name__ == '__main__': reactor.listenTCP(6789, site) reactor.callWhenRunning(test) reactor.run()from twisted.web import server, resource, template, client from twisted.internet import reactor, defer import pprint class ExampleTemplate(template.Element): loader = template.XMLString("""http://twistedmatrix.com/ns/twisted.web.template/0.1";>result:""") @template.renderer def result(self, request, tag): return request.result class ExampleResource(resource.Resource): inventory = { 'item' : 1 } isLeaf = True def shop(self, act, item, request):
[Twisted-Python] web Resource question
what is child.server = self.server in Resource.putChild? shouldn't it be child.server = self? ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] web Resource question
Hey, thanks for the reply. It seems like there's the possibility of more beneficial assignments...but first trying to understand if there's a specific reason for the child.server assignment. On Thu, Jun 16, 2016 at 5:11 AM, Cory Benfield wrote: > > > On 16 Jun 2016, at 05:16, Kevin Mcintyre wrote: > > > > > > what is child.server = self.server in Resource.putChild? > > > > shouldn't it be child.server = self? > > I don’t think so. The reference to the server is presumably intended to > refer to the Site object that is hosting the Resource. With that said, I > can’t actually *find* anywhere that sets the server on a Resource, so it’s > possible that the whole thing is vestigial. > > Cory > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] web client FileBodyProducer - transfer encoding
hey all - quick question. Trying to understand FileBodyProducer as it pertains to POST. json_body = FileBodyProducer(StringIO(json.dumps({'key': 'value'}))) agent.request("POST", ~*uri*, Headers({'User-Agent': ['AkamaiTest']}), json_body) Does the agent chunk the POST body? Is this controllable? Any insight appreciated! ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] web client FileBodyProducer - transfer encoding
Dealing with older apache, ruby, passenger setup. Hoping to mimic behavior and show OPS that chunked encoding isn't working correctly. With python requests (http://docs.python-requests.org/en/master/) all POST calls work correctly, but twisted requests are failing. I'm thinking (and hoping) it's failing because twisted is chunking the post body. When I send the requests to a twisted endpoint like below the requests are successful and the response is identical. from twisted.web import server, resource from twisted.internet import reactor class Simple(resource.Resource): isLeaf = True def render_POST(self, request): print request.content.getvalue() return request.content.getvalue() site = server.Site(Simple()) reactor.listenTCP(8080, site) reactor.run() On Mon, Jan 30, 2017 at 2:39 PM, Glyph Lefkowitz wrote: > > On Jan 30, 2017, at 13:47, Kevin Mcintyre wrote: > > hey all - quick question. Trying to understand FileBodyProducer as it > pertains to POST. > > json_body = FileBodyProducer(StringIO(json.dumps({'key': 'value'}))) > > agent.request("POST", ~*uri*, Headers({'User-Agent': ['AkamaiTest']}), > json_body) > > Does the agent chunk the POST body? > > Maybe! > > Is this controllable? > > Not really. The current implementation will certainly use chunked > encoding sometimes, but there isn't a strong API guarantee of this anywhere. > > Any insight appreciated! > > You probably don't want to try to control this. I believe that proxies > are within their rights to mess around with chunk boundaries and re-buffer > things, so you don't have any strong guarantees that chunk sizes will be > preserved. > > Why is it that you want to control chunking in the first place? > > -glyph > > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] web client FileBodyProducer - transfer encoding
Thanks - that's exactly what I was looking for. *But note* when using FileBodyProducer(StringIO(json.dumps(~blah))) -- I still see a content-length in the request header as it's received by twisted. Am I correct to assume the request is agnostic...meaning it's shaped the same for twisted as it is for apache. Headers({'host': ['localhost:8080'], 'connection': ['close'], 'content-length': ['671'], 'user-agent': ['PassengerTest']}) from twisted.web import server, resource from twisted.internet import reactor class Simple(resource.Resource): isLeaf = True def render_POST(self, request): print request.requestHeaders return request.content.getvalue() site = server.Site(Simple()) reactor.listenTCP(8080, site) reactor.run() On Mon, Jan 30, 2017 at 4:59 PM, Jean-Paul Calderone < exar...@twistedmatrix.com> wrote: > On Mon, Jan 30, 2017 at 7:44 PM, Glyph Lefkowitz > wrote: > >> Gotcha. I guess what I meant was that you shouldn't care about this at >> the application level, but you're talking about an operational concern, not >> an application-level concern. >> >> Perhaps this should be a tunable on Agent somehow. Can you file a ticket? >> > > > Well... It *is* sort of tuneable, as you implied earlier. > > If you pass an IBodyProducer with a non-None length, Agent will send a > Content-Length header - not use chunked Transfer-Encoding. > FileBodyProducer doesn't know how to determine the length of a StringIO, so > you get chunked with this example. If you write the JSON to a regular file > and FileBodyProducer(open(the file)) you'll get a Content-Length request. > You could also write a new (trivial) IBodyProducer that does know how to > compute the length of a StringIO. > > The documentation doesn't exactly spell this out - but the only reason > `length` is part of the interface is to be able to generate the > Content-Length header. > > What would the ticket be? Expanded documentation to make this behavior > into an explicit guarantee of the interface? A new toggle somewhere to > force Agent.request into one mode or the other (regardless of the > performance consequences)? > > And that stuff you said about proxies before was true ... So even if you > can control this in Agent, you're still not *guaranteed* the server will > see what you send. > > Jean-Paul > > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] web client FileBodyProducer - transfer encoding
"It is shaped the same. The reason you're seeing the error is due to the issue I pointed out above." Just to be clear you mean that proxies can reshape the call in flight? On Mon, Jan 30, 2017 at 6:01 PM, Glyph Lefkowitz wrote: > > On Jan 30, 2017, at 19:44, Kevin Mcintyre wrote: > > On Mon, Jan 30, 2017 at 4:59 PM, Jean-Paul Calderone < > exar...@twistedmatrix.com> wrote: > >> On Mon, Jan 30, 2017 at 7:44 PM, Glyph Lefkowitz > > wrote: >> >>> Gotcha. I guess what I meant was that you shouldn't care about this at >>> the application level, but you're talking about an operational concern, not >>> an application-level concern. >>> >>> Perhaps this should be a tunable on Agent somehow. Can you file a >>> ticket? >>> >> >> >> Well... It *is* sort of tuneable, as you implied earlier. >> >> If you pass an IBodyProducer with a non-None length, Agent will send a >> Content-Length header - not use chunked Transfer-Encoding. >> FileBodyProducer doesn't know how to determine the length of a StringIO, so >> you get chunked with this example. >> > > Not quite true: FileBodyProducer *does* know how to compute the length of > a StringIO, via seek(): > > > >>> from twisted.web.client import FileBodyProducer > >>> from io import BytesIO > >>> from cStringIO import StringIO > >>> FileBodyProducer(BytesIO(b"some bytes")).length > 10L > >>> FileBodyProducer(StringIO(b"some bytes")).length > 10 > > > If you write the JSON to a regular file and FileBodyProducer(open(the >> file)) you'll get a Content-Length request. You could also write a new >> (trivial) IBodyProducer that does know how to compute the length of a >> StringIO. >> >> The documentation doesn't exactly spell this out - but the only reason >> `length` is part of the interface is to be able to generate the >> Content-Length header. >> >> What would the ticket be? Expanded documentation to make this behavior >> into an explicit guarantee of the interface? A new toggle somewhere to >> force Agent.request into one mode or the other (regardless of the >> performance consequences)? >> > > Rather than "regardless" of the consequences, perhaps just a maximum body > size. If we made it an explicit guarantee in Agent, perhaps the interface > change would not be in '.request' (which, as a formal interface used both > inside and outside of Twisted, is fairly fixed), but rather a new > `NonChunkedBodyProducer` concrete class that would implement IBodyProducer > in terms of another IBodyProducer which either does or doesn't have a > `length`. (Or a function that does same, always returning its argument if > `length` is already set...) > > And that stuff you said about proxies before was true ... So even if you >> can control this in Agent, you're still not *guaranteed* the server will >> see what you send. >> > > Thanks - that's exactly what I was looking for. > > *But note* when using FileBodyProducer(StringIO(json.dumps(~blah))) -- I > still see a content-length in the request header as it's received by > twisted. Am I correct to assume the request is agnostic...meaning it's > shaped the same for twisted as it is for apache. > > > It is shaped the same. The reason you're seeing the error is due to the > issue I pointed out above. > > Headers({'host': ['localhost:8080'], 'connection': ['close'], > 'content-length': ['671'], 'user-agent': ['PassengerTest']}) > > > -glyph > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] catching twisted pyqt segmentation fault
I have a long running twisted pyqt process which occasionally throws a Received signal 11. I believe it's in the underlying QtWebEngineView since the twisted process continues to run. Whatever the reason I would like to figure out a way to catch this and stop the reactor either in process, or by another process monitor. Any suggestions either twisted or otherwise would be greatly appreciated. Below is a bit of the stack trace dumped to /var/log/syslog 0x7fd855447a8f Feb 25 22:01:32 ip-10-0-0-116 python[1836]: 2 0x7fd855447c74 Feb 25 22:01:32 ip-10-0-0-116 python[1836]: 3 0x7fd8555d6a82 Feb 25 22:01:32 ip-10-0-0-116 python[1836]: 4 0x7fd8556ababd Feb 25 22:01:32 ip-10-0-0-116 python[1836]: 5 0x7fd8556af2f6 Feb 25 22:01:32 ip-10-0-0-116 python[1836]: 6 0x7fd85586e52d Feb 25 22:01:32 ip-10-0-0-116 python[1836]: 7 0x7fd85586f511 Feb 25 22:01:32 ip-10-0-0-116 python[1836]: 8 0x7fd85570dbc3 Feb 25 22:01:32 ip-10-0-0-116 python[1836]: 9 0x7fd85570dd92 Feb 25 22:01:32 ip-10-0-0-116 python[1836]: 10 0x7fd855516c0c Feb 25 22:01:32 ip-10-0-0-116 python[1836]: 11 0x7fd85551dacd Feb 25 22:01:32 ip-10-0-0-116 python[1836]: 12 0x7fd85596c5d3 Feb 25 22:01:32 ip-10-0-0-116 python[1836]: 13 0x35946920c1fb Feb 25 22:01:32 ip-10-0-0-116 python[1836]: Received signal 11 SEGV_MAPERR fbadbeef Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #0 0x7fd856d5ca2e Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #1 0x7fd856d5cdeb Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #2 0x7fd8526d3390 Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #3 0x7fd858404408 Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #4 0x7fd855447a8f Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #5 0x7fd855447c74 Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #6 0x7fd8555d6a82 Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #7 0x7fd8556ababd Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #8 0x7fd8556af2f6 Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #9 0x7fd85586e52d Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #10 0x7fd85586f511 Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #11 0x7fd85570dbc3 Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #12 0x7fd85570dd92 Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #13 0x7fd855516c0c Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #14 0x7fd85551dacd Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #15 0x7fd85596c5d3 Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #16 0x35946920c1fb Feb 25 22:01:32 ip-10-0-0-116 python[1836]: r8: 7fd85a2f38c0 r9: 0013 r10: r11: Feb 25 22:01:32 ip-10-0-0-116 python[1836]: r12: 7fd858aa25c8 r13: 7fd858ab91ee r14: 7fd85a0c9848 r15: Feb 25 22:01:32 ip-10-0-0-116 python[1836]: di: si: 0013 bp: 7fd858ab91ee bx: Feb 25 22:01:32 ip-10-0-0-116 python[1836]: dx: 7fd851e1d770 ax: fbadbeef cx: 7fd851b4e6fd sp: Feb 25 22:01:32 ip-10-0-0-116 python[1836]: ip: 7fd858404408 efl: 00010246 cgf: 0033 erf: Feb 25 22:01:32 ip-10-0-0-116 python[1836]: trp: 000e msk: cr2: fbadbeef Feb 25 22:01:32 ip-10-0-0-116 python[1836]: [end of stack trace] ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] catching twisted pyqt segmentation fault
Thanks glyph - really appreciate all the times you've answered my questions! QtWebEngineView is running a subprocess. /home/ubuntu/Qt5.8.0/5.8/gcc_64/libexec/QtWebEngineProcess --type=renderer --disable-accelerated-video-decode --disable-gpu-memory-buffer-video-frames --enable-threaded-compositing --use-gl=desktop --disable-webrtc-hw-encoding --primordial-pipe-token=032B94B57A61DAE34BD0F90551F9ABB6 --lang=en-US --enable-pinch --num-raster-threads=2 --content-image-texture-target=3553 QT can sometimes be helpful, but bug resolution can take a while. Any suggestions outside of twisted for detecting segfaults? I'm thinking just to monitor /var/log/syslog On Sat, Feb 25, 2017 at 11:59 PM, Glyph Lefkowitz wrote: > > > On Feb 25, 2017, at 4:41 PM, Kevin Mcintyre wrote: > > > > I have a long running twisted pyqt process which occasionally throws a > Received signal 11. > > > > I believe it's in the underlying QtWebEngineView since the twisted > process continues to run. Whatever the reason I would like to figure out a > way to catch this and stop the reactor either in process, or by another > process monitor. Any suggestions either twisted or otherwise would be > greatly appreciated. > > Presumably the QtWebEngineView is running a subprocess, then? This is > really a Qt question rather than a Twisted question, because the > QtWebEngineView should be emitting some kind of signal when its rendering > process crashes. (If it doesn't I imagine you're just out of luck here and > you should report a bug to the Qt developers...) > > -glyph > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] fastcgi client
Saw this... https://twistedmatrix.com/documents/8.2.0/api/twisted.web2.channel.fastcgi.html Was wondering if there's a fastcgi client out there for twisted? Thanks, ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] ProcessMonitor stderr stdout
Newbie question: I'm calling addProcess and everything works well, but I'm trying to understand how to capture/handle stderr stdout for that spawned process. -- Kevin McIntyre | Software Engineer | *Green Charge Networks* Cell 718-249-5058 | kmcint...@greenchargenet.com | www.greenchargenet.com ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] DeprecationWarning and PotentialZombieWarning
I recently upgraded to 12.0.0 and was running my process in the shell, when this error/warning was thrown. Any idea what it means? DeprecationWarning: twisted.internet.error.PotentialZombieWarning was deprecated in Twisted 10.0.0: There is no longer any potential for zombie process. o = getattr (m, sym) DeprecationWarning: twisted.protocols.basic.COMMA was deprecated in Twisted 10.2.0: NetstringReceiver parser state is private. o = getattr (m, sym) DeprecationWarning: twisted.protocols.basic.DATA was deprecated in Twisted 10.2.0: NetstringReceiver parser state is private. o = getattr (m, sym) DeprecationWarning: twisted.protocols.basic.LENGTH was deprecated in Twisted 10.2.0: NetstringReceiver parser state is private. o = getattr (m, sym) DeprecationWarning: twisted.protocols.basic.NUMBER was deprecated in Twisted 10.2.0: NetstringReceiver parser state is private. o = getattr (m, sym) DeprecationWarning: twisted.internet.interfaces.IFinishableConsumer was deprecated in Twisted 11.1.0: Please use IConsumer (and IConsumer.unregisterProducer) instead. o = getattr (m, sym) DeprecationWarning: twisted.internet.interfaces.IReactorArbitrary was deprecated in Twisted 10.1.0: See IReactorFDSet. o = getattr (m, sym) -- Kevin McIntyre | Software Engineer | Green Charge Networks Cell 718-249-5058 | kmcint...@greenchargenet.com | www.greenchargenet.com ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] protocol connectionLost clientConnectionLost
Below is my sample code. I'm hoping to unplug my internet cable and get a connectionLost. But this doesn't happen. Is there anyway to detect this? -- from twisted.internet.protocol import Protocol, ReconnectingClientFactory from twisted.internet import reactor class Echo(Protocol): def connectionMade(self): print "cool and the gang" def connectionLost(self, reason): print "ut-oh %s" % reason def dataReceived(self, data): pass class EchoClientFactory(ReconnectingClientFactory): def startedConnecting(self, connector): print 'Started to connect.' def buildProtocol(self, addr): print 'Connected.' print 'Resetting reconnection delay' self.resetDelay() return Echo() def clientConnectionLost(self, connector, reason): print 'Lost connection. Reason:', reason ReconnectingClientFactory.clientConnectionLost(self, connector, reason) def clientConnectionFailed(self, connector, reason): print 'Connection failed. Reason:', reason ReconnectingClientFactory.clientConnectionFailed(self, connector, reason) if ( __name__ == "__main__" ): reactor.connectTCP("74.125.224.242", 80, EchoClientFactory() ) #connect to google on port 80 reactor.run() -- Kevin McIntyre | Software Engineer | Green Charge Networks Cell 718-249-5058 | kmcint...@greenchargenet.com | www.greenchargenet.com ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] more complex IBodyProducer?
has anyone implemented a more complex IBodyProducer than class POSTRequestProducer(): implements(IBodyProducer) def __init__(self, body): self.body = body self.length = len(self.body) def startProducing(self, consumer): consumer.write(self.body) return succeed(None) def pauseProducing(self): pass def stopProducing(self): pass I have an upload intensive application, that during periods of poor connectivity ends up with a lots on zombie http TCP connections. I'm thinking maybe that the "pause" is being called and that my request deferred is in a paused state. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] web templating with html
https://twistedmatrix.com/documents/current/web/howto/twisted-templates.html#auto7 I see the example with quoting, but what if I want to render html? I'm trying to get the text/html part of an email, and want to render as is. Thanks in advance. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] maybe video
Wanted to share my qt4reactor webkit cv2 video capture script. It's comes in handy whenever I've needed to scrape just the browser and not my whole screen or capture unit tests etc. here is a demo quick demo in action: http://www.youtube.com/watch?v=2XMXtUISY1g&feature=youtu.be It's a hack, but that's why I like python (espeically twisted)...it's so versatile. https://github.com/kmcintyre/scewpt/blob/master/pyscewpt/maybevideo.py Cheers, Kevin from PyQt4.QtGui import QApplication, QKeyEvent, QPainter, QImage import cv2 import sys print 'args:', sys.argv video = None if len(sys.argv) > 2: print 'start video:', sys.argv[2] video = cv2.VideoWriter( sys.argv[2] + '.avi', cv2.cv.CV_FOURCC('M','J','P','G'), 24, (1024, 768),True) from PyQt4.QtWebKit import QWebView, QWebPage, QWebSettings class MaybeVideo(QApplication): # you may need def spoof_hit_enter(self, receiver): print 'spoof' super(MaybeVideo, self).notify(receiver, QKeyEvent(QEvent.KeyPress, Qt.Key_Enter, Qt.NoModifier)) super(MaybeVideo, self).notify(receiver, QKeyEvent(QEvent.KeyRelease, Qt.Key_Enter, Qt.NoModifier)) painter = QPainter() events = [] ready = False def notify(self, receiver, event): if MaybeVideo.ready and isinstance(receiver, QWebView) and not MaybeVideo.painter.isActive() and video is not None: try: #print 'clip screenshot' image = QImage(QSize(1024, 768), QImage.Format_RGB32) MaybeVideo.painter.begin(image) MaybeVideo.painter.setRenderHint(QPainter.Antialiasing, True) MaybeVideo.painter.setRenderHint(QPainter.TextAntialiasing, True) MaybeVideo.painter.setRenderHint(QPainter.SmoothPixmapTransform, True) MaybeVideo.painter.setRenderHint(QPainter.HighQualityAntialiasing, True) receiver.page().mainFrame().render(MaybeVideo.painter) image.save("temp.jpg", "jpg") video.write( cv2.imread('temp.jpg')) MaybeVideo.painter.end() except Exception as e: print e if event.type() == QEvent.SockAct: MaybeVideo.ready = True elif not event.type() in MaybeVideo.events: #print 'new type', receiver, event.type(), QEvent.User MaybeVideo.events.append(event.type()) else: pass return super(MaybeVideo, self).notify(receiver, event) app = MaybeVideo([]) import qt4reactor qt4reactor.install() from PyQt4.QtGui import QMainWindow from PyQt4.QtCore import QEvent, Qt, QSize, QUrl from twisted.internet import reactor import signal from PyQt4.QtNetwork import QNetworkAccessManager, QNetworkCookieJar class MaybeVideoWindow(QMainWindow): def __init__(self, parent=None): QMainWindow.__init__(self, parent) self.web_page = QWebPage() print self.web_page.settings() self.web_page.setNetworkAccessManager(QNetworkAccessManager()) self.web_page.networkAccessManager().setCookieJar(QNetworkCookieJar()) self.resize(QSize(1024, 768)) web_view = QWebView() web_view.resize(QSize(1024, 768)) web_view.settings().setAttribute(QWebSettings.AutoLoadImages, True) web_view.settings().setAttribute(QWebSettings.JavascriptEnabled, True) web_view.settings().setAttribute(QWebSettings.JavaEnabled, False) web_view.settings().setAttribute(QWebSettings.JavascriptCanOpenWindows, False) web_view.settings().setAttribute(QWebSettings.PluginsEnabled, True) self.web_page.setView(web_view) self.setCentralWidget(web_view) self.setWindowTitle('MaybeVideo') self.web_page.view().urlChanged.connect(self._page_url_change) self.web_page.loadStarted.connect(self._page_start) self.web_page.loadProgress.connect(self._page_progress) self.web_page.loadFinished.connect(self._page_finished) def go(self, url): #print 'go:', url self.web_page.view().load(QUrl(url)) def _page_progress(self, percent): #print 'page progress', percent pass def _page_start(self): #print 'page start' pass def _page_finished(self, ok): #print 'page finished:', str(ok) pass def _page_url_change(self, url): print '_url_change:', str(url.toString()) mvw = MaybeVideoWindow() def end(ign, ign2): print 'stopping' reactor.stop() def start(): mvw.show() if len(sys.argv) > 1: mvw.go('http://' + sys.argv[1] ) else
[Twisted-Python] stmp client with multipart
How can I get something similar into a smtp client 12? multipart message fn = "example.mp3" multipart = MIMEMultipart('alternative') multipart['Subject'] = 'Tutorate!' multipart['To'] = 'Selfie' multipart['From'] = 'Selfie' text = "Hello, how are you, goodbye." textpart = MIMEText(text) multipart.attach(textpart) htmlpart = MIMEText("" + text + "", 'html') multipart.attach(htmlpart) part = MIMEBase('audio', "mp3") part.set_payload( open(fn,"rb").read() ) Encoders.encode_base64(part) part.add_header('Content-Disposition', 'attachment; filename="%s"' % os.path.basename(fn)) multipart.attach(part) io = StringIO.StringIO() g = Generator(io, False) g.flatten(multipart) v = io.getvalue() class SMTPTutorialClient(smtp.ESMTPClient): mailFrom = "mg_selfie@ " mailTo = "mg@ " def getMailFrom(self): result = self.mailFrom self.mailFrom = None return result def getMailTo(self): return [self.mailTo] def getMailData(self): print v return StringIO.StringIO(v) def sentMail(self, code, resp, numOk, addresses, log): print 'Sent', numOk, 'messages' from twisted.internet import reactor reactor.stop() ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] stmp client with multipart
ah, it just seems like the next smtp client (potentially 12) from the samples could be multipart mime. On Fri, Mar 28, 2014 at 3:12 PM, wrote: > On 08:36 pm, kebi...@gmail.com wrote: > >> How can I get something similar into a smtp client 12? multipart message >> > > Hi Kevin, > > MIME and SMTP are at different layers. > > Twisted's SMTP client doesn't care what bytes you shove through it. > They're just bytes. They can be MIME or not. Construct the bytes using > the stdlib's MIME functionality if you want - once you have the bytes, the > way you use them with Twisted's SMTP client is the same no matter what they > are. > > Jean-Paul > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Using twisted for a smtp dialog
I've been working a lot with smtp and twisted is great for easily doing what you're describing. Have a look at the smtpclient docs...that should get you most of the way there. http://twistedmatrix.com/documents/current/mail/tutorial/smtpclient/smtpclient.html On Wed, Apr 2, 2014 at 9:03 AM, Grigorescu Ciprian < grigorescu_cipr...@yahoo.com> wrote: > Hello, > > I am new in twisted world and I am struggle to understand it. > I am trying to create a smtp connection and initiate a smtp dialog: helo, > mail from, rcpt, for a list of address, and catch the answer from the smtp > server > > Can anyone point me the right direction, where to start? > > Thanks a lot, > Grig > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] XMLString clarification
I'm still having trouble getting my multipart text/html from an email into XMLString. I run the html through beautifulsoup then remove the doctype, but if there's anything at all off about the resulting xml it still craps out. Is there anyway to stick an arbitrary string into a template? Or should I add a callback to flatten and add the placeholder in the flatten stage and replace the placeholder with the actually html. Any guidance appreciated, Kevin ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] XMLString clarification
I've opted for placeholder and added callback to flatten. I'm a hack, and it was holding up delivery of email. On Tue, Apr 8, 2014 at 4:19 PM, Werner Thie wrote: > Hi Kevin > > > On 4/8/14 9:34 AM, Kevin Mcintyre wrote: > >> I'm still having trouble getting my multipart text/html from an email >> into XMLString. >> >> I run the html through beautifulsoup then remove the doctype, but if >> there's anything at all off about the resulting xml it still craps out. >> > > How? Traceback? > > > Is there anyway to stick an arbitrary string into a template? Or should >> I add a callback to flatten and add the placeholder in the flatten stage >> and replace the placeholder with the actually html. >> > > Werner > > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] XMLString clarification
I will try to get a test class together to demonstrate. On Tue, Apr 8, 2014 at 4:38 PM, Kevin Mcintyre wrote: > I've opted for placeholder and added callback to flatten. > > I'm a hack, and it was holding up delivery of email. > > > On Tue, Apr 8, 2014 at 4:19 PM, Werner Thie wrote: > >> Hi Kevin >> >> >> On 4/8/14 9:34 AM, Kevin Mcintyre wrote: >> >>> I'm still having trouble getting my multipart text/html from an email >>> into XMLString. >>> >>> I run the html through beautifulsoup then remove the doctype, but if >>> there's anything at all off about the resulting xml it still craps out. >>> >> >> How? Traceback? >> >> >> Is there anyway to stick an arbitrary string into a template? Or should >>> I add a callback to flatten and add the placeholder in the flatten stage >>> and replace the placeholder with the actually html. >>> >> >> Werner >> >> >> ___ >> Twisted-Python mailing list >> Twisted-Python@twistedmatrix.com >> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >> > > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] dumb question on qt4reactor
calling reactor.callLater within qt4reactor in an attempt to async non webkit work but not seeing expected behavior. Webkit locks while the callLater function executes. Any insight appreciated. Should I move real work into separate reactor and "broker" calls into qt4reactor/webkit. I have xmlrpc and websocket hooks into browser...but looking for guidance as to most direct method of brokering calls between reactors. Any insight appreciated. Having a blast using QT browser as a back-end, see demo site www.athleets.com. See below for amazon S3 upload worker that I expected to async, but in qt4reactor does not appear to work as planned. from pyscewpt import fixed from pyscewpt.s3 import bucket_util import boto from boto.s3.bucket import Bucket from boto.s3.key import Key from twisted.internet import defer, reactor bucket_queue = defer.DeferredQueue() def upload_worker(data): print 'macth upload_worker', data save_match( data['league'], data['twitter'], data['local_file'], data['meta'], data['profile']) return len(bucket_queue.pending) def match_queue(ign=None): print 'match_queue:', len(bucket_queue.pending), len(bucket_queue.waiting) if ign is None or ign >= 2: print 'requeue:', len(bucket_queue.pending) , len(bucket_queue.waiting) d = bucket_queue.get() d.addCallback(upload_worker) d.addBoth(match_queue) return d else: return None def queue_save_match(queue_dic): bucket_queue.put(queue_dic) print 'queue_save_match', len(bucket_queue.pending), len(bucket_queue.waiting), queue_dic['twitter']# if len(bucket_queue.pending) >= 5 and len(bucket_queue.waiting) == 0: print 'queue drop worker' reactor.callLater(0,match_queue,None) def save_match(league, twitter, systemfile, meta, profile): print 'save_match:', twitter, profile s3_filename = league + '/' + twitter + '.png' bucket_util.save_s3( boto.connect_s3().get_bucket('~bucket'), s3_filename, None, systemfile, 'image/png', 'public-read', meta + [('yes', False), ('no', False), ('maybe', False), ('profile', profile)] ) ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] dumb question on qt4reactor
Another "Oh right" twisted moment. callInThread instead of callLater? This is a case where I've been laying tracks for 2 years without totally understanding the train that was eventually coming down the tracks. I guess I implicitly understood by wanting to move to a separate reactor as a separate thread, and in the past when I've used callLater it's encapsulated non-blocking code. In this sense is callLater ever utilized as a pseudo asynchronous method? On Sat, Jul 5, 2014 at 5:24 PM, wrote: > On 5 Jul, 08:35 pm, kebi...@gmail.com wrote: > >> calling reactor.callLater within qt4reactor in an attempt to async non >> webkit work but not seeing expected behavior. Webkit locks while the >> callLater function executes. >> > > `callLater` does not make a blocking function into a non-blocking one. It > just calls a function after a specified delay. If the function blocks, > then the reactor will block when it gets around to calling it. > > If you need to call blocking APIs you may need to call them in a separate > thread (Twisted includes a threadpool that can help you with this) or a > separate process (Twisted has some APIs for launching and interacting with > other processes, too). Of course, the ideal solution is often to find a > non-blocking equivalent to the blocking API and use that instead (for > example, I know that txaws is a project that exists - perhaps it can do the > things to S3 that you want to do). > > Jean-Paul > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] dumb question on qt4reactor
finalizing - maybe I'm not describing my usage correctly. I've used reactor.callLater(0,blah) as a means of starting a unit of work the by-product of which the caller didn't need. this is really a case of laying tracks for 2 years not fully comprehending the train coming down the tracks behind me! I sorta refused to learn/use inlinecallbacks or generators and I've never used callInThread until now. I inserted that into my S3 upload pipeline and it works, simplifying my code but detaches the caller from reactive failure. The mechanism you've described is different ~ a callLater to an estimated point of work completion, maybe a holding pattern if work isn't completed after the delay, and maybe not a bad means of observation (?)...but yes disastrous if the callLater function is wired poorly. Unless I'm misreading. Anyway I'm sure my code is illogical since I've was left to my own devices to solve problems of my own choosing. The by-product you can see here www.pokertalon.com - a single connection html5 app. On Wed, Jul 9, 2014 at 10:51 AM, Kevin Horn wrote: > It might be, but I can't think of a reason off hand why it ever _should_ > be. I can kind of see how it might work, but if your "async" process ever > takes longer than your callLater delay it will break, and probably quite > horribly. > > > On Wed, Jul 9, 2014 at 12:11 PM, Kevin Mcintyre wrote: > >> Another "Oh right" twisted moment. callInThread instead of callLater? >> >> This is a case where I've been laying tracks for 2 years without totally >> understanding the train that was eventually coming down the tracks. I >> guess I implicitly understood by wanting to move to a separate reactor as a >> separate thread, and in the past when I've used callLater it's encapsulated >> non-blocking code. >> >> In this sense is callLater ever utilized as a pseudo asynchronous method? >> >> >> >> On Sat, Jul 5, 2014 at 5:24 PM, wrote: >> >>> On 5 Jul, 08:35 pm, kebi...@gmail.com wrote: >>> >>>> calling reactor.callLater within qt4reactor in an attempt to async non >>>> webkit work but not seeing expected behavior. Webkit locks while the >>>> callLater function executes. >>>> >>> >>> `callLater` does not make a blocking function into a non-blocking one. >>> It just calls a function after a specified delay. If the function blocks, >>> then the reactor will block when it gets around to calling it. >>> >>> If you need to call blocking APIs you may need to call them in a >>> separate thread (Twisted includes a threadpool that can help you with this) >>> or a separate process (Twisted has some APIs for launching and interacting >>> with other processes, too). Of course, the ideal solution is often to find >>> a non-blocking equivalent to the blocking API and use that instead (for >>> example, I know that txaws is a project that exists - perhaps it can do the >>> things to S3 that you want to do). >>> >>> Jean-Paul >>> >>> ___ >>> Twisted-Python mailing list >>> Twisted-Python@twistedmatrix.com >>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >>> >> >> >> ___ >> Twisted-Python mailing list >> Twisted-Python@twistedmatrix.com >> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >> >> > > > -- > -- > Kevin Horn > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Is there pb documentation somewhere?
This discussion seems relevant to a design pattern "Yearbook Lifecycle" that I'm using for athleets.com and junkeet.com. Design Goal: 1) Maintain yearbooks across state, allowing students to delegate access control to classmates as a series of transformations (recursive web renders) - here sign my yearbook...here read what so-and-so wrote..can I take your yearbook to PE, we might skip class. 2) Persist inside a context indexer that answers yearbook queries, with generic access to the t0 and viewable access to t1+ transforms with request controls for represented as e0, e1 being another recursive web template kept within the master at t-1 So a complex use case would be a yearbook message like "OMG - did you read {s1.p3.t4}? cause {e4.s3} said f_td(after|before) {e2.s6} was kissing behind {p3.t0.a4}. See you this summer good luck at {owner.ext['college']}! BFF, {s3}" Fake Model YearbookPage(Element): SignedYearbookPage(YearbookPage): Yearbook(pb.Copyable) -pages -accessControlLog StudentYearbook(pb.Viewable) --transformation YearbookServiceTransformer(??) Publish(pb.Root) remote_issue_yearbook(studentid=None): // none can read but not signed Athleets.com is ready(ing) for yearbook-ish events, ie when a players get traded or hurt. Other events act across yearbook motif regenerating the t0 when athlete data changes which republishes all copies non t0 that exist. The Sender Receiver Model from the example is difficult to follow cause I tend to think of Originator only. The originator being a service that waits for clients in need of a working copy. A sender, doesn't exist...it would be more of a challenger, that if recognized as more authoritative, replaces the rb.Root service. Outside of the copyable I'm trying to upstream via remotes pieces within the copyable that need updating. I'm delving into viewable to bring caller identity data to the web templating process not necessary restrict access. But also to build a system that maintains audit history within the yearbook model. Getting to areas of non-working code, so let me report back any success. Cheers Kevin On Tue, Aug 5, 2014 at 11:55 AM, Glyph Lefkowitz wrote: > > On Aug 4, 2014, at 9:47 PM, Daniel Sank wrote: > > glyph, > > >> 2. Is there a specification for the pb dialect of banana? > >> > > Beyond the code, no. > > Ok. > > > I would be happy to answer questions, but obviously I'm not super > responsive :). > > Let me know what you need. > > For two personal projects, I would like to have a reasonable remote > objects library in python. I need something which can announce state > changes to clients, and receive state change requests from clients. My > solution: > > 1. Make server side class which can spawn Cacheables when it wants to tell > clients of its existence. > 2. Give RemoteCaches to clients and use observe_* methods as appropriate. > 3. Stuff a Viewable into the RemoteCaches so that clients can request > state changes. > > Question #1: Is this a reasonable use of pb? > > > Yes. > > This all worked great until I ran into a bug. In trying to fix the bug, I > found that > > 1. pb code is really hard to understand > > > Sorry about that. > > 2. exarkun's thinks pb is bad and that I should implement what I need in > AMP. > > > I really wish we would stop calling things "bad" and "good". This isn't a > helpful classification. PB is adequate for a particular set of > requirements. Those requirements are somewhat unusual, and AMP is better > for a lot of use-cases. > > It sounds to me like you are a lot more interested in > > 3. exarkun thinks banana and jelly are reasonable. > > > Again, what does "reasonable" mean in this context? > > Let me explain my own opinion about this. > > Banana is a perfectly serviceable low-level marshaling format. It's > pretty efficient when compared to something like JSON, and has compression > mechanisms which can make it even more efficient (the "dialect" support you > referred to). The only thing about it that isn't very general is that its > implementation (although not the protocol specification) hard-codes the PB > abbreviated-string dialect. > > Jelly is higher level, but more language-specific. Its specification > implicitly encodes numerous Python implementation details, like the > distinction between "tuple" and "list". It also couples very tightly to > your program's structure. This can be a real benefit to getting a protocol > up and running quickly, but it still allows you to create protocols where > you don't really know what the wire format is, where you develop hidden > dependencies. In more complex protocols (where the "ease of getting up and > running quickly" thing really starts to shine) this attribute of Jelly can > cause real difficulty in any kind of cross-system communication: > communicating with a peer from a different language, or even in Python > without access to all the protocol class definitions from the original > system, is hard because it requires rev
[Twisted-Python] pb.Copyable knowledge barrier
goal: Give knowitall.py awareness of wordsmith.py addition. to run ./library.sh # opens 7999 for remote_dictionary and remote_define methods then both: python knowitall.py python wordsmith.py library.sh due to main/jelly issues from twisted.spread import pb class UknownWordException(Exception): pass class UnknownDefinition(UknownWordException): pass # would like to remote define via error class Dictionary(): def __init__(self, language, word=None, definition=None): self.language = language self.words = {} if not word: word = language if not definition: definition = language + ' is a word worthy of a language' self.words[word] = set([definition]) print self.words def get_language(self): return self.language def lookup_word(self, word, definition = None): print 'lookup_word', self.words try: return self.words[word] except: if definition: raise UknownWordException(word) else: raise UnknownDefinition(word, definition) class CopyDictionary(Dictionary, pb.Copyable): def getStateToCopy(self): print 'state getting copied', self.language return self.language, self.words class DictionaryEditor(pb.Referenceable): def remote_word_definition(self, lanague, word, definition): pass class RemoteDictionary(pb.RemoteCopy, Dictionary): def setCopyableState(self, state): print 'remote state receives copy', state self.language = state[0] self.words = state[1] class Library(pb.Root): dictionaries = [] def remote_define(self, language, word, definition): dictionary = self.remote_dictionary(language, word, definition) try: definition_set = dictionary.lookup_word(word, definition) print 'got definition_set', definition_set if definition not in definition_set: print 'new definition' definition_set.add(definition) else: print 'existing definition' except Exception as e: print e print 'new word' dictionary.words[word] = set([definition]) return dictionary.words def remote_dictionary(self, language, word = None, definition = None): for d in Library.dictionaries: if d.get_language() == language: print 'remote' return d print 'create copyable version' dictionary = CopyDictionary(language, word, definition) Library.dictionaries.append(dictionary) return dictionary from twisted.spread import pb from example import CopyDictionary, RemoteDictionary from twisted.internet import reactor from twisted.internet import defer factory = pb.PBClientFactory() reactor.connectTCP("localhost", 7999, factory) pb.setUnjellyableForClass(CopyDictionary, RemoteDictionary) def use_dictionary(dictionary): for w in dictionary.words.keys(): for d in dictionary.words[w]: print w, d reactor.callLater(10, use_dictionary, dictionary) def goto_library(): return factory.getRootObject() import sys language = 'english' if len(sys.argv) > 1: language = sys.argv[1] def test(): d = goto_library() d.addCallback(lambda root: root.callRemote('dictionary', language)) d.addCallback(use_dictionary) return d reactor.callWhenRunning(test) reactor.run() library.sh Description: Bourne shell script from twisted.spread import pb from twisted.internet import reactor from twisted.internet import defer factory = pb.PBClientFactory() reactor.connectTCP("localhost", 7999, factory) def goto_library(): return factory.getRootObject() def define_result(ans, language, word, definition): # audit submission later print ans reactor.stop() language = 'english' word = language definition = word + ' is part of ' + language import sys if len(sys.argv) > 1: language = sys.argv[1] if len(sys.argv) > 2: word = sys.argv[2] if len(sys.argv) > 3: definition = sys.argv[3] def test(): d = goto_library() d.addCallback(lambda root: root.callRemote('define', language, word, definition)) d.addCallback(define_result, language, word, definition) return d reactor.callWhenRunning(test) reactor.run() ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] pb.Copyable knowledge barrier
wordsmith argument order issues re-take. Cheers, Kevin On Thu, Aug 7, 2014 at 11:29 PM, Kevin Mcintyre wrote: > goal: Give knowitall.py awareness of wordsmith.py addition. > > to run > ./library.sh # opens 7999 for remote_dictionary and remote_define methods > then both: > python knowitall.py > python wordsmith.py > > library.sh due to main/jelly issues > > from twisted.spread import pb class UknownWordException(Exception): pass class UnknownDefinition(UknownWordException): pass # would like to remote define via error class Dictionary(): def __init__(self, language, word=None, definition=None): self.language = language self.words = {} if not word: word = language if not definition: definition = language + ' is a word worthy of a language' self.words[word] = set([definition]) print self.words def get_language(self): return self.language def lookup_word(self, word, definition = None): print 'lookup_word', self.words try: return self.words[word] except: if definition: raise UknownWordException(word) else: raise UnknownDefinition(word, definition) class CopyDictionary(Dictionary, pb.Copyable): def getStateToCopy(self): print 'state getting copied', self.language return self.language, self.words class DictionaryEditor(pb.Referenceable): def remote_word_definition(self, lanague, word, definition): pass class RemoteDictionary(pb.RemoteCopy, Dictionary): def setCopyableState(self, state): print 'remote state receives copy', state self.language = state[0] self.words = state[1] class Library(pb.Root): dictionaries = [] def remote_define(self, language, word, definition): dictionary = self.remote_dictionary(language, word, definition) try: definition_set = dictionary.lookup_word(word, definition) print 'got definition_set', definition_set if definition not in definition_set: print 'new definition' definition_set.add(definition) else: print 'existing definition' except Exception as e: print e print 'new word' dictionary.words[word] = set([definition]) return dictionary.words def remote_dictionary(self, language, word = None, definition = None): for d in Library.dictionaries: if d.get_language() == language: print 'remote' return d print 'create copyable version' dictionary = CopyDictionary(language, word, definition) Library.dictionaries.append(dictionary) return dictionary from twisted.spread import pb from example import CopyDictionary, RemoteDictionary from twisted.internet import reactor from twisted.internet import defer factory = pb.PBClientFactory() reactor.connectTCP("localhost", 7999, factory) pb.setUnjellyableForClass(CopyDictionary, RemoteDictionary) def use_dictionary(dictionary): for w in dictionary.words.keys(): for d in dictionary.words[w]: print w, d reactor.callLater(10, use_dictionary, dictionary) def goto_library(): return factory.getRootObject() import sys language = 'english' if len(sys.argv) > 1: language = sys.argv[1] def test(): d = goto_library() d.addCallback(lambda root: root.callRemote('dictionary', language)) d.addCallback(use_dictionary) return d reactor.callWhenRunning(test) reactor.run() library.sh Description: Bourne shell script from twisted.spread import pb from twisted.internet import reactor from twisted.internet import defer factory = pb.PBClientFactory() reactor.connectTCP("localhost", 7999, factory) def goto_library(): return factory.getRootObject() def define_result(ans, language, word, definition): # audit submission later print ans reactor.stop() import sys language = 'english' if len(sys.argv) > 1: language = sys.argv[1] word = language if len(sys.argv) > 2: word = sys.argv[2] definition = word + ' is part of ' + language if len(sys.argv) > 3: definition = sys.argv[3] print 'language', language print 'word', word print 'definition', definition def test(): d = goto_library() d.addCallback(lambda root: root.callRemote('define', language, word, definition)) d.addCallback(define_result, language, word, definition) return d reactor.callWhenRunning(test) reactor.run() ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] pb.Copyable knowledge barrier
The question is what's necessary to get knowitall.py to receive updates to held copy. I start server, start knowitall, run wordsmith. Knowitall doesn't receive updates. I understand why, setCopyableState only gets fired once. Guessing jelly doesn't hold onto the RemoteCopy or a reference to RemoteCopy. Is the next step to add pb.Referenceable to Dictionary class, and use observe_? On Fri, Aug 8, 2014 at 12:30 AM, Daniel Sank wrote: > Kevin, > > Is there a question there? I'd be happy to try to help if I knew what you > were asking. > > > On Thu, Aug 7, 2014 at 11:42 PM, Kevin Mcintyre wrote: > >> wordsmith argument order issues re-take. >> >> Cheers, >> Kevin >> >> >> On Thu, Aug 7, 2014 at 11:29 PM, Kevin Mcintyre >> wrote: >> >>> goal: Give knowitall.py awareness of wordsmith.py addition. >>> >>> to run >>> ./library.sh # opens 7999 for remote_dictionary and remote_define methods >>> then both: >>> python knowitall.py >>> python wordsmith.py >>> >>> library.sh due to main/jelly issues >>> >>> >> >> ___ >> Twisted-Python mailing list >> Twisted-Python@twistedmatrix.com >> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >> >> > > > -- > Daniel Sank > Department of Physics > Broida Hall > University of California > Santa Barbara, CA 93117 > (805)893-3899 > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] pb.Copyable knowledge barrier
I think so too, but I'm starting to lean toward websockets. The warnings alone right? But I will take another pass and report back any success. Cheers, On Fri, Aug 8, 2014 at 9:04 AM, Daniel Sank wrote: > I think Cacheable exists to handle the case you want. Beware there are > some funny issues with Cacheable > > https://twistedmatrix.com/trac/ticket/7274 > > Daniel > > > On Fri, Aug 8, 2014 at 8:01 AM, Kevin Mcintyre wrote: > >> The question is what's necessary to get knowitall.py to receive updates >> to held copy. >> >> I start server, start knowitall, run wordsmith. Knowitall doesn't >> receive updates. I understand why, setCopyableState only gets fired once. >> Guessing jelly doesn't hold onto the RemoteCopy or a reference to >> RemoteCopy. >> >> Is the next step to add pb.Referenceable to Dictionary class, and use >> observe_? >> >> >> >> On Fri, Aug 8, 2014 at 12:30 AM, Daniel Sank >> wrote: >> >>> Kevin, >>> >>> Is there a question there? I'd be happy to try to help if I knew what >>> you were asking. >>> >>> >>> On Thu, Aug 7, 2014 at 11:42 PM, Kevin Mcintyre >>> wrote: >>> >>>> wordsmith argument order issues re-take. >>>> >>>> Cheers, >>>> Kevin >>>> >>>> >>>> On Thu, Aug 7, 2014 at 11:29 PM, Kevin Mcintyre >>>> wrote: >>>> >>>>> goal: Give knowitall.py awareness of wordsmith.py addition. >>>>> >>>>> to run >>>>> ./library.sh # opens 7999 for remote_dictionary and remote_define >>>>> methods >>>>> then both: >>>>> python knowitall.py >>>>> python wordsmith.py >>>>> >>>>> library.sh due to main/jelly issues >>>>> >>>>> >>>> >>>> ___ >>>> Twisted-Python mailing list >>>> Twisted-Python@twistedmatrix.com >>>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >>>> >>>> >>> >>> >>> -- >>> Daniel Sank >>> Department of Physics >>> Broida Hall >>> University of California >>> Santa Barbara, CA 93117 >>> (805)893-3899 >>> >>> ___ >>> Twisted-Python mailing list >>> Twisted-Python@twistedmatrix.com >>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >>> >>> >> >> ___ >> Twisted-Python mailing list >> Twisted-Python@twistedmatrix.com >> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >> >> > > > -- > Daniel Sank > Department of Physics > Broida Hall > University of California > Santa Barbara, CA 93117 > (805)893-3899 > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] pb.Copyable knowledge barrier
I experimented today with a table scan from dynamodb. Taking each record and converting to a dictionary into a list. Initial scan and transform took ~8 seconds from amazon to my house over TCP boto which is soapish? i think. ~7800 records. The next call for the remote copy took 7 seconds across the bus. Is that expected? On Fri, Aug 8, 2014 at 3:01 PM, Glyph Lefkowitz wrote: > > On Aug 8, 2014, at 1:31 PM, Kevin Mcintyre wrote: > > I think so too, but I'm starting to lean toward websockets. The warnings > alone right? > > > Websockets are a layer well below what PB does. In fact you could run PB > over websockets, and it would work roughly the same as over TCP. > > But I will take another pass and report back any success. > > > Please do let us know. > > -glyph > > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] pb.Copyable knowledge barrier
correction. I was stupidly reprocessing list on remote copy side. Seeing much better results now ~10K per second. On Fri, Aug 8, 2014 at 6:32 PM, Kevin Mcintyre wrote: > I experimented today with a table scan from dynamodb. Taking each record > and converting to a dictionary into a list. Initial scan and transform > took ~8 seconds from amazon to my house over TCP boto which is soapish? i > think. ~7800 records. > > The next call for the remote copy took 7 seconds across the bus. Is that > expected? > > > > > > On Fri, Aug 8, 2014 at 3:01 PM, Glyph Lefkowitz > wrote: > >> >> On Aug 8, 2014, at 1:31 PM, Kevin Mcintyre wrote: >> >> I think so too, but I'm starting to lean toward websockets. The warnings >> alone right? >> >> >> Websockets are a layer well below what PB does. In fact you could run PB >> over websockets, and it would work roughly the same as over TCP. >> >> But I will take another pass and report back any success. >> >> >> Please do let us know. >> >> -glyph >> >> >> ___ >> Twisted-Python mailing list >> Twisted-Python@twistedmatrix.com >> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >> >> > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] pb.Copyable knowledge barrier
Jelly performance factors below expectations. Can we say Copyable is the lowest order jelly? The notion that a copy holder can't ask "is my copy good anymore?" makes it so. Essentially root says, I'd prefer not to repeat unit of work nor keep track of the resulting copies, here have the original or resulting copy. My main issue is a copy-holder calling for a copy to determine is the copy is good anymore. I know, see cacheable but it's problematic. At this point I'm unsure of what jelly actually does well. who's the JellyCon2014 keynote speaker? On Sat, Aug 9, 2014 at 6:58 PM, Glyph wrote: > On Aug 8, 2014, at 9:17 PM, Kevin Mcintyre wrote: > > correction. I was stupidly reprocessing list on remote copy side. Seeing > much better results now ~10K per second. > > > Whew. I don't know exactly how well I would expect this perform, but > those other results looked off by a couple orders of magnitude :-). > > -glyph > > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] pb.Copyable knowledge barrier
Jelly performance factors below expectations. I am not clear on the significance of this statement. Not significant, just an observation. I've made this statement before, backed-off after finding errors on my code, but after fixing I see performance across the bus that still seem slow. "Seem slow" lacks any scientific backing. I would be curious to hear use cases for PB/Jelly that went beyond the docs. Can we say Copyable is the lowest order jelly? The notion that a copy holder can't ask "is my copy good anymore?" makes it so. Essentially root says, I'd prefer not to repeat unit of work nor keep track of the resulting copies, here have the original or resulting copy. Asking whether your copy is good any more is a PB-level task. Jelly itself is a separate layer which is about getting the right data to the right place, not keeping it updated. Yes PB level, I guess I'm looking for a convention where 1 does not exist. In the renewed interest the comments have tended to overlook copyable, or see copyable as being flushed out. My main issue is a copy-holder calling for a copy to determine is the copy is good anymore. I know, see cacheable but it's problematic. "problematic" is passive voice :-). What are the problems? Only parroting what I've read on cacheable, haven't delved into yet. At this point I'm unsure of what jelly actually does well. Malheureusement, Jelly's main claim to fame is "it's better than pickle". But with all this renewed interest in PB perhaps we'll get pre-deserialization schema enforcement and type checking, and then it will have some real advantages :). Agreed. This is what I looked to Jelly/PB for, having found so much in twisted that simplifies. who's the JellyCon2014 keynote speaker? *One* day we will have a conference - Tx/Rx, of course - and there will be a talk on this. But first we need several people in the community who wants their contribution to Twisted to be organization and community oriented rather than code ;-). On Sun, Sep 7, 2014 at 9:43 AM, Glyph wrote: > > On Sep 7, 2014, at 12:51 AM, Kevin Mcintyre wrote: > > Jelly performance factors below expectations. > > > I am not clear on the significance of this statement. > > Can we say Copyable is the lowest order jelly? The notion that a copy > holder can't ask "is my copy good anymore?" makes it so. Essentially root > says, I'd prefer not to repeat unit of work nor keep track of the resulting > copies, here have the original or resulting copy. > > > Asking whether your copy is good any more is a PB-level task. Jelly > itself is a separate layer which is about getting the right data to the > right place, not keeping it updated. > > My main issue is a copy-holder calling for a copy to determine is the copy > is good anymore. I know, see cacheable but it's problematic. > > > "problematic" is passive voice :-). What are the problems? > > At this point I'm unsure of what jelly actually does well. > > > Malheureusement, Jelly's main claim to fame is "it's better than pickle". > But with all this renewed interest in PB perhaps we'll get > pre-deserialization schema enforcement and type checking, and then it will > have some real advantages :). > > who's the JellyCon2014 keynote speaker? > > > *One* day we will have a conference - Tx/Rx, of course - and there will > be a talk on this. But first we need several people in the community who > wants their contribution to Twisted to be organization and community > oriented rather than code ;-). > > -glyph > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] not terrible object oriented rpc system in js?
flood with 50 permutations? On Sat, Dec 27, 2014 at 4:43 PM, Clayton Daley wrote: > Not sure which features of PB matter to you, but I just started playing > with Autobahn/Crossbar. It's > >- A JSON-based RPC (and/or pubsub) system, >- With clients in many languages (JS, Python, C++, etc.), and >- The ability to make cross-language RPC calls > > Out of the box, object serialization is limited (likely to ensure > cross-platform compatibility), but the serializer is (purportedly) > pluggable. > > > Clayton Daley > clayton.da...@gmail.com > 513.505.1236 > > On Sat, Dec 27, 2014 at 3:06 PM, Daniel Sank > wrote: > >> This looks promising. Thanks. >> >> On Sat, Dec 27, 2014 at 1:23 PM, Jens Sauer wrote: >> >>> Hi Daniel, >>> >>> whats about node.js with http://uber.github.io/multitransport-jsonrpc/ >>> >>> Regards >>> >>> J.Sauer >>> >>> -- >>> *Von:* Daniel Sank >>> *An:* twisted-python >>> *Gesendet:* 16:30 Samstag, 27.Dezember 2014 >>> *Betreff:* [Twisted-Python] not terrible object oriented rpc system in >>> js? >>> >>> Are you guys aware of anything which solves the same kind of problem as >>> t.s.pb but in js? I asked about this on the js stack exchange chat room and >>> the general response was that there's isn't anything. I figured y'all might >>> know more. >>> >>> -- >>> Daniel Sank >>> >>> >>> >>> ___ >>> Twisted-Python mailing list >>> Twisted-Python@twistedmatrix.com >>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >>> >>> >>> >>> ___ >>> Twisted-Python mailing list >>> Twisted-Python@twistedmatrix.com >>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >>> >>> >> >> >> -- >> Daniel Sank >> >> >> >> ___ >> Twisted-Python mailing list >> Twisted-Python@twistedmatrix.com >> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >> >> > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python