[Twisted-Python] can QtWebKit fire a twisted deferred?

2015-05-06 Thread Kevin Mcintyre
'''

I know this is meant for pyqt but I've had little luck this that board.


The short answer is yes, but it's a total hack...and I'm wondering if
anybody has a more elegant solution.


Specifically I would like to be able to fire a defer from the javascript
instead of having to call alert.


Below is an example of how I currently gain access...not necessarily a
working version.

'''


url = 'http://www.google.com'


mo = """

var observer = new MutationObserver(function(mutations) {

for (var x = 0; x < mutations.length; x++) {

if (mutations[x].addedNodes.length > 0) {

alert('add nodes')

}

}

});

var config = { attributes: true, childList: true, characterData: false }

var nti = document.querySelector('document.body')

observer.observe(nti, config);

"""


page = QWebPage()

def alarmAction(qframe, msg):

print 'deferred action', qframe, msg

def mutationHook():

d = defer.Deferred()

d.addCallback(alarmAction)

page.javaScriptAlert = d.callback

page.mainFrame().documentElement().evaluateJavaScript(mo)



page.loadFinished(mutationHook)

page.mainFrame().setUrl(url)
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


Re: [Twisted-Python] can QtWebKit fire a twisted deferred?

2015-05-06 Thread Kevin Mcintyre
That was pseudo code, I've been using qt5reactor to happy results.  All the
same good to know on deferreds working in non-twisted code...

My challenge is having only 2 hooks into the DOM - javaScriptAlert and
javaScriptConfirm that ultimately with QtWebEngine aren't available.

thank for the feedback.

On Wed, May 6, 2015 at 3:35 PM, Glyph  wrote:

>
> On May 6, 2015, at 11:04 AM, Kevin Mcintyre  wrote:
>
> '''
> I know this is meant for pyqt but I've had little luck this that board.
>
> The short answer is yes, but it's a total hack...and I'm wondering if
> anybody has a more elegant solution.
>
> Specifically I would like to be able to fire a defer from the javascript
> instead of having to call alert.
>
> Below is an example of how I currently gain access...not necessarily a
> working version.
> '''
>
>
> I don't see any import of qt4reactor or Twisted (other than the implied
> import of Deferred) in this code.  Firing a Deferred in this way should be
> totally fine though; you don't actually need the reactor for Deferreds to
> work (it's just important that if it's going to interact with any *other* 
> Twisted
> code it needs to run on the main thread).
>
> Sorry I don't have a better suggestion, but I don't see much wrong with
> what you're doing here :).
>
> -g
>
> url = 'http://www.google.com'
>
> mo = """
> var observer = new MutationObserver(function(mutations) {
> for (var x = 0; x < mutations.length; x++) {
> if (mutations[x].addedNodes.length > 0) {
> alert('add nodes')
> }
> }
> });
> var config = { attributes: true, childList: true, characterData:
> false }
> var nti = document.querySelector('document.body')
> observer.observe(nti, config);
> """
>
> page = QWebPage()
> def alarmAction(qframe, msg):
> print 'deferred action', qframe, msg
> def mutationHook():
> d = defer.Deferred()
> d.addCallback(alarmAction)
> page.javaScriptAlert = d.callback
> page.mainFrame().documentElement().evaluateJavaScript(mo)
>
>
> page.loadFinished(mutationHook)
> page.mainFrame().setUrl(url)
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


[Twisted-Python] Dumb Logger API question

2015-11-09 Thread Kevin Mcintyre
http://twistedmatrix.com/documents/current/core/howto/logger.html

I'm looking at the new-ish logger api and wonder how to log to console as a
first step to transitioning.

Any pointers?  Thanks.
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


Re: [Twisted-Python] Dumb Logger API question

2015-11-10 Thread Kevin Mcintyre
yes, I stared at that section for a bit.  Not to get off topic, but the
readability of the api docs from back when is slightly better in my
opinion.  So the topic being I'm an idiot - thanks for both
examples...that's a perfect example.

I got confused by "from myobservers import PrintingObserver" cause I'm
wondering if myobservers is a core package.

On Tue, Nov 10, 2015 at 6:05 PM, Glyph Lefkowitz 
wrote:

>
> On Nov 9, 2015, at 2:29 PM, Kevin Mcintyre  wrote:
>
> http://twistedmatrix.com/documents/current/core/howto/logger.html
>
> I'm looking at the new-ish logger api and wonder how to log to console as
> a first step to transitioning.
>
> Any pointers?  Thanks.
>
>
> Do you mean "emit structured log text to stdout"?  In that case, that is
> covered in the document you link to; these two sections:
>
>
> http://twistedmatrix.com/documents/current/core/howto/logger.html#avoid-mutable-event-keys
>
> http://twistedmatrix.com/documents/current/core/howto/logger.html#starting-the-global-log-publisher
>
> create a jsonFileLogObserver and start logging to it:
>
> import sys
> from twisted.logger import jsonFileLogObserver, globalLogBeginner, Logger
> globalLogBeginner.beginLoggingTo([jsonFileLogObserver(sys.stdout)])
> log = Logger()
> log.info("Information.")
>
>
> If you want unstructured log output for human reading (not a good choice
> for an automated system, but perhaps useful for debugging), you can instead
> go with textFileLogObserver, in almost exactly the same way:
>
> import sys
> from twisted.logger import textFileLogObserver, globalLogBeginner, Logger
> globalLogBeginner.beginLoggingTo([textFileLogObserver(sys.stdout)])
> log = Logger()
> log.info("Information.")
>
>
> Is this what you were looking for?
>
> -glyph
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


[Twisted-Python] asynchronous response

2015-12-16 Thread Kevin Mcintyre
Hey - I'm confused, so nothing new :) ...but I'm running at this example
and I'm scratching my head.

http://twistedmatrix.com/documents/current/web/howto/web-in-60/asynchronous-deferred.html

I would've thought 2 requests could be served simultaneously, but when I
fire off 2 requests the first received gets it's response after 5 seconds,
while the second response takes 10 seconds.

I think I understand what's happening, but I don't know why...and I would
love an example where the subsequent request doesn't have to wait for the
first request to finish.

Thanks,
Kevin
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


Re: [Twisted-Python] asynchronous response

2015-12-16 Thread Kevin Mcintyre
Just a follow-up with my own sanity check example.

http://twistedmatrix.com/documents/current/web/howto/web-in-60/asynchronous-deferred.html

My conclusion is that deferLater is a blocking method - am I wrong?


On Wed, Dec 16, 2015 at 4:11 PM, Kevin Mcintyre  wrote:

> Hey - I'm confused, so nothing new :) ...but I'm running at this example
> and I'm scratching my head.
>
>
> http://twistedmatrix.com/documents/current/web/howto/web-in-60/asynchronous-deferred.html
>
> I would've thought 2 requests could be served simultaneously, but when I
> fire off 2 requests the first received gets it's response after 5 seconds,
> while the second response takes 10 seconds.
>
> I think I understand what's happening, but I don't know why...and I would
> love an example where the subsequent request doesn't have to wait for the
> first request to finish.
>
> Thanks,
> Kevin
>
>
>
>
>
>
from twisted.internet.task import deferLater
from twisted.web.resource import Resource
from twisted.web.server import NOT_DONE_YET, Site
from twisted.internet import reactor, defer, threads
from twisted.web.xmlrpc import Proxy
from twisted.web.xmlrpc import XMLRPC
from twisted.web.client import getPage

import time
import xmlrpclib

class TestResource(Resource):

isLeaf = True

def render_ERROR(self, err, request):
print 'xmlrpc error:', self.__class__.__name__, err
request.finish()

def render_TIMESTAMP(self, ts, request):
request.write(str(ts))
request.finish()

def render_GET(self, request):
print 'test:', int(time.time())
d = Proxy('http://localhost:8082').callRemote('timestamp')
d.addCallback(self.render_TIMESTAMP, request)
d.addErrback(self.render_ERROR, request)
return NOT_DONE_YET

class Test1Resource(TestResource):
 
isLeaf = True

def render_GET(self, request):
print 'test1:', int(time.time())
d = threads.deferToThread(xmlrpclib.ServerProxy("http://localhost:8082";).timestamp)
d.addCallback(self.render_TIMESTAMP, request)
d.addErrback(self.render_ERROR, request)
return NOT_DONE_YET

class Test2Resource(XMLRPC):

def delayed_html(self):
time.sleep(5)
return int(time.time())

def xmlrpc_timestamp(self):
print 'timestamp:', int(time.time())
'''
blocking!!
return self.delayed_html()
''' 
return threads.deferToThread(self.delayed_html)
 
def done_or_error(ans = None):
print 'done_or_error:', ans
reactor.stop()

def results(res, init_time):
print init_time, res[0][1], res[1][1], res[2][1], res[3][1], int(time.time()) 
 
def quad_request():
init_time = int(time.time())
dl = defer.DeferredList([getPage('http://localhost:8080'), getPage('http://localhost:8080'), getPage('http://localhost:8081'), getPage('http://localhost:8081')])
dl.addCallback(results, init_time)
dl.addBoth(done_or_error)
 
if __name__ == '__main__':
print 'start reactor'
reactor.listenTCP(8080, Site(TestResource()))
reactor.listenTCP(8081, Site(Test1Resource()))
reactor.listenTCP(8082, Site(Test2Resource()))
reactor.callWhenRunning(quad_request)
reactor.run()___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


Re: [Twisted-Python] asynchronous response

2015-12-17 Thread Kevin Mcintyre
holy moly - you're right!  This fool will sleep so much better...a million
thanks.

On Thu, Dec 17, 2015 at 12:09 AM, Amber "Hawkie" Brown <
hawk...@atleastfornow.net> wrote:

>
> > On 17 Dec 2015, at 08:11, Kevin Mcintyre  wrote:
> >
> > Hey - I'm confused, so nothing new :) ...but I'm running at this example
> and I'm scratching my head.
> >
> >
> http://twistedmatrix.com/documents/current/web/howto/web-in-60/asynchronous-deferred.html
> >
> > I would've thought 2 requests could be served simultaneously, but when I
> fire off 2 requests the first received gets it's response after 5 seconds,
> while the second response takes 10 seconds.
> >
> > I think I understand what's happening, but I don't know why...and I
> would love an example where the subsequent request doesn't have to wait for
> the first request to finish.
> >
> > Thanks,
> > Kevin
>
>
> I've ran into this before --  browsers sometimes rate-limit requests, and
> won't actually send the second request until the first is done, over the
> same connection, rather than making a second TCP connection -- try using
> cURL or wget, which has no such limitation, and see if it works any better.
>
> - Amber
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
from twisted.internet.task import deferLater
from twisted.web.resource import Resource
from twisted.web.server import NOT_DONE_YET, Site
from twisted.internet import reactor, defer
from twisted.web.client import getPage
import time

class DelayedResource(Resource):

isLeaf = True

def _delayedRender(self, request):
request.write(str(int(time.time(
request.finish()

def render_GET(self, request):
d = deferLater(reactor, 5, lambda: request)
d.addCallback(self._delayedRender)
return NOT_DONE_YET

def results(res, init_time):
print init_time, res[0][1], res[1][1], int(time.time())

def done_or_error(ans = None):
print 'done_or_error:', ans
reactor.stop() 

def double_request():
init_time = int(time.time())
dl = defer.DeferredList([getPage('http://localhost:8080'), getPage('http://localhost:8080')])
dl.addCallback(results, init_time)
dl.addBoth(done_or_error)

if __name__ == '__main__':
reactor.listenTCP(8080, Site(DelayedResource()))
reactor.callWhenRunning(double_request)
print 'start reactor'
reactor.run()___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


[Twisted-Python] on template processing

2015-12-26 Thread Kevin Mcintyre
1) Is template block processing done top to bottom sequentially?  I've
included example.py - it's interesting that by having a blocking call first
the second request is blocked, but having the blocking call after a
non-blocking call the second request is not.  I would've thought it would
behave as a deferred list, but looking at _flattenTree and guessing not.
Maybe related to wait_for_it example?

2) Is it possible for a Resource to act as an Element too?  I've included a
non-working elementresource.py.  I'm a total hack, but I would think that
if an instance had a loader attr it could be processable.

3) Is it possible to include xmlns:t="
http://twistedmatrix.com/ns/twisted.web.template/0.1"; not in a tag itself?
Or perhaps have a tag like 'render-block' that could be transparent-like?

4) Is it possible to have xmlns:t="
http://twistedmatrix.com/ns/twisted.web.template/0.1"; in multiple places in
a template?  I have some cases with inline scripts that bonk out with > or
< characters.  This is very much related to #3.

5) Is it possible for a render element to return something like "<~sometag
t:render>..." and process recursively?

6) Is there any examples of connection keep-alive long polling?

7) Examples of request based scoping would be great.  All the examples on
http://twistedmatrix.com/documents/13.0.0/web/howto/twisted-templates.html
have flatten(None... -  I've included request_scope.py

8) The wait_for_it example, is that meant as a chunked transfer example?
It would be cool to have an example I could open in a browser.  Trying to
wrap my head around this and subviews in the meantime.

Happy holidays.
from twisted.web import server, resource, template, client
from twisted.internet import reactor, defer, threads

import time
import pprint

blocking_first = """http://twistedmatrix.com/ns/twisted.web.template/0.1";> Blocking:Non-blocking:"""
blocking_second = """http://twistedmatrix.com/ns/twisted.web.template/0.1";> Non-blocking:Blocking:"""

class ExampleTemplate(template.Element):

loader = template.XMLString(blocking_second)

def non_blocking_call(self):
print 'called non-blocking'
d = defer.Deferred()
d.addCallback(lambda ign: str(int(time.time(
reactor.callLater(2, d.callback, None)
return d

def blocking_call(self):
print 'called blocking'
time.sleep(2)
return str(int(time.time()))

@template.renderer
def blocking(self, request, tag):
'''
switch to 
yield threads.deferToThread(self.blocking_call)
to unblock
'''
return self.blocking_call()

@template.renderer
def non_blocking(self, request, tag):
yield self.non_blocking_call()

class ExampleResource(resource.Resource):

isLeaf = True 

def render_GET(self, request):   
print 'get:', int(time.time()) 
d = template.flattenString(request, ExampleTemplate())  
d.addCallback(request.write)
d.addCallback(lambda ign: request.finish())
return server.NOT_DONE_YET

site = server.Site(ExampleResource())

def simulataneous():
dl = defer.DeferredList([client.getPage('http://localhost:6789'), client.getPage('http://localhost:6789')])
dl.addCallback(lambda res: [pprint.pprint(ans[1]) for ans in res ])
dl.addErrback(lambda err: pprint.pprint(err))

if __name__ == '__main__':
reactor.listenTCP(6789, site)
reactor.callWhenRunning(simulataneous)
reactor.run()from twisted.web import server, resource, template, client
from twisted.internet import reactor

import pprint

class ElementResource(resource.Resource, template.Element):

isLeaf = True
loader = template.XMLString("""http://twistedmatrix.com/ns/twisted.web.template/0.1";>""")

@template.renderer
def whatever(self, request, tag):
return 'whatever' 

def render_GET(self, request):
d = template.flattenString(request, self)  
d.addCallback(request.write)
d.addCallback(lambda ign: request.finish())
return server.NOT_DONE_YET


site = server.Site(ElementResource())

def test():
d = client.getPage('http://localhost:6789')
d.addCallback(lambda res: pprint.pprint(res))
d.addErrback(lambda err: pprint.pprint(err.getErrorMessage()))

if __name__ == '__main__':
reactor.listenTCP(6789, site)
reactor.callWhenRunning(test)
reactor.run()from twisted.web import server, resource, template, client
from twisted.internet import reactor, defer

import pprint

class ExampleTemplate(template.Element):

loader = template.XMLString("""http://twistedmatrix.com/ns/twisted.web.template/0.1";>result:""")

@template.renderer
def result(self, request, tag):
return request.result

class ExampleResource(resource.Resource):

inventory = { 'item' : 1 }
isLeaf = True

def shop(self, act, item, request):
  

[Twisted-Python] web Resource question

2016-06-15 Thread Kevin Mcintyre
what is child.server = self.server in Resource.putChild?

shouldn't it be child.server = self?
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


Re: [Twisted-Python] web Resource question

2016-06-16 Thread Kevin Mcintyre
Hey, thanks for the reply.

It seems like there's the possibility of more beneficial assignments...but
first trying to understand if there's a specific reason for the
child.server assignment.

On Thu, Jun 16, 2016 at 5:11 AM, Cory Benfield  wrote:

>
> > On 16 Jun 2016, at 05:16, Kevin Mcintyre  wrote:
> >
> >
> > what is child.server = self.server in Resource.putChild?
> >
> > shouldn't it be child.server = self?
>
> I don’t think so. The reference to the server is presumably intended to
> refer to the Site object that is hosting the Resource. With that said, I
> can’t actually *find* anywhere that sets the server on a Resource, so it’s
> possible that the whole thing is vestigial.
>
> Cory
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


[Twisted-Python] web client FileBodyProducer - transfer encoding

2017-01-30 Thread Kevin Mcintyre
hey all - quick question.  Trying to understand FileBodyProducer as it
pertains to POST.

json_body = FileBodyProducer(StringIO(json.dumps({'key': 'value'})))

agent.request("POST", ~*uri*, Headers({'User-Agent': ['AkamaiTest']}),
json_body)

Does the agent chunk the POST body?  Is this controllable?

Any insight appreciated!
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


Re: [Twisted-Python] web client FileBodyProducer - transfer encoding

2017-01-30 Thread Kevin Mcintyre
Dealing with older apache, ruby, passenger setup.  Hoping to mimic behavior
and show OPS that chunked encoding isn't working correctly.

With python requests (http://docs.python-requests.org/en/master/) all POST
calls work correctly, but twisted requests are failing.

I'm thinking (and hoping) it's failing because twisted is chunking the post
body.  When I send the requests to a twisted endpoint like below the
requests are successful and the response is identical.

from twisted.web import server, resource
from twisted.internet import reactor

class Simple(resource.Resource):
isLeaf = True
def render_POST(self, request):
print request.content.getvalue()
return request.content.getvalue()

site = server.Site(Simple())
reactor.listenTCP(8080, site)
reactor.run()


On Mon, Jan 30, 2017 at 2:39 PM, Glyph Lefkowitz 
wrote:

>
> On Jan 30, 2017, at 13:47, Kevin Mcintyre  wrote:
>
> hey all - quick question.  Trying to understand FileBodyProducer as it
> pertains to POST.
>
> json_body = FileBodyProducer(StringIO(json.dumps({'key': 'value'})))
>
> agent.request("POST", ~*uri*, Headers({'User-Agent': ['AkamaiTest']}), 
> json_body)
>
> Does the agent chunk the POST body?
>
> Maybe!
>
> Is this controllable?
>
> Not really.  The current implementation will certainly use chunked
> encoding sometimes, but there isn't a strong API guarantee of this anywhere.
>
> Any insight appreciated!
>
> You probably don't want to try to control this.  I believe that proxies
> are within their rights to mess around with chunk boundaries and re-buffer
> things, so you don't have any strong guarantees that chunk sizes will be
> preserved.
>
> Why is it that you want to control chunking in the first place?
>
> -glyph
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


Re: [Twisted-Python] web client FileBodyProducer - transfer encoding

2017-01-30 Thread Kevin Mcintyre
Thanks - that's exactly what I was looking for.

*But note* when using FileBodyProducer(StringIO(json.dumps(~blah))) -- I
still see a content-length in the request header as it's received by
twisted.  Am I correct to assume the request is agnostic...meaning it's
shaped the same for twisted as it is for apache.

Headers({'host': ['localhost:8080'], 'connection': ['close'],
'content-length': ['671'], 'user-agent': ['PassengerTest']})

from twisted.web import server, resource
from twisted.internet import reactor

class Simple(resource.Resource):
isLeaf = True
def render_POST(self, request):
print request.requestHeaders
return request.content.getvalue()

site = server.Site(Simple())
reactor.listenTCP(8080, site)
reactor.run()



On Mon, Jan 30, 2017 at 4:59 PM, Jean-Paul Calderone <
exar...@twistedmatrix.com> wrote:

> On Mon, Jan 30, 2017 at 7:44 PM, Glyph Lefkowitz 
> wrote:
>
>> Gotcha.  I guess what I meant was that you shouldn't care about this at
>> the application level, but you're talking about an operational concern, not
>> an application-level concern.
>>
>> Perhaps this should be a tunable on Agent somehow.  Can you file a ticket?
>>
>
>
> Well... It *is* sort of tuneable, as you implied earlier.
>
> If you pass an IBodyProducer with a non-None length, Agent will send a
> Content-Length header - not use chunked Transfer-Encoding.
> FileBodyProducer doesn't know how to determine the length of a StringIO, so
> you get chunked with this example.  If you write the JSON to a regular file
> and FileBodyProducer(open(the file)) you'll get a Content-Length request.
> You could also write a new (trivial) IBodyProducer that does know how to
> compute the length of a StringIO.
>
> The documentation doesn't exactly spell this out - but the only reason
> `length` is part of the interface is to be able to generate the
> Content-Length header.
>
> What would the ticket be?  Expanded documentation to make this behavior
> into an explicit guarantee of the interface?  A new toggle somewhere to
> force Agent.request into one mode or the other (regardless of the
> performance consequences)?
>
> And that stuff you said about proxies before was true ... So even if you
> can control this in Agent, you're still not *guaranteed* the server will
> see what you send.
>
> Jean-Paul
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


Re: [Twisted-Python] web client FileBodyProducer - transfer encoding

2017-01-30 Thread Kevin Mcintyre
"It is shaped the same.  The reason you're seeing the error is due to the
issue I pointed out above."

Just to be clear you mean that proxies can reshape the call in flight?



On Mon, Jan 30, 2017 at 6:01 PM, Glyph Lefkowitz 
wrote:

>
> On Jan 30, 2017, at 19:44, Kevin Mcintyre  wrote:
>
> On Mon, Jan 30, 2017 at 4:59 PM, Jean-Paul Calderone <
> exar...@twistedmatrix.com> wrote:
>
>> On Mon, Jan 30, 2017 at 7:44 PM, Glyph Lefkowitz > > wrote:
>>
>>> Gotcha.  I guess what I meant was that you shouldn't care about this at
>>> the application level, but you're talking about an operational concern, not
>>> an application-level concern.
>>>
>>> Perhaps this should be a tunable on Agent somehow.  Can you file a
>>> ticket?
>>>
>>
>>
>> Well... It *is* sort of tuneable, as you implied earlier.
>>
>> If you pass an IBodyProducer with a non-None length, Agent will send a
>> Content-Length header - not use chunked Transfer-Encoding.
>> FileBodyProducer doesn't know how to determine the length of a StringIO, so
>> you get chunked with this example.
>>
>
> Not quite true: FileBodyProducer *does* know how to compute the length of
> a StringIO, via seek():
>
>
> >>> from twisted.web.client import FileBodyProducer
> >>> from io import BytesIO
> >>> from cStringIO import StringIO
> >>> FileBodyProducer(BytesIO(b"some bytes")).length
> 10L
> >>> FileBodyProducer(StringIO(b"some bytes")).length
> 10
>
>
> If you write the JSON to a regular file and FileBodyProducer(open(the
>> file)) you'll get a Content-Length request.  You could also write a new
>> (trivial) IBodyProducer that does know how to compute the length of a
>> StringIO.
>>
>> The documentation doesn't exactly spell this out - but the only reason
>> `length` is part of the interface is to be able to generate the
>> Content-Length header.
>>
>> What would the ticket be?  Expanded documentation to make this behavior
>> into an explicit guarantee of the interface?  A new toggle somewhere to
>> force Agent.request into one mode or the other (regardless of the
>> performance consequences)?
>>
>
> Rather than "regardless" of the consequences, perhaps just a maximum body
> size.  If we made it an explicit guarantee in Agent, perhaps the interface
> change would not be in '.request' (which, as a formal interface used both
> inside and outside of Twisted, is fairly fixed), but rather a new
> `NonChunkedBodyProducer` concrete class that would implement IBodyProducer
> in terms of another IBodyProducer which either does or doesn't have a
> `length`.  (Or a function that does same, always returning its argument if
> `length` is already set...)
>
> And that stuff you said about proxies before was true ... So even if you
>> can control this in Agent, you're still not *guaranteed* the server will
>> see what you send.
>>
>
> Thanks - that's exactly what I was looking for.
>
> *But note* when using FileBodyProducer(StringIO(json.dumps(~blah))) -- I
> still see a content-length in the request header as it's received by
> twisted.  Am I correct to assume the request is agnostic...meaning it's
> shaped the same for twisted as it is for apache.
>
>
> It is shaped the same.  The reason you're seeing the error is due to the
> issue I pointed out above.
>
> Headers({'host': ['localhost:8080'], 'connection': ['close'],
> 'content-length': ['671'], 'user-agent': ['PassengerTest']})
>
>
> -glyph
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


[Twisted-Python] catching twisted pyqt segmentation fault

2017-02-25 Thread Kevin Mcintyre
I have a long running twisted pyqt process which occasionally throws a
Received signal 11.

I believe it's in the underlying QtWebEngineView since the twisted process
continues to run.  Whatever the reason I would like to figure out a way to
catch this and stop the reactor either in process, or by another process
monitor.  Any suggestions either twisted or otherwise would be greatly
appreciated.

Below is a bit of the stack trace dumped to /var/log/syslog

0x7fd855447a8f
Feb 25 22:01:32 ip-10-0-0-116 python[1836]: 2   0x7fd855447c74
Feb 25 22:01:32 ip-10-0-0-116 python[1836]: 3   0x7fd8555d6a82
Feb 25 22:01:32 ip-10-0-0-116 python[1836]: 4   0x7fd8556ababd
Feb 25 22:01:32 ip-10-0-0-116 python[1836]: 5   0x7fd8556af2f6
Feb 25 22:01:32 ip-10-0-0-116 python[1836]: 6   0x7fd85586e52d
Feb 25 22:01:32 ip-10-0-0-116 python[1836]: 7   0x7fd85586f511
Feb 25 22:01:32 ip-10-0-0-116 python[1836]: 8   0x7fd85570dbc3
Feb 25 22:01:32 ip-10-0-0-116 python[1836]: 9   0x7fd85570dd92
Feb 25 22:01:32 ip-10-0-0-116 python[1836]: 10  0x7fd855516c0c
Feb 25 22:01:32 ip-10-0-0-116 python[1836]: 11  0x7fd85551dacd
Feb 25 22:01:32 ip-10-0-0-116 python[1836]: 12  0x7fd85596c5d3
Feb 25 22:01:32 ip-10-0-0-116 python[1836]: 13  0x35946920c1fb
Feb 25 22:01:32 ip-10-0-0-116 python[1836]: Received signal 11 SEGV_MAPERR
fbadbeef
Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #0 0x7fd856d5ca2e 
Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #1 0x7fd856d5cdeb 
Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #2 0x7fd8526d3390 
Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #3 0x7fd858404408 
Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #4 0x7fd855447a8f 
Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #5 0x7fd855447c74 
Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #6 0x7fd8555d6a82 
Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #7 0x7fd8556ababd 
Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #8 0x7fd8556af2f6 
Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #9 0x7fd85586e52d 
Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #10 0x7fd85586f511 
Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #11 0x7fd85570dbc3 
Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #12 0x7fd85570dd92 
Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #13 0x7fd855516c0c 
Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #14 0x7fd85551dacd 
Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #15 0x7fd85596c5d3 
Feb 25 22:01:32 ip-10-0-0-116 python[1836]: #16 0x35946920c1fb 
Feb 25 22:01:32 ip-10-0-0-116 python[1836]:   r8: 7fd85a2f38c0  r9:
0013 r10:  r11:
Feb 25 22:01:32 ip-10-0-0-116 python[1836]:  r12: 7fd858aa25c8 r13:
7fd858ab91ee r14: 7fd85a0c9848 r15:
Feb 25 22:01:32 ip-10-0-0-116 python[1836]:   di:   si:
0013  bp: 7fd858ab91ee  bx:
Feb 25 22:01:32 ip-10-0-0-116 python[1836]:   dx: 7fd851e1d770  ax:
fbadbeef  cx: 7fd851b4e6fd  sp:
Feb 25 22:01:32 ip-10-0-0-116 python[1836]:   ip: 7fd858404408 efl:
00010246 cgf: 0033 erf:
Feb 25 22:01:32 ip-10-0-0-116 python[1836]:  trp: 000e msk:
 cr2: fbadbeef
Feb 25 22:01:32 ip-10-0-0-116 python[1836]: [end of stack trace]
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


Re: [Twisted-Python] catching twisted pyqt segmentation fault

2017-02-26 Thread Kevin Mcintyre
Thanks glyph - really appreciate all the times you've answered my questions!

QtWebEngineView is running a subprocess.

/home/ubuntu/Qt5.8.0/5.8/gcc_64/libexec/QtWebEngineProcess --type=renderer
--disable-accelerated-video-decode --disable-gpu-memory-buffer-video-frames
--enable-threaded-compositing --use-gl=desktop --disable-webrtc-hw-encoding
--primordial-pipe-token=032B94B57A61DAE34BD0F90551F9ABB6 --lang=en-US
--enable-pinch --num-raster-threads=2 --content-image-texture-target=3553

QT can sometimes be helpful, but bug resolution can take a while.

Any suggestions outside of twisted for detecting segfaults?  I'm thinking
just to monitor /var/log/syslog





On Sat, Feb 25, 2017 at 11:59 PM, Glyph Lefkowitz 
wrote:

>
> > On Feb 25, 2017, at 4:41 PM, Kevin Mcintyre  wrote:
> >
> > I have a long running twisted pyqt process which occasionally throws a
> Received signal 11.
> >
> > I believe it's in the underlying QtWebEngineView since the twisted
> process continues to run.  Whatever the reason I would like to figure out a
> way to catch this and stop the reactor either in process, or by another
> process monitor.  Any suggestions either twisted or otherwise would be
> greatly appreciated.
>
> Presumably the QtWebEngineView is running a subprocess, then?  This is
> really a Qt question rather than a Twisted question, because the
> QtWebEngineView should be emitting some kind of signal when its rendering
> process crashes.  (If it doesn't I imagine you're just out of luck here and
> you should report a bug to the Qt developers...)
>
> -glyph
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


[Twisted-Python] fastcgi client

2018-12-04 Thread Kevin Mcintyre
Saw this...

https://twistedmatrix.com/documents/8.2.0/api/twisted.web2.channel.fastcgi.html

Was wondering if there's a fastcgi client out there for twisted?

Thanks,
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


[Twisted-Python] ProcessMonitor stderr stdout

2012-02-29 Thread Kevin McIntyre
Newbie question:

I'm calling addProcess and everything works well, but I'm trying to
understand how to capture/handle stderr stdout for that spawned process.

-- 

Kevin McIntyre | Software Engineer | *Green Charge Networks*

Cell 718-249-5058 | kmcint...@greenchargenet.com | www.greenchargenet.com
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


[Twisted-Python] DeprecationWarning and PotentialZombieWarning

2012-06-06 Thread Kevin McIntyre
I recently upgraded to 12.0.0 and was running my process in the shell,
when this error/warning was thrown.  Any idea what it means?

DeprecationWarning: twisted.internet.error.PotentialZombieWarning was
deprecated in Twisted 10.0.0: There is no longer any potential for
zombie process.
 o = getattr (m, sym)
DeprecationWarning: twisted.protocols.basic.COMMA was deprecated in
Twisted 10.2.0: NetstringReceiver parser state is private.
 o = getattr (m, sym)
DeprecationWarning: twisted.protocols.basic.DATA was deprecated in
Twisted 10.2.0: NetstringReceiver parser state is private.
 o = getattr (m, sym)
DeprecationWarning: twisted.protocols.basic.LENGTH was deprecated in
Twisted 10.2.0: NetstringReceiver parser state is private.
 o = getattr (m, sym)
DeprecationWarning: twisted.protocols.basic.NUMBER was deprecated in
Twisted 10.2.0: NetstringReceiver parser state is private.
 o = getattr (m, sym)
DeprecationWarning: twisted.internet.interfaces.IFinishableConsumer
was deprecated in Twisted 11.1.0: Please use IConsumer (and
IConsumer.unregisterProducer) instead.
 o = getattr (m, sym)
DeprecationWarning: twisted.internet.interfaces.IReactorArbitrary was
deprecated in Twisted 10.1.0: See IReactorFDSet.
 o = getattr (m, sym)

-- 

Kevin McIntyre | Software Engineer | Green Charge Networks

Cell 718-249-5058 | kmcint...@greenchargenet.com | www.greenchargenet.com

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


[Twisted-Python] protocol connectionLost clientConnectionLost

2012-06-07 Thread Kevin McIntyre
Below is my sample code.  I'm hoping to unplug my internet cable and
get a connectionLost.  But this doesn't happen.  Is there anyway to
detect this?

--

from twisted.internet.protocol import Protocol, ReconnectingClientFactory
from twisted.internet import reactor

class Echo(Protocol):

def connectionMade(self):
print "cool and the gang"

def connectionLost(self, reason):
print "ut-oh %s" % reason

def dataReceived(self, data):
pass

class EchoClientFactory(ReconnectingClientFactory):

def startedConnecting(self, connector):
print 'Started to connect.'

def buildProtocol(self, addr):
print 'Connected.'
print 'Resetting reconnection delay'
self.resetDelay()
return Echo()

def clientConnectionLost(self, connector, reason):
print 'Lost connection.  Reason:', reason
ReconnectingClientFactory.clientConnectionLost(self, connector, reason)

def clientConnectionFailed(self, connector, reason):
print 'Connection failed. Reason:', reason
ReconnectingClientFactory.clientConnectionFailed(self, connector,
 reason)
if ( __name__ == "__main__" ):
reactor.connectTCP("74.125.224.242", 80, EchoClientFactory() )
#connect to google on port 80
reactor.run()

-- 

Kevin McIntyre | Software Engineer | Green Charge Networks

Cell 718-249-5058 | kmcint...@greenchargenet.com | www.greenchargenet.com

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


[Twisted-Python] more complex IBodyProducer?

2013-03-18 Thread Kevin McIntyre
has anyone implemented a more complex IBodyProducer than

class POSTRequestProducer():
implements(IBodyProducer)
def __init__(self, body):
self.body = body
self.length = len(self.body)

def startProducing(self, consumer):
consumer.write(self.body)
return succeed(None)

def pauseProducing(self):
pass

def stopProducing(self):
pass


I have an upload intensive application, that during periods of poor
connectivity ends up with a lots on zombie http TCP connections.  I'm
thinking maybe that the "pause" is being called and that my request
deferred is in a paused state.
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


[Twisted-Python] web templating with html

2013-10-11 Thread kevin mcintyre
https://twistedmatrix.com/documents/current/web/howto/twisted-templates.html#auto7

I see the example with quoting, but what if I want to render html?

I'm trying to get the text/html part of an email, and want to render as is.

Thanks in advance.
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


[Twisted-Python] maybe video

2014-02-25 Thread kevin mcintyre
Wanted to share my qt4reactor webkit cv2 video capture script.

It's comes in handy whenever I've needed to scrape just the browser and not
my whole screen or capture unit tests etc.

here is a demo quick demo in action:

http://www.youtube.com/watch?v=2XMXtUISY1g&feature=youtu.be

It's a hack, but that's why I like python (espeically twisted)...it's so
versatile.

https://github.com/kmcintyre/scewpt/blob/master/pyscewpt/maybevideo.py

Cheers,
Kevin
from PyQt4.QtGui import QApplication, QKeyEvent, QPainter, QImage
import cv2
import sys

print 'args:', sys.argv
video = None
if len(sys.argv) > 2:
print 'start video:', sys.argv[2]
video = cv2.VideoWriter( sys.argv[2] + '.avi',  cv2.cv.CV_FOURCC('M','J','P','G'), 24, (1024, 768),True)

from PyQt4.QtWebKit import QWebView, QWebPage, QWebSettings

class MaybeVideo(QApplication):

# you may need
def spoof_hit_enter(self, receiver):
print 'spoof'
super(MaybeVideo, self).notify(receiver, QKeyEvent(QEvent.KeyPress, Qt.Key_Enter, Qt.NoModifier)) 
super(MaybeVideo, self).notify(receiver, QKeyEvent(QEvent.KeyRelease, Qt.Key_Enter, Qt.NoModifier)) 

painter = QPainter()
events = []
ready = False

def notify(self, receiver, event):
if MaybeVideo.ready and isinstance(receiver, QWebView) and not MaybeVideo.painter.isActive() and video is not None:
try: 
#print 'clip screenshot'   
image = QImage(QSize(1024, 768), QImage.Format_RGB32)
MaybeVideo.painter.begin(image)
MaybeVideo.painter.setRenderHint(QPainter.Antialiasing, True)
MaybeVideo.painter.setRenderHint(QPainter.TextAntialiasing, True)
MaybeVideo.painter.setRenderHint(QPainter.SmoothPixmapTransform, True)
MaybeVideo.painter.setRenderHint(QPainter.HighQualityAntialiasing, True)
receiver.page().mainFrame().render(MaybeVideo.painter)
image.save("temp.jpg", "jpg")
video.write( cv2.imread('temp.jpg'))
MaybeVideo.painter.end()
except Exception as e:
print e
if event.type() == QEvent.SockAct:
MaybeVideo.ready = True
elif not event.type() in MaybeVideo.events:
#print 'new type', receiver, event.type(), QEvent.User
MaybeVideo.events.append(event.type())
else:
pass  
return super(MaybeVideo, self).notify(receiver, event)

app = MaybeVideo([])

import qt4reactor
qt4reactor.install()

from PyQt4.QtGui import QMainWindow
from PyQt4.QtCore import QEvent, Qt, QSize, QUrl

from twisted.internet import reactor

import signal

from PyQt4.QtNetwork import QNetworkAccessManager, QNetworkCookieJar

class MaybeVideoWindow(QMainWindow):

def __init__(self, parent=None):
QMainWindow.__init__(self, parent)
self.web_page = QWebPage()

print self.web_page.settings()

self.web_page.setNetworkAccessManager(QNetworkAccessManager())
self.web_page.networkAccessManager().setCookieJar(QNetworkCookieJar())
self.resize(QSize(1024, 768))

web_view = QWebView()
web_view.resize(QSize(1024, 768))
web_view.settings().setAttribute(QWebSettings.AutoLoadImages, True)
web_view.settings().setAttribute(QWebSettings.JavascriptEnabled, True)
web_view.settings().setAttribute(QWebSettings.JavaEnabled, False)
web_view.settings().setAttribute(QWebSettings.JavascriptCanOpenWindows, False)
web_view.settings().setAttribute(QWebSettings.PluginsEnabled, True)

self.web_page.setView(web_view)
self.setCentralWidget(web_view)
self.setWindowTitle('MaybeVideo')



self.web_page.view().urlChanged.connect(self._page_url_change)
self.web_page.loadStarted.connect(self._page_start)
self.web_page.loadProgress.connect(self._page_progress)
self.web_page.loadFinished.connect(self._page_finished)

def go(self, url):
#print 'go:', url
self.web_page.view().load(QUrl(url))

def _page_progress(self, percent):
#print 'page progress', percent
pass

def _page_start(self):
#print 'page start'
pass

def _page_finished(self, ok):
#print 'page finished:', str(ok)
pass

def _page_url_change(self, url):
print '_url_change:', str(url.toString())

mvw = MaybeVideoWindow()

def end(ign, ign2):
print 'stopping'
reactor.stop()

def start(): 
mvw.show()
if len(sys.argv) > 1:
mvw.go('http://' + sys.argv[1] )
else

[Twisted-Python] stmp client with multipart

2014-03-28 Thread Kevin Mcintyre
How can I get something similar into a smtp client 12? multipart message

fn = "example.mp3"
multipart = MIMEMultipart('alternative')
multipart['Subject'] = 'Tutorate!'
multipart['To'] = 'Selfie'
multipart['From'] = 'Selfie'

text = "Hello, how are you, goodbye."
textpart = MIMEText(text)
multipart.attach(textpart)
htmlpart = MIMEText("" + text + "", 'html')
multipart.attach(htmlpart)

part = MIMEBase('audio', "mp3")
part.set_payload( open(fn,"rb").read() )
Encoders.encode_base64(part)
part.add_header('Content-Disposition', 'attachment; filename="%s"' %
os.path.basename(fn))
multipart.attach(part)

io = StringIO.StringIO()
g = Generator(io, False)
g.flatten(multipart)
v = io.getvalue()

class SMTPTutorialClient(smtp.ESMTPClient):
mailFrom = "mg_selfie@ "
mailTo = "mg@ "
def getMailFrom(self):
result = self.mailFrom
self.mailFrom = None
return result

def getMailTo(self):
return [self.mailTo]

def getMailData(self):
print v
return StringIO.StringIO(v)

def sentMail(self, code, resp, numOk, addresses, log):
print 'Sent', numOk, 'messages'

from twisted.internet import reactor
reactor.stop()
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


Re: [Twisted-Python] stmp client with multipart

2014-03-28 Thread Kevin Mcintyre
ah, it just seems like the next smtp client (potentially 12) from the
samples could be multipart mime.


On Fri, Mar 28, 2014 at 3:12 PM,  wrote:

> On 08:36 pm, kebi...@gmail.com wrote:
>
>> How can I get something similar into a smtp client 12? multipart message
>>
>
> Hi Kevin,
>
> MIME and SMTP are at different layers.
>
> Twisted's SMTP client doesn't care what bytes you shove through it.
> They're just bytes.  They can be MIME or not.  Construct the bytes using
> the stdlib's MIME functionality if you want - once you have the bytes, the
> way you use them with Twisted's SMTP client is the same no matter what they
> are.
>
> Jean-Paul
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


Re: [Twisted-Python] Using twisted for a smtp dialog

2014-04-02 Thread Kevin Mcintyre
I've been working a lot with smtp and twisted is great for easily doing
what you're describing.

Have a look at the smtpclient docs...that should get you most of the way
there.

http://twistedmatrix.com/documents/current/mail/tutorial/smtpclient/smtpclient.html


On Wed, Apr 2, 2014 at 9:03 AM, Grigorescu Ciprian <
grigorescu_cipr...@yahoo.com> wrote:

> Hello,
>
> I am new in twisted world and I am struggle to understand it.
> I am trying to create a smtp connection and initiate a smtp dialog: helo,
> mail from, rcpt, for a list of address, and catch the answer from the smtp
> server
>
> Can anyone point me the right direction, where to start?
>
> Thanks a lot,
> Grig
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


[Twisted-Python] XMLString clarification

2014-04-08 Thread Kevin Mcintyre
I'm still having trouble getting my multipart text/html from an email into
XMLString.

I run the html through beautifulsoup then remove the doctype, but if
there's anything at all off about the resulting xml it still craps out.  Is
there anyway to stick an arbitrary string into a template?  Or should I add
a callback to flatten and add the placeholder in the flatten stage and
replace the placeholder with the actually html.

Any guidance appreciated,
Kevin
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


Re: [Twisted-Python] XMLString clarification

2014-04-08 Thread Kevin Mcintyre
I've opted for placeholder and added callback to flatten.

I'm a hack, and it was holding up delivery of email.


On Tue, Apr 8, 2014 at 4:19 PM, Werner Thie  wrote:

> Hi Kevin
>
>
> On 4/8/14 9:34 AM, Kevin Mcintyre wrote:
>
>> I'm still having trouble getting my multipart text/html from an email
>> into XMLString.
>>
>> I run the html through beautifulsoup then remove the doctype, but if
>> there's anything at all off about the resulting xml it still craps out.
>>
>
> How? Traceback?
>
>
>  Is there anyway to stick an arbitrary string into a template?  Or should
>> I add a callback to flatten and add the placeholder in the flatten stage
>> and replace the placeholder with the actually html.
>>
>
> Werner
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


Re: [Twisted-Python] XMLString clarification

2014-04-08 Thread Kevin Mcintyre
I will try to get a test class together to demonstrate.


On Tue, Apr 8, 2014 at 4:38 PM, Kevin Mcintyre  wrote:

> I've opted for placeholder and added callback to flatten.
>
> I'm a hack, and it was holding up delivery of email.
>
>
> On Tue, Apr 8, 2014 at 4:19 PM, Werner Thie wrote:
>
>> Hi Kevin
>>
>>
>> On 4/8/14 9:34 AM, Kevin Mcintyre wrote:
>>
>>> I'm still having trouble getting my multipart text/html from an email
>>> into XMLString.
>>>
>>> I run the html through beautifulsoup then remove the doctype, but if
>>> there's anything at all off about the resulting xml it still craps out.
>>>
>>
>> How? Traceback?
>>
>>
>>  Is there anyway to stick an arbitrary string into a template?  Or should
>>> I add a callback to flatten and add the placeholder in the flatten stage
>>> and replace the placeholder with the actually html.
>>>
>>
>> Werner
>>
>>
>> ___
>> Twisted-Python mailing list
>> Twisted-Python@twistedmatrix.com
>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


[Twisted-Python] dumb question on qt4reactor

2014-07-05 Thread Kevin Mcintyre
calling reactor.callLater within qt4reactor in an attempt to async non
webkit work but not seeing expected behavior.  Webkit locks while the
callLater function executes.

Any insight appreciated.

Should I move real work into separate reactor and "broker" calls into
qt4reactor/webkit.  I have xmlrpc and websocket  hooks into browser...but
looking for guidance as to most direct method of brokering calls between
reactors.

Any insight appreciated.

Having a blast using QT browser as a back-end, see demo site
www.athleets.com.

See below for amazon S3 upload worker that I expected to async, but in
qt4reactor does not appear to work as planned.

from pyscewpt import fixed
from pyscewpt.s3 import bucket_util
import boto
from boto.s3.bucket import Bucket
from boto.s3.key import Key

from twisted.internet import defer, reactor

bucket_queue = defer.DeferredQueue()

def upload_worker(data):
print 'macth upload_worker', data
save_match( data['league'], data['twitter'], data['local_file'],
data['meta'], data['profile'])
return len(bucket_queue.pending)
def match_queue(ign=None):
print 'match_queue:', len(bucket_queue.pending),
len(bucket_queue.waiting)
if ign is None or ign >= 2:
print 'requeue:', len(bucket_queue.pending) ,
len(bucket_queue.waiting)
d = bucket_queue.get()
d.addCallback(upload_worker)
d.addBoth(match_queue)
return d
else:
return None
def queue_save_match(queue_dic):
bucket_queue.put(queue_dic)
print 'queue_save_match', len(bucket_queue.pending),
len(bucket_queue.waiting), queue_dic['twitter']#
if len(bucket_queue.pending) >= 5 and len(bucket_queue.waiting) == 0:
print 'queue drop worker'
reactor.callLater(0,match_queue,None)

def save_match(league, twitter, systemfile, meta, profile):
print 'save_match:', twitter, profile
s3_filename = league + '/' + twitter + '.png'
bucket_util.save_s3(
boto.connect_s3().get_bucket('~bucket'),
s3_filename,
None,
systemfile,
'image/png',
'public-read',
meta + [('yes', False), ('no', False), ('maybe', False),
('profile', profile)]
)
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


Re: [Twisted-Python] dumb question on qt4reactor

2014-07-09 Thread Kevin Mcintyre
Another "Oh right" twisted moment.  callInThread instead of callLater?

This is a case where I've been laying tracks for 2 years without totally
understanding the train that was eventually coming down the tracks.  I
guess I implicitly understood by wanting to move to a separate reactor as a
separate thread, and in the past when I've used callLater it's encapsulated
non-blocking code.

In this sense is callLater ever utilized as a pseudo asynchronous method?



On Sat, Jul 5, 2014 at 5:24 PM,  wrote:

> On 5 Jul, 08:35 pm, kebi...@gmail.com wrote:
>
>> calling reactor.callLater within qt4reactor in an attempt to async non
>> webkit work but not seeing expected behavior.  Webkit locks while the
>> callLater function executes.
>>
>
> `callLater` does not make a blocking function into a non-blocking one. It
> just calls a function after a specified delay.  If the function blocks,
> then the reactor will block when it gets around to calling it.
>
> If you need to call blocking APIs you may need to call them in a separate
> thread (Twisted includes a threadpool that can help you with this) or a
> separate process (Twisted has some APIs for launching and interacting with
> other processes, too).  Of course, the ideal solution is often to find a
> non-blocking equivalent to the blocking API and use that instead (for
> example, I know that txaws is a project that exists - perhaps it can do the
> things to S3 that you want to do).
>
> Jean-Paul
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


Re: [Twisted-Python] dumb question on qt4reactor

2014-07-14 Thread Kevin Mcintyre
finalizing - maybe I'm not describing my usage correctly.  I've used
reactor.callLater(0,blah) as a means of starting a unit of work the
by-product of which the caller didn't need.

this is really a case of laying tracks for 2 years not fully comprehending
the train coming down the tracks behind me!  I sorta refused to learn/use
inlinecallbacks or generators and I've never used callInThread until now.
I inserted that into my S3 upload pipeline and it works, simplifying my
code but detaches the caller from reactive failure.

The mechanism you've described is different ~ a callLater to an estimated
point of work completion, maybe a holding pattern if work isn't completed
after the delay, and maybe not a bad means of observation (?)...but yes
disastrous if the callLater function is wired poorly.  Unless I'm
misreading.

Anyway I'm sure my code is illogical since I've was left to my own devices
to solve problems of my own choosing.

The by-product you can see here www.pokertalon.com - a single connection
html5 app.


On Wed, Jul 9, 2014 at 10:51 AM, Kevin Horn  wrote:

> It might be, but I can't think of a reason off hand why it ever _should_
> be.  I can kind of see how it might work, but if your "async" process ever
> takes longer than your callLater delay it will break, and probably quite
> horribly.
>
>
> On Wed, Jul 9, 2014 at 12:11 PM, Kevin Mcintyre  wrote:
>
>> Another "Oh right" twisted moment.  callInThread instead of callLater?
>>
>> This is a case where I've been laying tracks for 2 years without totally
>> understanding the train that was eventually coming down the tracks.  I
>> guess I implicitly understood by wanting to move to a separate reactor as a
>> separate thread, and in the past when I've used callLater it's encapsulated
>> non-blocking code.
>>
>> In this sense is callLater ever utilized as a pseudo asynchronous method?
>>
>>
>>
>> On Sat, Jul 5, 2014 at 5:24 PM,  wrote:
>>
>>> On 5 Jul, 08:35 pm, kebi...@gmail.com wrote:
>>>
>>>> calling reactor.callLater within qt4reactor in an attempt to async non
>>>> webkit work but not seeing expected behavior.  Webkit locks while the
>>>> callLater function executes.
>>>>
>>>
>>> `callLater` does not make a blocking function into a non-blocking one.
>>> It just calls a function after a specified delay.  If the function blocks,
>>> then the reactor will block when it gets around to calling it.
>>>
>>> If you need to call blocking APIs you may need to call them in a
>>> separate thread (Twisted includes a threadpool that can help you with this)
>>> or a separate process (Twisted has some APIs for launching and interacting
>>> with other processes, too).  Of course, the ideal solution is often to find
>>> a non-blocking equivalent to the blocking API and use that instead (for
>>> example, I know that txaws is a project that exists - perhaps it can do the
>>> things to S3 that you want to do).
>>>
>>> Jean-Paul
>>>
>>> ___
>>> Twisted-Python mailing list
>>> Twisted-Python@twistedmatrix.com
>>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>>
>>
>>
>> ___
>> Twisted-Python mailing list
>> Twisted-Python@twistedmatrix.com
>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>
>>
>
>
> --
> --
> Kevin Horn
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


Re: [Twisted-Python] Is there pb documentation somewhere?

2014-08-06 Thread Kevin Mcintyre
This discussion seems relevant to a design pattern "Yearbook Lifecycle"
that I'm using for athleets.com and junkeet.com.

Design Goal:

1) Maintain yearbooks across state, allowing students to delegate access
control to classmates as a series of transformations (recursive web
renders) - here sign my yearbook...here read what so-and-so wrote..can I
take your yearbook to PE, we might skip class.

2) Persist inside a context indexer that answers yearbook queries, with
generic access to the t0 and viewable access to t1+ transforms with request
controls for represented as e0, e1 being another recursive web template
kept within the master at t-1

So a complex use case would be a yearbook message like "OMG - did you read
{s1.p3.t4}? cause {e4.s3} said f_td(after|before) {e2.s6} was kissing
behind {p3.t0.a4}.  See you this summer good luck at
{owner.ext['college']}! BFF, {s3}"

Fake Model

YearbookPage(Element):
SignedYearbookPage(YearbookPage):

Yearbook(pb.Copyable)
-pages
-accessControlLog
StudentYearbook(pb.Viewable)
--transformation

YearbookServiceTransformer(??)

Publish(pb.Root)
remote_issue_yearbook(studentid=None): // none can read but not signed

Athleets.com is ready(ing) for yearbook-ish events, ie when a players get
traded or hurt.  Other events act across yearbook motif regenerating the
t0 when athlete data changes which republishes all copies non t0 that exist.

The Sender Receiver Model from the example is difficult to follow cause I
tend to think of Originator only.  The originator being a service that
waits for clients in need of a working copy.  A sender, doesn't exist...it
would be more of a challenger, that if recognized as more authoritative,
replaces the rb.Root service.  Outside of the copyable I'm trying to
upstream via remotes pieces within the copyable that need updating.  I'm
delving into viewable to bring caller identity data to the web templating
process not necessary restrict access.  But also to build a system that
maintains audit history within the yearbook model. Getting to areas of
non-working code, so let me report back any success.

Cheers
Kevin



On Tue, Aug 5, 2014 at 11:55 AM, Glyph Lefkowitz 
wrote:

>
> On Aug 4, 2014, at 9:47 PM, Daniel Sank  wrote:
>
> glyph,
>
> >> 2. Is there a specification for the pb dialect of banana?
> >>
> > Beyond the code, no.
>
> Ok.
>
> > I would be happy to answer questions, but obviously I'm not super
> responsive :).
> > Let me know what you need.
>
> For two personal projects, I would like to have a reasonable remote
> objects library in python. I need something which can announce state
> changes to clients, and receive state change requests from clients. My
> solution:
>
> 1. Make server side class which can spawn Cacheables when it wants to tell
> clients of its existence.
> 2. Give RemoteCaches to clients and use observe_* methods as appropriate.
> 3. Stuff a Viewable into the RemoteCaches so that clients can request
> state changes.
>
> Question #1: Is this a reasonable use of pb?
>
>
> Yes.
>
> This all worked great until I ran into a bug. In trying to fix the bug, I
> found that
>
> 1. pb code is really hard to understand
>
>
> Sorry about that.
>
> 2. exarkun's thinks pb is bad and that I should implement what I need in
> AMP.
>
>
> I really wish we would stop calling things "bad" and "good".  This isn't a
> helpful classification.  PB is adequate for a particular set of
> requirements.  Those requirements are somewhat unusual, and AMP is better
> for a lot of use-cases.
>
> It sounds to me like you are a lot more interested in
>
> 3. exarkun thinks banana and jelly are reasonable.
>
>
> Again, what does "reasonable" mean in this context?
>
> Let me explain my own opinion about this.
>
> Banana is a perfectly serviceable low-level marshaling format.  It's
> pretty efficient when compared to something like JSON, and has compression
> mechanisms which can make it even more efficient (the "dialect" support you
> referred to).  The only thing about it that isn't very general is that its
> implementation (although not the protocol specification) hard-codes the PB
> abbreviated-string dialect.
>
> Jelly is higher level, but more language-specific. Its specification
> implicitly encodes numerous Python implementation details, like the
> distinction between "tuple" and "list".  It also couples very tightly to
> your program's structure.  This can be a real benefit to getting a protocol
> up and running quickly, but it still allows you to create protocols where
> you don't really know what the wire format is, where you develop hidden
> dependencies.  In more complex protocols (where the "ease of getting up and
> running quickly" thing really starts to shine) this attribute of Jelly can
> cause real difficulty in any kind of cross-system communication:
> communicating with a peer from a different language, or even in Python
> without access to all the protocol class definitions from the original
> system, is hard because it requires rev

[Twisted-Python] pb.Copyable knowledge barrier

2014-08-07 Thread Kevin Mcintyre
goal: Give knowitall.py awareness of wordsmith.py addition.

to run
./library.sh # opens 7999 for remote_dictionary and remote_define methods
then both:
python knowitall.py
python wordsmith.py

library.sh due to main/jelly issues
from twisted.spread import pb

class UknownWordException(Exception):
pass

class UnknownDefinition(UknownWordException):
pass
# would like to remote define via error

class Dictionary():

def __init__(self, language, word=None, definition=None):
self.language = language
	self.words = {}
	if not word:
	word = language
	if not definition:
	definition = language + ' is a word worthy of a language'
	self.words[word] = set([definition])
	print self.words

def get_language(self):
return self.language

def lookup_word(self, word, definition = None):
	print 'lookup_word', self.words
try:
	return self.words[word]
except:
	if definition:
	raise UknownWordException(word)
	else:
	raise UnknownDefinition(word, definition)

class CopyDictionary(Dictionary, pb.Copyable):

def getStateToCopy(self):
print 'state getting copied', self.language
return self.language, self.words

class DictionaryEditor(pb.Referenceable):

def remote_word_definition(self, lanague, word, definition):
pass

class RemoteDictionary(pb.RemoteCopy, Dictionary):

def setCopyableState(self, state):
print 'remote state receives copy', state
self.language = state[0]
self.words = state[1]

class Library(pb.Root):

dictionaries = []

def remote_define(self, language, word, definition):
	dictionary = self.remote_dictionary(language, word, definition)	
	try:
	definition_set = dictionary.lookup_word(word, definition)
	print 'got definition_set', definition_set
	if definition not in definition_set:
		print 'new definition'
		definition_set.add(definition)
	else:
		print 'existing definition'
	except Exception as e:
	print e
	print 'new word'
	dictionary.words[word] = set([definition])
	return dictionary.words
		

def remote_dictionary(self, language, word = None, definition = None):
for d in Library.dictionaries:
if d.get_language() == language:
print 'remote'
return d
print 'create copyable version'
dictionary = CopyDictionary(language, word, definition)
Library.dictionaries.append(dictionary)
return dictionary
from twisted.spread import pb
from example import CopyDictionary, RemoteDictionary
from twisted.internet import reactor
from twisted.internet import defer

factory = pb.PBClientFactory()
reactor.connectTCP("localhost", 7999, factory)

pb.setUnjellyableForClass(CopyDictionary, RemoteDictionary)

def use_dictionary(dictionary):
for w in dictionary.words.keys():
	for d in dictionary.words[w]:
	print w, d
reactor.callLater(10, use_dictionary, dictionary)

def goto_library():
return factory.getRootObject()

import sys
language = 'english'
if len(sys.argv) > 1:
language = sys.argv[1]

def test():
d = goto_library()
d.addCallback(lambda root: root.callRemote('dictionary', language))
d.addCallback(use_dictionary)
return d

reactor.callWhenRunning(test)
reactor.run()



library.sh
Description: Bourne shell script
from twisted.spread import pb
from twisted.internet import reactor
from twisted.internet import defer

factory = pb.PBClientFactory()
reactor.connectTCP("localhost", 7999, factory)

def goto_library():
return factory.getRootObject()

def define_result(ans, language, word, definition):
# audit submission later
print ans
reactor.stop()

language = 'english'
word = language
definition = word + ' is part of ' + language

import sys 
if len(sys.argv) > 1:
language = sys.argv[1]
if len(sys.argv) > 2:
word = sys.argv[2]
if len(sys.argv) > 3:
definition = sys.argv[3]

def test():
d = goto_library()
d.addCallback(lambda root: root.callRemote('define', language, word, definition))
d.addCallback(define_result, language, word, definition)
return d

reactor.callWhenRunning(test)
reactor.run()

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


Re: [Twisted-Python] pb.Copyable knowledge barrier

2014-08-07 Thread Kevin Mcintyre
wordsmith argument order issues re-take.

Cheers,
Kevin


On Thu, Aug 7, 2014 at 11:29 PM, Kevin Mcintyre  wrote:

> goal: Give knowitall.py awareness of wordsmith.py addition.
>
> to run
> ./library.sh # opens 7999 for remote_dictionary and remote_define methods
> then both:
> python knowitall.py
> python wordsmith.py
>
> library.sh due to main/jelly issues
>
>
from twisted.spread import pb

class UknownWordException(Exception):
pass

class UnknownDefinition(UknownWordException):
pass
# would like to remote define via error

class Dictionary():

def __init__(self, language, word=None, definition=None):
self.language = language
	self.words = {}
	if not word:
	word = language
	if not definition:
	definition = language + ' is a word worthy of a language'
	self.words[word] = set([definition])
	print self.words

def get_language(self):
return self.language

def lookup_word(self, word, definition = None):
	print 'lookup_word', self.words
try:
	return self.words[word]
except:
	if definition:
	raise UknownWordException(word)
	else:
	raise UnknownDefinition(word, definition)

class CopyDictionary(Dictionary, pb.Copyable):

def getStateToCopy(self):
print 'state getting copied', self.language
return self.language, self.words

class DictionaryEditor(pb.Referenceable):

def remote_word_definition(self, lanague, word, definition):
pass

class RemoteDictionary(pb.RemoteCopy, Dictionary):

def setCopyableState(self, state):
print 'remote state receives copy', state
self.language = state[0]
self.words = state[1]

class Library(pb.Root):

dictionaries = []

def remote_define(self, language, word, definition):
	dictionary = self.remote_dictionary(language, word, definition)	
	try:
	definition_set = dictionary.lookup_word(word, definition)
	print 'got definition_set', definition_set
	if definition not in definition_set:
		print 'new definition'
		definition_set.add(definition)
	else:
		print 'existing definition'
	except Exception as e:
	print e
	print 'new word'
	dictionary.words[word] = set([definition])
	return dictionary.words
		

def remote_dictionary(self, language, word = None, definition = None):
for d in Library.dictionaries:
if d.get_language() == language:
print 'remote'
return d
print 'create copyable version'
dictionary = CopyDictionary(language, word, definition)
Library.dictionaries.append(dictionary)
return dictionary
from twisted.spread import pb
from example import CopyDictionary, RemoteDictionary
from twisted.internet import reactor
from twisted.internet import defer

factory = pb.PBClientFactory()
reactor.connectTCP("localhost", 7999, factory)

pb.setUnjellyableForClass(CopyDictionary, RemoteDictionary)

def use_dictionary(dictionary):
for w in dictionary.words.keys():
	for d in dictionary.words[w]:
	print w, d
reactor.callLater(10, use_dictionary, dictionary)

def goto_library():
return factory.getRootObject()

import sys
language = 'english'
if len(sys.argv) > 1:
language = sys.argv[1]

def test():
d = goto_library()
d.addCallback(lambda root: root.callRemote('dictionary', language))
d.addCallback(use_dictionary)
return d

reactor.callWhenRunning(test)
reactor.run()



library.sh
Description: Bourne shell script
from twisted.spread import pb
from twisted.internet import reactor
from twisted.internet import defer

factory = pb.PBClientFactory()
reactor.connectTCP("localhost", 7999, factory)

def goto_library():
return factory.getRootObject()

def define_result(ans, language, word, definition):
# audit submission later
print ans
reactor.stop()
import sys 
language = 'english'
if len(sys.argv) > 1:
language = sys.argv[1]
word = language
if len(sys.argv) > 2:
word = sys.argv[2]
definition = word + ' is part of ' + language
if len(sys.argv) > 3:
definition = sys.argv[3]

print 'language', language
print 'word', word
print 'definition', definition

def test():
d = goto_library()
d.addCallback(lambda root: root.callRemote('define', language, word, definition))
d.addCallback(define_result, language, word, definition)
return d

reactor.callWhenRunning(test)
reactor.run()

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


Re: [Twisted-Python] pb.Copyable knowledge barrier

2014-08-08 Thread Kevin Mcintyre
The question is what's necessary to get knowitall.py to receive updates to
held copy.

I start server, start knowitall, run wordsmith.  Knowitall doesn't receive
updates.  I understand why, setCopyableState only gets fired once.
Guessing jelly doesn't hold onto the RemoteCopy or a reference to
RemoteCopy.

Is the next step to add pb.Referenceable to Dictionary class, and use
observe_?



On Fri, Aug 8, 2014 at 12:30 AM, Daniel Sank  wrote:

> Kevin,
>
> Is there a question there? I'd be happy to try to help if I knew what you
> were asking.
>
>
> On Thu, Aug 7, 2014 at 11:42 PM, Kevin Mcintyre  wrote:
>
>> wordsmith argument order issues re-take.
>>
>> Cheers,
>> Kevin
>>
>>
>> On Thu, Aug 7, 2014 at 11:29 PM, Kevin Mcintyre 
>> wrote:
>>
>>> goal: Give knowitall.py awareness of wordsmith.py addition.
>>>
>>> to run
>>> ./library.sh # opens 7999 for remote_dictionary and remote_define methods
>>> then both:
>>> python knowitall.py
>>> python wordsmith.py
>>>
>>> library.sh due to main/jelly issues
>>>
>>>
>>
>> ___
>> Twisted-Python mailing list
>> Twisted-Python@twistedmatrix.com
>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>
>>
>
>
> --
> Daniel Sank
> Department of Physics
> Broida Hall
> University of California
> Santa Barbara, CA 93117
> (805)893-3899
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


Re: [Twisted-Python] pb.Copyable knowledge barrier

2014-08-08 Thread Kevin Mcintyre
I think so too, but I'm starting to lean toward websockets.  The warnings
alone right?

But I will take another pass and report back any success.

Cheers,


On Fri, Aug 8, 2014 at 9:04 AM, Daniel Sank  wrote:

> I think Cacheable exists to handle the case you want. Beware there are
> some funny issues with Cacheable
>
> https://twistedmatrix.com/trac/ticket/7274
>
> Daniel
>
>
> On Fri, Aug 8, 2014 at 8:01 AM, Kevin Mcintyre  wrote:
>
>> The question is what's necessary to get knowitall.py to receive updates
>> to held copy.
>>
>> I start server, start knowitall, run wordsmith.  Knowitall doesn't
>> receive updates.  I understand why, setCopyableState only gets fired once.
>> Guessing jelly doesn't hold onto the RemoteCopy or a reference to
>> RemoteCopy.
>>
>> Is the next step to add pb.Referenceable to Dictionary class, and use
>> observe_?
>>
>>
>>
>> On Fri, Aug 8, 2014 at 12:30 AM, Daniel Sank 
>> wrote:
>>
>>> Kevin,
>>>
>>> Is there a question there? I'd be happy to try to help if I knew what
>>> you were asking.
>>>
>>>
>>> On Thu, Aug 7, 2014 at 11:42 PM, Kevin Mcintyre 
>>> wrote:
>>>
>>>> wordsmith argument order issues re-take.
>>>>
>>>> Cheers,
>>>> Kevin
>>>>
>>>>
>>>> On Thu, Aug 7, 2014 at 11:29 PM, Kevin Mcintyre 
>>>> wrote:
>>>>
>>>>> goal: Give knowitall.py awareness of wordsmith.py addition.
>>>>>
>>>>> to run
>>>>> ./library.sh # opens 7999 for remote_dictionary and remote_define
>>>>> methods
>>>>> then both:
>>>>> python knowitall.py
>>>>> python wordsmith.py
>>>>>
>>>>> library.sh due to main/jelly issues
>>>>>
>>>>>
>>>>
>>>> ___
>>>> Twisted-Python mailing list
>>>> Twisted-Python@twistedmatrix.com
>>>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>>>
>>>>
>>>
>>>
>>> --
>>> Daniel Sank
>>> Department of Physics
>>> Broida Hall
>>> University of California
>>> Santa Barbara, CA 93117
>>> (805)893-3899
>>>
>>> ___
>>> Twisted-Python mailing list
>>> Twisted-Python@twistedmatrix.com
>>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>>
>>>
>>
>> ___
>> Twisted-Python mailing list
>> Twisted-Python@twistedmatrix.com
>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>
>>
>
>
> --
> Daniel Sank
> Department of Physics
> Broida Hall
> University of California
> Santa Barbara, CA 93117
> (805)893-3899
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


Re: [Twisted-Python] pb.Copyable knowledge barrier

2014-08-08 Thread Kevin Mcintyre
I experimented today with a table scan from dynamodb.  Taking each record
and converting to a dictionary into a list.  Initial scan and transform
took ~8 seconds from amazon to my house over TCP boto which is soapish? i
think.  ~7800 records.

The next call for the remote copy took 7 seconds across the bus.  Is that
expected?





On Fri, Aug 8, 2014 at 3:01 PM, Glyph Lefkowitz 
wrote:

>
> On Aug 8, 2014, at 1:31 PM, Kevin Mcintyre  wrote:
>
> I think so too, but I'm starting to lean toward websockets.  The warnings
> alone right?
>
>
> Websockets are a layer well below what PB does.  In fact you could run PB
> over websockets, and it would work roughly the same as over TCP.
>
> But I will take another pass and report back any success.
>
>
> Please do let us know.
>
> -glyph
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


Re: [Twisted-Python] pb.Copyable knowledge barrier

2014-08-08 Thread Kevin Mcintyre
correction.  I was stupidly reprocessing list on remote copy side.  Seeing
much better results now ~10K per second.


On Fri, Aug 8, 2014 at 6:32 PM, Kevin Mcintyre  wrote:

> I experimented today with a table scan from dynamodb.  Taking each record
> and converting to a dictionary into a list.  Initial scan and transform
> took ~8 seconds from amazon to my house over TCP boto which is soapish? i
> think.  ~7800 records.
>
> The next call for the remote copy took 7 seconds across the bus.  Is that
> expected?
>
>
>
>
>
> On Fri, Aug 8, 2014 at 3:01 PM, Glyph Lefkowitz 
> wrote:
>
>>
>> On Aug 8, 2014, at 1:31 PM, Kevin Mcintyre  wrote:
>>
>> I think so too, but I'm starting to lean toward websockets.  The warnings
>> alone right?
>>
>>
>> Websockets are a layer well below what PB does.  In fact you could run PB
>> over websockets, and it would work roughly the same as over TCP.
>>
>> But I will take another pass and report back any success.
>>
>>
>> Please do let us know.
>>
>> -glyph
>>
>>
>> ___
>> Twisted-Python mailing list
>> Twisted-Python@twistedmatrix.com
>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>
>>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


Re: [Twisted-Python] pb.Copyable knowledge barrier

2014-09-07 Thread Kevin Mcintyre
Jelly performance factors below expectations.

Can we say Copyable is the lowest order jelly? The notion that a copy
holder can't ask "is my copy good anymore?"  makes it so.  Essentially root
says, I'd prefer not to repeat unit of work nor keep track of the resulting
copies, here have the original or resulting copy.

My main issue is a copy-holder calling for a copy to determine is the copy
is good anymore.  I know, see cacheable but it's problematic.  At this
point I'm unsure of what jelly actually does well.  who's the JellyCon2014
keynote speaker?














On Sat, Aug 9, 2014 at 6:58 PM, Glyph  wrote:

> On Aug 8, 2014, at 9:17 PM, Kevin Mcintyre  wrote:
>
> correction.  I was stupidly reprocessing list on remote copy side.  Seeing
> much better results now ~10K per second.
>
>
> Whew.  I don't know exactly how well I would expect this perform, but
> those other results looked off by a couple orders of magnitude :-).
>
> -glyph
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


Re: [Twisted-Python] pb.Copyable knowledge barrier

2014-09-08 Thread Kevin Mcintyre
Jelly performance factors below expectations.


I am not clear on the significance of this statement.

Not significant, just an observation.  I've made this statement before,
backed-off after finding errors on my code, but after fixing I see
performance across the bus that still seem slow.  "Seem slow" lacks any
scientific backing.  I would be curious to hear use cases for PB/Jelly that
went beyond the docs.

Can we say Copyable is the lowest order jelly? The notion that a copy
holder can't ask "is my copy good anymore?"  makes it so.  Essentially root
says, I'd prefer not to repeat unit of work nor keep track of the resulting
copies, here have the original or resulting copy.


Asking whether your copy is good any more is a PB-level task.  Jelly itself
is a separate layer which is about getting the right data to the right
place, not keeping it updated.

Yes PB level, I guess I'm looking for a convention where 1 does not exist.
In the renewed interest the comments have tended to overlook copyable, or
see copyable as being flushed out.

My main issue is a copy-holder calling for a copy to determine is the copy
is good anymore.  I know, see cacheable but it's problematic.


"problematic" is passive voice :-).  What are the problems?

Only parroting what I've read on cacheable, haven't delved into yet.

At this point I'm unsure of what jelly actually does well.


Malheureusement, Jelly's main claim to fame is "it's better than pickle".
 But with all this renewed interest in PB perhaps we'll get
pre-deserialization schema enforcement and type checking, and then it will
have some real advantages :).

Agreed.  This is what I looked to Jelly/PB for, having found so much in
twisted that simplifies.

who's the JellyCon2014 keynote speaker?


*One* day we will have a conference - Tx/Rx, of course - and there will be
a talk on this.  But first we need several people in the community who
wants their contribution to Twisted to be organization and community
oriented rather than code ;-).

On Sun, Sep 7, 2014 at 9:43 AM, Glyph  wrote:

>
> On Sep 7, 2014, at 12:51 AM, Kevin Mcintyre  wrote:
>
> Jelly performance factors below expectations.
>
>
> I am not clear on the significance of this statement.
>
> Can we say Copyable is the lowest order jelly? The notion that a copy
> holder can't ask "is my copy good anymore?"  makes it so.  Essentially root
> says, I'd prefer not to repeat unit of work nor keep track of the resulting
> copies, here have the original or resulting copy.
>
>
> Asking whether your copy is good any more is a PB-level task.  Jelly
> itself is a separate layer which is about getting the right data to the
> right place, not keeping it updated.
>
> My main issue is a copy-holder calling for a copy to determine is the copy
> is good anymore.  I know, see cacheable but it's problematic.
>
>
> "problematic" is passive voice :-).  What are the problems?
>
> At this point I'm unsure of what jelly actually does well.
>
>
> Malheureusement, Jelly's main claim to fame is "it's better than pickle".
>  But with all this renewed interest in PB perhaps we'll get
> pre-deserialization schema enforcement and type checking, and then it will
> have some real advantages :).
>
> who's the JellyCon2014 keynote speaker?
>
>
> *One* day we will have a conference - Tx/Rx, of course - and there will
> be a talk on this.  But first we need several people in the community who
> wants their contribution to Twisted to be organization and community
> oriented rather than code ;-).
>
> -glyph
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


Re: [Twisted-Python] not terrible object oriented rpc system in js?

2014-12-27 Thread Kevin Mcintyre
flood with 50 permutations?

On Sat, Dec 27, 2014 at 4:43 PM, Clayton Daley 
wrote:

> Not sure which features of PB matter to you, but I just started playing
> with Autobahn/Crossbar. It's
>
>- A JSON-based RPC (and/or pubsub) system,
>- With clients in many languages (JS, Python, C++, etc.), and
>- The ability to make cross-language RPC calls
>
> Out of the box, object serialization is limited (likely to ensure
> cross-platform compatibility), but the serializer is (purportedly)
> pluggable.
>
>
> Clayton Daley
> clayton.da...@gmail.com
> 513.505.1236
>
> On Sat, Dec 27, 2014 at 3:06 PM, Daniel Sank 
> wrote:
>
>> This looks promising. Thanks.
>>
>> On Sat, Dec 27, 2014 at 1:23 PM, Jens Sauer  wrote:
>>
>>> Hi Daniel,
>>>
>>> whats about node.js with http://uber.github.io/multitransport-jsonrpc/
>>>
>>> Regards
>>>
>>> J.Sauer
>>>
>>>   --
>>>  *Von:* Daniel Sank 
>>> *An:* twisted-python 
>>> *Gesendet:* 16:30 Samstag, 27.Dezember 2014
>>> *Betreff:* [Twisted-Python] not terrible object oriented rpc system in
>>> js?
>>>
>>> Are you guys aware of anything which solves the same kind of problem as
>>> t.s.pb but in js? I asked about this on the js stack exchange chat room and
>>> the general response was that there's isn't anything. I figured y'all might
>>> know more.
>>>
>>> --
>>> Daniel Sank
>>>
>>>
>>>
>>> ___
>>> Twisted-Python mailing list
>>> Twisted-Python@twistedmatrix.com
>>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>>
>>>
>>>
>>> ___
>>> Twisted-Python mailing list
>>> Twisted-Python@twistedmatrix.com
>>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>>
>>>
>>
>>
>> --
>> Daniel Sank
>>
>>
>>
>> ___
>> Twisted-Python mailing list
>> Twisted-Python@twistedmatrix.com
>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>
>>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python