Re: [Twisted-Python] getting rid of semantic newlines

[James Broadhead]

On 6 March 2015 at 14:06, Adi Roiban  wrote:

> I don't care what standard is used, just make sure that it is not
> stated on a page with 99% of obsolete info and in a section named
> "XHTML Layout" like it is now on
>

<3

Enforcement by tooling would be better again.
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] buildbot.twistedmatrix.com is down a lot

[James Broadhead]

On 17 July 2016 at 07:21, Amber Brown  wrote:

> It's OOMing  (...)
>

Have you considered something like monit[1] to detect & restart in cases
like this?

[1] https://mmonit.com/monit/
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] gRPC support in Twisted Python

[James Broadhead]

If you're planning on working on this, it'd probably be worth taking a look
at the Twisted bindings for Thrift (a competing RPC system from FB, based
on the Google PB paper).

The generator[2] isn't the best place to start (imperative C++ that outputs
python), but the code that it generates served us very well for years
(generally, communicating twisted->cassandra, or twisted->scala services).

[1] https://thrift.apache.org/
[2]
https://github.com/apache/thrift/blob/master/compiler/cpp/src/thrift/generate/t_py_generator.cc

On 29 September 2016 at 08:34, Stephen Thorne  wrote:

> [+Nathaniel]
>
> https://groups.google.com/forum/m/#!topic/grpc-io/RpkyqqQy8TU/discussion
>
> Hi. I'd like to link you to the above discussion and cc it's original
> poster.
>
>
> On Wed, Sep 28, 2016, 22:51 Werner Thie  wrote:
>
>> On 9/28/16 9:13 AM, Nursimulu, Khen wrote:
>> > Thanks Glyth for the prompt response.
>>
>> > On Sep 28, 2016, at 6:13 AM, Nursimulu, Khen > > > wrote:
>> >
>> > Is there a plan (or an implementation) to support gRPC within
>> > Twisted Python?   My understanding is that gRPC is built using
>> > Futures and creates its own threads for all its event handling.
>> > There is also a gRPC Python package (grpcio 1.0.0) that is available
>> > for python 2.7.In order to use gRPC with Twisted Python in 2.7
>> > is the only way to have gRPC run in its own thread?
>> >
>> > There's no plan that I'm aware of.  You could definitely run gRPC in a
>> > thread currently, although it would be nice if grpc worked natively with
>> > Twisted.
>> >
>> > Probably contributing this upstream into the gRPC project would be the
>> > best way to start, and if they're not receptive, starting a separate
>> > 'txgRPC' project.
>> >
>>
>> Interesting, with the browser implementation in the works this could
>> become a full replacement of nevow/athena
>>
>> Werner
>>
>> ___
>> Twisted-Python mailing list
>> Twisted-Python@twistedmatrix.com
>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Python3 twistd daemon for Ubuntu 14.04 alternatives

[James Broadhead]

On 23 February 2017 at 08:36, Glyph Lefkowitz 
wrote:
>
>
> Go's build toolchain has many features worth envying but most of its
> advantages have to do with deployments _outside_ of containers, where you
> have to ship to customer environments with fraught and unknown system
> configurations.  If you have any level of control over your deployment
> target, Go and, say, Python with PEX are ~equivalent .


I was going to chime in about pex earlier - but since you need to provide
your own interpreter, I didn't think it solved his whole issue.

Otherwise, pex is a great format for deploying python easily.
At Twitter, we deployed the (Twisted) TweetDeck API using it for years,
with a ton of dependencies, both c-libs and pure python.

Really nice to be able to produce a deployable artefact, which can be
hashed & stored for re-deploy later if needed (and ideally
integration-tested & staged by passing different args on the cli).
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Python3 twistd daemon for Ubuntu 14.04 alternatives

[James Broadhead]

On 24 February 2017 at 10:25, Hynek Schlawack  wrote:
>
> ...  Contrary I’m not a super fan of having one opaque blob on server; the
> transparent structure of a virtualenv is something I learned to appreciate.
> ...
>

A zipfile containing a virtualenv (pex) isn't all that opaque -- with the
right editor plugins, you can even *cough* edit the contained source inside
one.

Back when pex was a lot younger, this was handy for quick debugging as we
were migrating from source-in-venvs-on-bare-metal-chroots to
pexed-in-mesos-containers
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] twistedmatrix.com TLS certificate

[James Broadhead]

Just ran into this. For interested parties, Chrome 58 is due to go stable
on Apr 25th [1]



[1] https://www.chromium.org/developers/calendar

On 7 March 2017 at 03:19, Glyph Lefkowitz  wrote:

>
> On Mar 6, 2017, at 12:16 AM, Cory Benfield  wrote:
>
>
> On 6 Mar 2017, at 07:22, Tristan Seligmann 
> wrote:
>
> twistedmatrix.com's current certificate is issued by StartCom
> Certification Authority; for certificates issued by this CA prior to
> 2016-09-21, the domain must be on a Chrome whitelist for it to be accepted.
> As of Chrome 58.0.3026.3 (canary/dev channel only, currently, but
> eventually this will presumably be in a release version) twistedmatrix.com
> is no longer[1] on the whitelist, which means that twistedmatrix.com will
> issue a certificate error. Can we switch to another CA? (Let's Encrypt, for
> example; I hear somebody wrote a Twisted library for using that)
>
> I'm sending this to the general list in case anyone else has been
> scratching their head about why they're getting cert warnings.
>
>
> This is an extremely good idea.
>
>
> Yes please.
>
> This is the rare ops task that will actually be quite easy for someone to
> add in to Braid as a PR: https://github.com/twisted-infra/braid
>
> If you have a look at https://github.com/twisted-infra/braid/blob/master/
> services/t-web/twisted-web/ports you might be able to guess how such a
> thing would go...
>
> -glyph
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Minor fixes in the “Customizing twistd logging” section

[James Broadhead]

Hey there --

One way would be to use virtualenv to gather the dependencies without
installing them globally:

mkdir -p ~/venv
virtualenv ~/venv/twisted_docs   # `sudo pip install virtualenv` first if
you don't have this command
source ~/venv/twisted_docs/bin/activate
pip install sphinx twisted
cd /docs
make html

James

On 13 March 2017 at 19:58, Роман Мещеряков 
wrote:

>
> Hello,
>
> I managed to read documentation finally. I made a fork of Twisted
> repository using these commands:
>
>
> git clone https://github.com//twisted.git Twisted
> cd Twisted
> git remote add upstream https://github.com/twisted/twisted.git
> git fetch upstream
> git branch -u upstream/trunk trunk
>
> Then created python virtualenv and installed Sphinx into it. Then I tried
> to generate narrative HTML documentation:
>
> (twisted-dev-env) pi@zyuzino11:~/ColourDreams/Twisted-dev $ cd docs
> (twisted-dev-env) pi@zyuzino11:~/ColourDreams/Twisted-dev/docs $ make html
> sphinx-build -b html -d _build/doctrees   . _build/html
> Running Sphinx v1.5.3
>
> Exception occurred:
>   File "conf.py", line 45, in 
> from twisted import version as twisted_version_object
> ImportError: No module named twisted
> The full traceback has been saved in /tmp/sphinx-err-75zxJ8.log, if you
> want to report the issue to the developers.
> Please also report this if it was a user error, so that a better error
> message can be provided next time.
> A bug report can be filed in the tracker at  doc/sphinx/issues>. Thanks!
> Makefile:53: ошибка выполнения рецепта для цели «html»
> make: *** [html] Ошибка 1
> (twisted-dev-env) pi@zyuzino11:~/ColourDreams/Twisted-dev/docs $
>
> Please help me to resolve this error!
>
> --
> Kind regards, Roman Mescheryakov
>
>
>
> 03.03.2017, 15:25, "Jean-Paul Calderone" :
>
> On Fri, Mar 3, 2017 at 2:57 AM, Роман Мещеряков <
> romanmescherya...@yandex.ru> wrote:
>
> Hi everyone,
>
>
> *On the documentation web page
> https://twistedmatrix.com/documents/current/core/howto/application.html
> 
> in the “Customizing twistd logging” section there are 2 examples which as I
> can see are a bit outdated in terms of imported module names and some other
> minor details. I would like to fix that. Is this change is worth doing? If
> so, please guide how can I do it. My GitHub login, if it’s needed, is
> RomanMeR.*
>
>
> Hi Роман,
>
> Thanks for your interest in improving Twisted!  Have a look at
> https://twistedmatrix.com/trac/wiki/TwistedDevelopment and then let us
> know if you have further questions.
>
> Jean-Paul
>
> ,
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] EuroPython Sprint Ideas?

[James Broadhead]

Hi guys,

EuroPython is just around the corner, so I was wondering if there are a few
people from this list going, and whether any of you have suggestions for a
sprinting session. There are around 60 people with the 'Twisted' tag on the
EuroPython site, so we could probably put together a moderate-sized team to
work on a larger feature.
It would be really helpful if someone who has actually made some Twisted
commits could be there too, for process and guidance etc. .

Suggestions / bug links welcome, even if you're not going to be there!

James
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] EuroPython Sprint Ideas?

[James Broadhead]

On 28 June 2013 01:39,  wrote:

>
> One project which is massively parallelizable is emptying this page:
>
>
> http://twistedmatrix.com/trac/**report/15
>
> Instructions for doing so, in some great detail, are available at:
>
>
> http://twistedmatrix.com/trac/**wiki/ReviewProcess
>
> Whatever happens, I hope everyone attending the conference has a great
> time!
>

Yep, that looks like the right place to start alright -- but having someone
there who is familiar with the process is always really valuable for quick
questions etc.

I'll be sure to be loitering in the chat room when we start work; if any
particular suggestions occur, do let me know. I've vaguely been thinking
about looking at the leaves of the python3x dependency graph, but we'll see
how far I get with the reviews.

I've had one email from an off-list volunteer, so let's see how many turn
up.

JB
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Trial & the mock library

[James Broadhead]

Hey all -

I've recently started working with the 'mock' library in our trial tests,
and am looking for some best-practice advice.  I'm really just starting to
get used to the library, so it might well have functionality that I'm
unaware of or am misusing.

I very quickly ran into a problem where I mistakenly returned a Mock() in
the place of a deferred, causing the asserts in callbacks to not be called,
and for the test to spuriously pass.

A basic example:

def test_foo():
  d = Mock()
  def check_result(res):
self.assertEqual(res.code, expected)  # never called
  d.addCallback(check_result)
  return d # Mock is truthy, test passes

This occurred where I was mocking some internals of the class under test;
something like the below

A slightly more believable example:
== myclass.py ==
def some_function(...):
  d = self.authenticate()
  d.addCallback(foo) # foo never called
  d.addErrback(bar) # bar never called
  return d

== test_myclass.py ==
def setUp(self):
  self.resource.authenticate = Mock(return_value=Mock())

def test_foo():
  d = self.resource.some_function
  def check_result(res):  # never called
self.assertEqual(res.code, expected)
  d.addCallback(check_result)
  return d # test passes

Currently, I'm experimenting with wrapping Mock instantiations by defining
common deferred methods on them in advance; this approach would eventually
lead to extending Mock itself with this functionality.

def nonDeferredMock():
m = Mock()
def notimpl(*args, **kwargs):
raise NotImplementedError('You treated a Mock like a Deferred!')
m.addCallback = notimpl
m.addErrback = notimpl
m.addBoth = notimpl
m.addCallbacks = notimpl
return m

Another approach might be extending TestCase to check that return values
are always not Mock objects.

Does anyone on the list have experience with this? Obviously, this only
happens when mistakes are made when writing tests, but I'd rather have
confidence that when my tests pass, that they've passed for the right
reasons.

Another antipattern that I've come across has been:

resource.mymethod = Mock(return_value=defer.succeed(None))

which works fine for tests in which mymethod() is called once, but always
returns the same deferred object if multiple calls are made. What would be
a better approach?

Cheers-

James
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted as a House for Sale

[James Broadhead]

This was hilarious, thank you for sharing it. Forwarded the link to my
whole team.



JB
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Bristol (England!) sprint hosted by HybridCluster

[James Broadhead]

Sounds exciting -- there have been some changes to trial that I've been
thinking about for a little while, so this would be a great opportunity.

Would you mind if I distributed this email farther afield? To
python...@python.org for instance (or are you waiting until you have a
fixed venue?)


On 5 November 2013 22:25,  wrote:

> Greetings all,
>
> I'm pleased to invite you to join Tom Prince (2013 Twisted Fellow), Luke
> Marsden (founder of HybridCluster), and the HybridCluster team (including
> myself!) for the first Twisted sprint hosted by HybridCluster!
>
> The sprint will take place on December 7th in central Bristol (exact
> location to be determined) from 11AM until no one remains capable of higher
> brain function(*).
>
> As usual sprint topics will surely be wide ranging - bring your favorite
> pet project, a personal loathing for unresolved bugs, or just find someone
> interesting to work with when you show up (personally, I'll probably be
> trying to add APIs to support more easily testing Agent- using
> applications).
>
> HybridCluster is a small Bristol-based company building high density web
> hosting platform using Twisted for all manner of tasks - from SSH, HTTP,
> SMTP, POP3, and IMAP4 to internal cluster coordination and asynchronous SQL
> access.
>
> Watch this space for a follow-up post giving details about the venue (and
> free food!).  In the meantime feel free to contact me or post to the list
> with any other questions.
>
> Jean-Paul
>
> (*) Due to excessive hacking.  Certainly not because of any kind of mind
> control technology deployed by the TSF.
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Silverberg CQL Driver

[James Broadhead]

We (TweetDeck@Twitter) have been using a patched version of Telephus for 3
years now.

We've had problems in the past where it would cause problems with down and
flaky seed hosts, so ymmv.

It's available here: https://github.com/tweetdeck/Telephus

On 3 December 2014 at 20:03, Jason J. W. Williams  wrote:

> Is anyone using the silverberg CQL (Cassandra) library for Twisted in
> production? Just curious what people's experience with it is, since it
> appears to be the most recently updated Twisted/CQL project.
>
> -J
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Silverberg CQL Driver

[James Broadhead]

On 29 December 2014 at 19:11, Jason J. W. Williams <
jasonjwwilli...@gmail.com> wrote:

> Hi James,
>
> Have you guys tried Silverberg? Wondering if its less flakly.
>
> -J
>

We haven't -- when I said that, I meant "on some occasions, after 3 years
of continuous use". It's a sporadic failure mode, and hasn't happened
enough times for us to investigate deeply. That said, we're in our own DC,
so the environment is pretty stable. If you're in AWS or similar, you might
want to run some testing around failure cases (let us know what you find!)
All that said, we've been very happy with it.

The likelihood is that we'll be working on migrating to Manhattan[1] rather
than trying out a new Cassandra library.
Not quite sure which way we'll go, but one option may be to write a
telephus-like wrapper around the thrift bindings.


[1]
https://blog.twitter.com/2014/manhattan-our-real-time-multi-tenant-distributed-database-for-twitter-scale
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Unit testing, trail, inlineCallbacks, deferreds and mocking

[James Broadhead]

Hey

I raised a similar question a while ago:
http://twistedmatrix.com/pipermail/twisted-python/2013-July/027241.html

Since then, our approach has evolved into the below, which may be useful.
https://github.com/jamesbroadhead/bttrtwisted/blob/master/bttrtwisted/testing.py

Usage:
  expected = Foo()
  remote_result = Result()
  thing = Thing()
  thing.call_external_service = dmockfunc(remote_result)

  d = thing.function_under_test(..)
  d.addCallback(self.assertEqual, expected)
  return d

You can use the func_dict param to gen_nondeferred_mock to have it stand
in-place-of an object

As always with mocks, a little can be helpful, but if you find you're
instantiating a lot of them, you may want to reconsider your approach.

Feedback welcome!
[ the repo is for experiments, so use with care ]

James

On 27 January 2015 at 13:00, Patryk Ściborek  wrote:

> Hi!
>
> I've just started a new project using Twisted and I want to write unit
> tests since the beginning. Unfortunately I've got some trouble
> understanding how should I do it. I read 'Test-driven development with
> Twisted', read some articles on the web and searched on the mailing list
> but I couldn't find anything which make it clear for me.
>
> I've got a class:
>
> class SessionCleaner(object):
> def __init__(self, session_db, interval=10):
> self.session_db = session_db
> self.lc = task.LoopingCall(self.check_old_sessions)
> self.lc.start(interval)
>
> @defer.inlineCallbacks
> def check_old_sessions(self):
> log.msg('check_old_sessions()', logLevel=logging.DEBUG)
> try:
> old_sessions = yield self.session_db.get_old_sessions()
> for s in old_sessions:
> yield self.session_db.process_stopped(s)
>
> except txredisapi.ConnectionError as e:
> log.msg('check_old_sessions - connection error {}'
> .format(e), logLevel=logging.WARNING)
>
> session_db is a object with methods which makes some calls to Redis.
>
> Testing if __init__ works correctly is easy - I can mock task.LoopingCall
> and check if it was called with correct attributes.
>
> I've got trouble testing check_old_sessions. Since I'm writing unit tests
> I don't want to call real session_db methods and make real Redis queries.
> I'd like to mock them and test just few things:
> - is the method get_old_sessions called?
> - is the method process_stopped called N times with the arguments returned
> by mocked get_old_sessions?
> - is txredisapi.ConnectionError handled correctly?
>
> So is there any "right" way of mocking functions which returns deferreds?
> Or maybe I should test this method differently?
>
> Best regards,
> Patryk
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python