Re: [Twisted-Python] Problem with osx10.10-py2.7 builder

> On Apr 2, 2018, at 7:13 AM, Adi Roiban wrote: > > FYI buildbot/osx10.10-py2.7 is broken again FWIW, `pip` on OSX (perhaps others?) swallows the 403 error about TLS brownouts on pypi for the 1.0/1.1 ending next week. No error is raised, and you just get an empty match for installable versio

Re: [Twisted-Python] twistd script python path

> On Mar 27, 2018, at 3:32 PM, Paul Albrecht wrote: > > I'm trying to debug a memory leak in a twisted application and I’d like to > use tracemalloc. Unfortunately, the application hasn’t been updated to python > 3 so I have to patch python 2.7. Here’s my problem: After downloading and > comp

Re: [Twisted-Python] process manager for twisted daemons?

On May 3, 2017, at 7:46 PM, Tristan Seligmann wrote: > It sounds like systemd has what you want here, and should work fine for your > Twisted services, assuming you don't have other constraints that exclude its > use. we were avoiding it because of a few reasons, but will need to rethink that

[Twisted-Python] process manager for twisted daemons?

We have a few twisted daemons running on a linux machine under Supervisord as the process manager. Unfortunately, Supervisord is missing a feature that is pretty critical to our needs, so must be replaced with something that supports it -- the feature is the ability to persist state across rebo

Re: [Twisted-Python] Removing support for old Python 3 versions

Dropping 3.3 makes sense, as does deprecating 3.4 The only thing I suggest/request is a webpage (and possibly txtfile in the source) that lists the supported Python versions. When people runs into issues, finding this info quickly is just so useful. e.g. Version Python2

Re: [Twisted-Python] suggestions sought on multi-service application dealing with shared database issue

On Jan 17, 2017, at 6:42 PM, Glyph Lefkowitz wrote: > > Sorry, I meant you should use > https://twistedmatrix.com/documents/16.6.0/api/twisted.application.service.IService.html#disownServiceParent > disownServiceParent. > wonderful! thanks! so much!

Re: [Twisted-Python] suggestions sought on multi-service application dealing with shared database issue

On Jan 17, 2017, at 5:08 AM, Glyph Lefkowitz wrote: > > It depends if it's easier to keep track of when changes happen, or determine > what the current state is. > > If it's easier to keep track of when changes happen, I would recommend > calling `setServiceParent` to add/remove the service to

[Twisted-Python] suggestions sought on multi-service application dealing with shared database issue

I have a few ideas on how to handle this, but wanted to see if anyone else has dealt with this and has a recommended approach: I have a twisted daemon that I will loosely describe as this: * an application defined in a `.tac` file that is an instance of `twisted.application.service.App

Re: [Twisted-Python] twistd and loglevel question

On Dec 22, 2016, at 5:00 PM, Glyph Lefkowitz wrote: > Nope. If you want interaction with stdlib logging, there's > https://twistedmatrix.com/documents/16.6.0/api/twisted.logger.STDLibLogObserver.html > and some associated utilites. But that's up to you, not twistd. (Maybe > twistd should star

Re: [Twisted-Python] twistd and loglevel question

On Dec 22, 2016, at 3:33 PM, Glyph Lefkowitz wrote: > By "normal" I assume you mean stdlib? No, we don't use stdlib logging. > > -glyph Yes, thanks! I just realized that "if I don't do anything at all", all the standard library's logging calls (all imported 3rd party packages) wind up on std

[Twisted-Python] twistd and loglevel question

i couldn't find anything in docs/source, but just want to double-check... am i correct that there is nothing in `twistd` that will automatically set the (normal) python logging level based on args/env vars? ___ Twisted-Python mailing list Twisted-Pyt

Re: [Twisted-Python] overview: new review queue venue

On May 24, 2016, at 6:49 PM, Glyph wrote: > I KNOW, RIGHT!!! However, protected statuses somewhat reduce the potential > race-condition here. And contributing to a couple dozen Github projects I > have to say that practically this has never been an issue, even though I find > it aestheticall

Re: [Twisted-Python] overview: new review queue venue

just a few thoughts: The current system as-explained seems to use an "Issue" as a queue item that is either a "bug report" or a "notice of a pull request, which may also reference another bug report". that is weird. IMHO, a quality contributor will never get turned off by handling the docs,

Re: [Twisted-Python] overview: new review queue venue

FWIW, I just noticed that the CONTRIBUTING message wasn't showing up when trying to make a PR. github is supposed to make a warning box that shows the contents of /CONTRIBUTING or /CONTRIBUTING.md when opening an issue or PR ___ Twisted-Python mailing

Re: [Twisted-Python] twisted listening on UDP port, why?

On Apr 23, 2016, at 4:52 AM, Phil Mayers wrote: > Not so. It's down inside the kernel. All applications using the socket API in > this way will display this behaviour, regardless of language. > > > As glyph says, this is an inherent feature of the socket API. When you create > a socket, i

Re: [Twisted-Python] twisted listening on UDP port, why?

Thanks for all this. A few quick notes: On Apr 20, 2016, at 9:15 PM, Glyph wrote: > This isn't so much a feature of Python as it is a feature of the BSD sockets > API. Sending traffic through a socket, whether it's TCP or UDP, has to bind > a client port. Given the nature of UDP, binding o

Re: [Twisted-Python] twisted listening on UDP port, why?

On Apr 20, 2016, at 1:31 PM, Jonathan Vanasco wrote: > A specific library was keeping the port open. I'm tracking down how/why > right now. So this was fun thing to learn... An undocumented (yay) feature of python appears to be... python binds to a random port on all interfac

Re: [Twisted-Python] twisted listening on UDP port, why?

Thanks all. I was going to write everything below, then finally tracked it down. A specific library was keeping the port open. I'm tracking down how/why right now. The worry on this, FYI, was that there is a critical vulnerability in glibc from a few months ago where a server could be comprom

[Twisted-Python] twisted listening on UDP port, why?

I'm trying to lock down a box, and came across a peculiarity with a twisted dameon -- it's binding to 0.0.0.0 for UDP on an arbitrary port I can'f figure out why this is happening -- I'm not consciously/explicitly using anything on UDP, and the port changes every time I start up a daemon. Does

Re: [Twisted-Python] Migration to Git / GitHub

On Nov 17, 2015, at 12:52 PM, Donald Stufft wrote: > I don’t disagree with the decision to not move to GitHub issues, however > if you wanted to you could do what Golang did to minimize the data loss. > They emailed prior participants to their bug tracker to ask them to grant > their migration ap

Re: [Twisted-Python] Migration to Git / GitHub

Two notes: 1. Have you considered bitbucket? I personally prefer github, but I know some python projects like it because of mercurial support. 2. There are a lot of projects for transitioning trac tickets to github/bitbucket via their api. You can't transition some fields (reports, comments,

Re: [Twisted-Python] [Twisted-web] Changing the Twisted Compatibility Policy

On Oct 25, 2015, at 11:34 PM, Glyph Lefkowitz wrote: > For another, we have no way to hide deprecated APIs in the API documentation. > If a new user is trying to figure out how to do something, if they're doing > it during the deprecation period, they should not start out by using a > depreca

Re: [Twisted-Python] txmongo vs. pymongo + callInThread

On Oct 9, 2015, at 1:06 PM, Clayton Daley wrote: > I (OP) was looking into these libraries/options as part of a side project > that's been delayed by real work. I'm also relegating Mongo to a caching > layer after stumbling upon Sarah Mei's "Why You Should Never Use MongoDB" and > realizing I

Re: [Twisted-Python] txmongo vs. pymongo + callInThread

On Oct 8, 2015, at 6:19 AM, bret curtis wrote: > We're not sure about what OP has done, but if there is ever anything lacking > in TxMongo... please open an issue/ticket with us over at: > https://github.com/twisted/txmongo Thanks for the reply! I actually did open a few lowball tickets on Tx

Re: [Twisted-Python] txmongo vs. pymongo + callInThread

On Dec 21, 2014, at 4:05 PM, Clayton Daley wrote: > Are there any major disadvantages of using pymongo with callInThread instead > of txmongo? I'd like to take advantage of some newer features in pymongo > (unfortunately not available in txmongo) and it's certainly easier to > maintain featur

[Twisted-Python] is it possible to change the isolation level of a psycopg2 connection under enterprise.adbapi ?

there doesn't seem to be a way to access the connection objects within the pool ( psycopg2 manages this via `connection.set_isolation_level(X)` the only workaround I can think of seems to be emitting raw sql when I first start the transaction - but this doesn't seem right. am i missing anything

Re: [Twisted-Python] advice sought - two database connections in a RunInteraction (adbapi.ConnectionPool)

oh great! I'll jump through the docs this weekend! thanks so much for the reply and for writing that! On Sep 23, 2014, at 1:45 AM, Glyph wrote: > > On Sep 22, 2014, at 9:00 AM, Jonathan Vanasco wrote: > >> has anyone had a need like this before? if so, what did you d

[Twisted-Python] advice sought - two database connections in a RunInteraction (adbapi.ConnectionPool)

I've got some database transactional code running within a thread via RunInteraction (it needs to rollback on error). One of the features on the logic is that it interacts with a 3rd party API, which may or may not have a ratelimit in effect. Most 3rd party APIs i consume have a minute/hourly

Re: [Twisted-Python] are there known ssl/threading/socket issues with httplib2/httplib and Twisted ?

On Sep 18, 2014, at 8:35 AM, Dustin J. Mitchell wrote: > Are you by any chance using the same HTTPConnection instance in both threads? > > Dustin Ack! It looks like the snippet of sample code that I used as a constructor did that. I did not see that. This api library and docs are... frustra

[Twisted-Python] are there known ssl/threading/socket issues with httplib2/httplib and Twisted ?

Background: I'm using a 3rd party API client, which uses httplib2 in the internals. ( the google-plus api ) I'm placing this blocking code within a DeferredList, that runs within a twisted.enterprise.adbapi.ConnectionPool.runInteraction In my initial tests, everything worked fi

Re: [Twisted-Python] running twisted with supervisord -- logging question

Thanks Hynek and Christopher. Hynek- That's basically what I want -- but i am always quickly LOST when it comes to logging. It is my weakness. Christopher-- The code is definitely contradicting me. I am admittedly lost. I'll have to play around with this more in depth later -- but now I have

Re: [Twisted-Python] running twisted with supervisord -- logging question

ot; command = /var/www/sites/myapp-virtualenv/bin/twistd -y controller-production.tac --pidfile /tmp/myapp-twisted.pid -- On Aug 22, 2014, at 11:50 PM, Christopher Armstrong wrote: > > On August 22, 2014 at 6:52:02 PM, Jonathan Vanasco (twisted-pyt...@2xlp.com) > wrote: &g

[Twisted-Python] running twisted with supervisord -- logging question

hoping someone can answer this question- i can't seem to get my logging right while using supervisord i'm not using twisted logging, just the regular python logging infrastructure. the two end-results I've had: - everything double-logs into twistd.log & /var/log/supervisor.log

Re: [Twisted-Python] Status of trac upgrade

On Jun 18, 2014, at 8:43 PM, exar...@twistedmatrix.com wrote: > This is a nice thought but I think it's entirely misguided. > > Overcoming simplistic, automated obstacles is what spammers have been > learning how to do extremely well for several decades now. If you choose to > participate in

Re: [Twisted-Python] Status of trac upgrade

On May 29, 2014, at 9:13 AM, Hynek Schlawack wrote: > So what *is* the status? The current state is really hardly bearable; the > spam is taking completely over. :( Wasn’t there a successful dry run at the > PyCon sprints? I recently had a similar problem. I didn't realize a "one click inst

Re: [Twisted-Python] adbapi.ConnectionPool - reconnecting to postgresql

On Jun 12, 2014, at 1:13 AM, Glyph wrote: > alchimia is not a re-implementation or a separate sqlalchemy package. It > *is* sqlalchemy; it's just a slight extension for integration sqlalchemy > operations with the Twisted main loop. Yeah, I get that. I'm really excited for it. I use the dec

Re: [Twisted-Python] adbapi.ConnectionPool - reconnecting to postgresql

On Jun 11, 2014, at 1:23 AM, Glyph wrote: > You might want to just try using SQLAlchemy via > if you don't want to do your own > thread-pool management as buildbot appears to be. thanks. i think we might need to go the buildbot route. we mainly use Sql

[Twisted-Python] adbapi.ConnectionPool - reconnecting to postgresql

I had to restart postgresql , and that triggered some endless errors in my logs. i don't have anything that can handle a suddenly dropped / resumed connection. does anyone have a recommendation for trying to reconnect or handle this in general from what I can tell from my logs... When I catch

Re: [Twisted-Python] possible error in twisted app

approach On Jan 18, 2014, at 7:17 PM, GMail wrote: > Hi Jonathan, > > On Jan 17, 2014, at 6:22 PM, Jonathan Vanasco wrote: > >> the following is a rough composite of what is going on. if anyone sees an >> obvious fix, i'd be greatly appreciative. > &

[Twisted-Python] possible error in twisted app

a portion of my twisted app is having some problems. i think i figured out the issue -- but if I'm right.. i'll be a bit lost. this portion of the app is essentially a web scraper. it grabs a batch of X urls from a data broker , and then updates a database with data about the URL ( which eith

[Twisted-Python] strategies for tracking down memory issues

some recent changes to a very-happy twisted daemon have resulted in a process that grows in memory until it crashes the box. boo! looking through the code and logs, i'm wondering if i''ve coded things in such a way that defferds or deferrd lists are somehow not getting cleaned up if an unhand

Re: [Twisted-Python] question about subprocess

gt; > > > On Sep 26, 2013, at 6:51 PM, Jonathan Vanasco > > wrote: > > > > Could you write up why you feel it's "annoying"? Maybe we can improve > > spawnProcess so it works better for you in the future. > > I'm also interested in wha

[Twisted-Python] question about subprocess

Quick question about (sub)processes in twisted : how bad is it to use the subprocess module ? ( in general / in this application ) I have the following setup : - Twisted Daemon that does 5 things (very well!) - A new requirement -- to execute 2 scripts on the commandli

Re: [Twisted-Python] SQL ORM for Twisted & PostgreSQL?

It would be really beneficial if this were something that fully works with twisted, but is not dependent on it. For example, I have a "Project" that mostly uses SqlAlchemy. It started out in Pylons, new development is on Pyramid and there are additional tasks in Celery + some more in Twisted.

Re: [Twisted-Python] reload / restart best practices ?

On Jun 12, 2013, at 4:31 PM, Glyph wrote: > I'm not sure what "autoload" is. I am assuming it's some kind of > auto-re-loading technology though. The earlier discussions talked about Django's autoload, which reloads the entire process on a config file change. > You do want to re-start your p

[Twisted-Python] reload / restart best practices ?

I have a twisted application that seems to occasionally jam up. It dispatches a lot of work via DeferToThread, and in some instances it appears that the specific thread pool locks up ( haven't found out why yet ). I'm fairly certain that the problem is stemming from getting HTTP documents ( t

[Twisted-Python] reload / restart best practices ?

I have a twisted application that seems to occasionally jam up. It dispatches a lot of work via DeferToThread, and in some instances it appears that the specific thread pool locks up ( haven't found out why yet ). I'm fairly certain that the problem is stemming from getting HTTP documents ( t

Re: [Twisted-Python] github, again

On Jun 6, 2013, at 12:52 PM, Phil Mayers wrote: > On 06/06/13 17:08, Jonathan Vanasco wrote: > >> the only downside to git, is that once something goes onto the >> server... it's there for good. it's possible to rebase a repo back >> to a specific commit , th

Re: [Twisted-Python] github, again

On Jun 4, 2013, at 2:49 AM, tds...@gmail.com wrote: > Hi, > > what about Bitbucket (www.bitbucket.org) and mercurial ? > > Don't they provide the same features ? > > I'm asking because we are in Python land. ;-) BitBucket isn't as slick as GitHub. Mercurial isn't as well known, and the stora

[Twisted-Python] Recommended daemon monitoring tools for ubuntu

Hi, i'm running a twisted daemon on ubuntu It seems to get shut down by the system periodically. Logs show that the kernel (or something) kills it for taking up too much memory. I definitely have coding issues in there that i'm working on to address this. Does anyone have a suggestion for to

Re: [Twisted-Python] unhandled exception in deferred has a delayed logging ?

uot;proper", which is how i found this out in the first place. On Sep 26, 2012, at 8:20 PM, Itamar Turner-Trauring wrote: > On Wed, Sep 26, 2012 at 7:46 PM, Jonathan Vanasco > wrote: > while migrating some functionality to a deferred callback chain and debugging > my

[Twisted-Python] unhandled exception in deferred has a delayed logging ?

while migrating some functionality to a deferred callback chain and debugging my app, i noticed this oddity... if a function doesn't have the proper args/kwargs definition, no error is raised and everything just hangs for example, if i have these 2 class functions in an internet.TimerService m

Re: [Twisted-Python] deferred releasing too many tokens

oooh! thanks. I'm going to try and reimplement with that right now! On Sep 26, 2012, at 5:09 PM, Stephen Thorne wrote: > This is a better way of using DeferredSemaphore: > >def queue_thread( self , data=None ): >self.queued_data= data >return self.semaphoreService.run( self

[Twisted-Python] deferred releasing too many tokens

I'm in the process of rewriting a web spider (originally in twisted circa 2005) , and keep running into an issue with deferreds releasing too many tokens. i've been playing around with this all day, and can't seem to shake this problem. i'm guessing that i designed the application logic wrong