[Twisted-Python] Fwd: Moving the Twisted-Python and Twisted-Web mailing lists too Python mailman3

[Thomas Grainger]

Thomas Grainger


-- Forwarded message -
From: Thomas Grainger 
Date: Mon, 2 Aug 2021 at 00:17
Subject: Re: Moving the Twisted-Python and Twisted-Web mailing lists
too Python mailman3
To: Mark Sapiro 
Cc: 


I'll look into this and get back to you, thanks so much for your help in this!

On Mon, 2 Aug 2021, 00:16 Mark Sapiro,  wrote:
>
> On 8/1/21 3:41 PM, Thomas Grainger wrote:
> > That sounds great! Do you know what MX configuration I'll need to keep
> > the same email addresses as aliases on the mailing lists?
>
>
> What is the current situation with regard to @twistedmatrix.com
> addresses. Are there addresses there not associated with these Mailman
> lists, and if so what should happen to them. If it's only the Mailman
> lists, you could set an MX for twistedmatrix.com to point to
> mail.python.org and we could teach our Postfix how to deliver that mail.
>
> A better solution from our point of view would be to have a separate MX
> somewhere for twistedmatrix.com and have that MTA relay mail for
> addresses like twisted-python@twistedmatrix.com,
> twisted-...@twistedmatrix.com, and maybe also the -join, -leave,
> -request, etc. addresses to the corresponding @python.org address.
>
> --
> Mark Sapiro The highway is for gamblers,
> San Francisco Bay Area, Californiabetter use your sense - B. Dylan

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Fwd: Moving the Twisted-Python and Twisted-Web mailing lists too Python mailman3

[Thomas Grainger]

Apologies I meant to forward this mail instead:

On 8/1/21 1:16 PM, Thomas Grainger wrote:
> Hello,
>
> The twisted ( https://pypi.org/project/Twisted/ ) web hosting provider is 
> winding
> down and we're looking for a new home.
>
> Would it be possible to move our mailing lists and archives onto the
> https://mail.python.org/mailman3/lists/ infrastructure?
>
> These are the current lists
> https://twistedmatrix.com/cgi-bin/mailman/listinfo


If you can get me copies of Mailman's lists/twisted-python/config.pck
and lists/twisted-web/config.pck configuration files and
archives/private/twisted-python.mbox/twisted-python.mbox and
archives/private/twisted-web.mbox/twisted-web.mbox archive mailboxes, I
can import these on mail.python.org.

The twisted-web list seems pretty inactive currently, but twisted-python
seems fairly active. Ideally we can arrange a time for you to shut down
the lists and get me the files and get the lists up on mail.python.org.
Otherwise, if the lists are still operational while I'm getting the
files and doing the imports, messages and perhaps other changes will be
lost, although it is possible to import additional messages to the
archives after the fact.

I will be available to do this during the next two weeks, but I will not
be available for the rest of August and the first two weeks of September.

Thomas Grainger

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] new success story: Battlehouse Games

[Glyph]

I ran across a PyCon APAC talk about this, and the speaker generously offered 
to write a blurb for our site, which I've posted here: 
https://twistedmatrix.com/trac/wiki/SuccessStories#BattlehouseGames 


> Battlehouse creates and operates massively-multiplayer strategy games for 
> browser and desktop platforms. Our most popular titles are nearly 10 years 
> old and have served over 4 million users.
> 
> Twisted plays an important role in our Python-based server stack. We use 
> Twisted to handle many protocols including HTTP, WebSockets, and SQL. The 
> asynchronous coroutine model helps us deliver low-latency gameplay without 
> adding too much complexity to the codebase. Across 10 years of continuous 
> evolution, Twisted has proven to be the most stable, well-documented, and 
> flexible low-level networking library for Python.
> 
> -- Dan Maas, CTO, Battlehouse Games

Do you have your own success story?  Please contact succ...@twistedmatrix.com 
 - it's always nice to hear from users!

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Moving the Twisted-Python and Twisted-Web mailing lists too Python mailman3

[Glyph]

Great news.  I'd be happy to move the mailing lists over to let someone else 
maintain their infrastructure.  Particularly if we could get off of Mailman 2, 
which has been no fun to maintain.

Many personal email addresses of core team members (including, um, this one) 
are forwarded by our mail hosting provider, Mailgun, and so moving the MX 
records directly is a non-starter, but we could easily point 
lists.twistedmatrix.com  at python.org 
, or we could just use @python.org addresses and be quite 
happy.  twis...@python.org  would be very cool.  We 
could also maintain a forwarding alias for compatibility for a little while, 
but I'd kinda rather deprecate posting to twisted-python@twistedmatrix.com 
 for a couple of other reasons anyway.

Do we have a way to get our pipermail archive URLs (which are linked in 
tickets, in source code, etc) to redirect to the fancy new hyperkitty 
interface, or should those just be static HTML?  (I'm kinda fine with either, 
honestly, although the former would be nice if we can manage it easily.)

I am happy to coordinate off-list about a secure way to exchange these data 
files (or anyone with administrator access to Dornkirk can just do it 
directly).  Please send the announcement of the new address here first so we 
know when to cut over posting, and so that folks aren't surprised to see a new 
list-id.

(At this point I think I'm happy to just nuke twisted-web as part of this 
migration; we can keep the archive up as static HTML for posterity, but it's 
mostly unused at this point, and the distinction doesn't make sense.)

-g

> On Aug 2, 2021, at 12:56 AM, Thomas Grainger  wrote:
> 
> Apologies I meant to forward this mail instead:
> 
> On 8/1/21 1:16 PM, Thomas Grainger wrote:
>> Hello,
>> 
>> The twisted ( https://pypi.org/project/Twisted/ ) web hosting provider is 
>> winding
>> down and we're looking for a new home.
>> 
>> Would it be possible to move our mailing lists and archives onto the
>> https://mail.python.org/mailman3/lists/ infrastructure?
>> 
>> These are the current lists
>> https://twistedmatrix.com/cgi-bin/mailman/listinfo
> 
> 
> If you can get me copies of Mailman's lists/twisted-python/config.pck
> and lists/twisted-web/config.pck configuration files and
> archives/private/twisted-python.mbox/twisted-python.mbox and
> archives/private/twisted-web.mbox/twisted-web.mbox archive mailboxes, I
> can import these on mail.python.org.
> 
> The twisted-web list seems pretty inactive currently, but twisted-python
> seems fairly active. Ideally we can arrange a time for you to shut down
> the lists and get me the files and get the lists up on mail.python.org.
> Otherwise, if the lists are still operational while I'm getting the
> files and doing the imports, messages and perhaps other changes will be
> lost, although it is possible to import additional messages to the
> archives after the fact.
> 
> I will be available to do this during the next two weeks, but I will not
> be available for the rest of August and the first two weeks of September.
> 
> Thomas Grainger
> 
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Moving the Twisted-Python and Twisted-Web mailing lists too Python mailman3

[Thomas Grainger]

> Great news.  I'd be happy to move the mailing lists over to let someone else 
> maintain their infrastructure.  Particularly if we could get off of Mailman 
> 2, which has been no fun to maintain.

> Many personal email addresses of core team members (including, um, this one) 
> are forwarded by our mail hosting provider, Mailgun, and so moving the MX 
> records directly is a non-starter, but we could easily point 
> lists.twistedmatrix.com  at python.org 
> , or we could just use @python.org addresses and be quite 
> happy.  twisted at python.org  would be very 
> cool.  We could also maintain a forwarding alias for compatibility for a 
> little while, but I'd kinda rather deprecate posting to twisted-python at 
> twistedmatrix.com  for a couple 
> of other reasons anyway.
A forwarding alias seems the best choice. twis...@python.org is indeed cool

> Do we have a way to get our pipermail archive URLs (which are linked in 
> tickets, in source code, etc) to redirect to the fancy new hyperkitty 
> interface, or should those just be static HTML?  (I'm kinda fine with either, 
> honestly, although the former would be nice if we can manage it easily.)
I think the redirects would be fairly easy as the URL structure is the same

> I am happy to coordinate off-list about a secure way to exchange these data 
> files (or anyone with administrator access to Dornkirk can just do it 
> directly).  Please send the announcement of the new address here first so we 
> know when to cut over posting, and so that folks aren't surprised to see a 
> new list-id.

I can do magic wormhole

> (At this point I think I'm happy to just nuke twisted-web as part of this 
> migration; we can keep the archive up as static HTML for posterity, but it's 
> mostly unused at this point, and the distinction doesn't make sense.)

I actually think it's better to move twisted-web first and see what it
looks, then mess about there with redirects, then do the main mailing
list. I think the redirects would be fairly easy as the URL structure
is the same


Thomas Grainger

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] bad news about twistedmatrix.com's hosting

[Glyph]

> On Aug 1, 2021, at 2:49 AM, Adi Roiban  > wrote:
> 
> Hi Glyph
> 
> Thanks for sending the message.

As a wise computer once sung, "We do what we must, because we can".

> This is bad, but I am confident that we can do the migration in 60 days, as 
> long as we don't expect to have the exact same Twisted infrastructure.

It's certainly annoying (and I'm very sad to see tummy go, they were great) but 
I think we may be happy with the opportunity to reduce the amount of stuff we 
operate.

> We have a monthly credit of 150 USD from MS to spend on Azure services.

This is great news!  Presumably I … already have access to this, somehow?  Is 
it in the 1password team vault?  How do I see it?  Do we have a commitment from 
MS to actually maintain this credit?

> A quick hack would be to create a new VM in Azure with Ubuntu 18.04 and 
> migrate the existing server to it.

If we need to provision a new server anyway, could we provision it on the most 
recent ubuntu?  Presumably most of the stuff we require will still be on there? 
 If it's a hassle let's fall back to 18.04 but let's not set ourselves up to be 
outdated as long as we're doing this work.

> Do we have other servers hosted with tummy.com , other 
> than dornkirk.twistedmatrix.com  ?

Nope.  We used to have a VPS called Wolfwood, but in an act of nominative 
determinism, it had to die for the greater good.  (It was our source-control 
server, back when that was a thing you had to run yourself.  We shut it down 
when we migrated to github.)

> I will not have too much time to do the migration, but I can ask a sysadmin 
> from my work to help with that and my company will cover that costs.

Thanks Adi, this is very generous.  Let me know if I'm needed to give this 
sysadmin any requisite privileges.

> I think that for now, we should see who is interested with helping with the 
> migration and after 1 week start to work with that team.

Looks like a team is coming together :).

> Maybe we will need to schedule an online meeting, but I hope we can do async 
> work via GitHub Issues (braid) and IRC/Gitter.

Yeah, async seems to be working fine.

> I saw that you have already considered using containers for Twisted infra 
> https://github.com/twisted-infra/braid/issues/265 
> 
Yes. I'd love to do this, and if such a migration reduces effort, then let's do 
it by all means - I just don't want to get caught up in trying to do this and 
ending up with things broken when the clock runs out.

> In parallel with migrating  dornkirk.twistedmatrix.com 
> , we can look at outsourcing some of our 
> services.
> For example
> 
> * Move mailing list to the main Python mailing list server

Looks like this is full steam ahead!

> * Move DNS to a cloud provider (Azure DNS)

I've already pulled the trigger to migrate our DNS to Gandi, since they're our 
registrar and I already know I can provision certificates against their LiveDNS 
API :).  We do still have to get a few other domains off of this infrastructure 
though, I'll do the ones we control (Divunal) and contact folks at the others.

Longer-term, I would like to host some DNS, since I want to make sure that 
twisted.names does not bit-rot.  However, perhaps a prerequisite to setting 
this up again will be (A) containerizing it and (B) providing something like 
Sentry for us to track exceptions, so that the dog-fooding here is interesting 
and not just a tedious obligation that produces gigabytes full of 
tracback-laden log files that nobody ever looks at or fixes.

> * Move Trac Wiki to GitHub Wiki

The trac wiki's main purpose is not really to serve as a wiki-style editable 
document repository, but rather to host the website.  I think migrating it to 
something like Pelican, or some other static-site management thing might make 
more sense.

> * Move Trac Tickets to GitHub Issues

Sure, sure, as soon as we figure out a way to preserve all the links :).

> * Move static file hosting to GitHub pages

I wonder if it would make sense for www.twistedmatrix.com 
 itself to be hosted by 

> * Move highscore to a container and update it to pull info from GitHub hooks

+1.  This thing might be interesting 
> in that it might be a better 
starting point than our own code (which is not even open source because, among 
other reasons, it is such trash) this version gets the point of the scoring 
wrong, which is to weight 

> * IRC bot to some sort of integration to Gitter and then IRC bridge

Given that this is already broken right now, I'd say "don't bother migrating 
the IRC bot right away".  This also means we don't need amptrac, and we can 
rebuild the IRC bot in a more sustainable way, inside a container, later

> * Retir

Re: [Twisted-Python] Moving the Twisted-Python and Twisted-Web mailing lists too Python mailman3

[Glyph]

> On Aug 2, 2021, at 1:10 AM, Thomas Grainger  wrote:
> 
> I actually think it's better to move twisted-web first and see what it
> looks, then mess about there with redirects, then do the main mailing
> list. I think the redirects would be fairly easy as the URL structure
> is the same

Sounds like a good plan.

Speaking of using testing areas rather than the most heavily used production 
stuff:

One thing that perhaps bears mentioning here is that we do have a second domain 
name that I bought at some expense for the project - in part to celebrate its 
20th anniversary at … PyCon 2020 (sob): twisted.org .  If 
this minor emergency is sparking anyone's interest in slowly building some new 
infrastructure (like a new website) on clean, modern infra, without the 
pressure to keep something working continuously, we could host it all there and 
cut over twistedmatrix.com  to be a redirect at the 
appropriate time.

-g

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] bad news about twistedmatrix.com's hosting

[Adi Roiban]

On Sun, 1 Aug 2021 at 10:29, Evilham  wrote:

> \o
>
> On dg., ag. 01 2021, Glyph wrote:
>
> > Hello Twistors,
> >
> > Our venerable hosting provider, tummy.com ,
> > will be terminating their business operations on September 30,
> > 2021.
> >
>  datetime.date(2021, 9, 30) - datetime.date.today()
> > datetime.timedelta(days=60)
> >
> > In addition to being a bit personally sad - Tummy has been an
> > avid supporter of the Twisted community since the very beginning
> > - this means we have just about 60 days to fully migrate our
> > website, mailing list, and DNS's hosting from Dornkirk, the
> > machine where it's been running continuously since 2013, to ...
> > something else.
>
>
> it is sad indeed when friendly businesses terminate operations,
> but Twisted will go on :-).
>
>
> > There are 2 problems here:
> >
> > We need a sponsor to help us find a new hosting or cloud
> > provider where things can be hosted.  We do pay tummy currently,
> > and we can probably ask the SFC to pay a different hosting
> > provider a similar price for equivalent services, but ideally
> > we'd find someone willing to donate something so as not to put a
> > drain on those resources.
>
>
> Let's define this so I and others can ask around with various
> infra providers:
>
> - What is the service currently? (e.g. VPS/Physical? Which
>   resources?)
>

We only have a single server. It is  called "dornkirk"
I don't know if it's a VM or bare-metal... but it shouldn't matter.

- Are the resources allocated adequate? Can/should they be
>   increased/reduced?
>

I don't understand this question. Sorry.
Can you please rephrase ? Thanks.


> - Is currently something on the "wishlist" for this service? (e.g.
>   IPv6 support!)
>

Many things are on the wishlist.
The priority is to have the services up and running, even after dornkirk is
dead.


> - Even if the goal is and should be for that expense to disappear
>   or go down significantly: roughly what is the compensation being
>   offered for this service? (it might be easier to convince
>   someone to offer it at cost/partial compensation than it is to
>   offer it for free)
>

I don't know what is the current cost to cover the expenses for dornkirk.



> - Is location important? E.g. if I asked around and it were
>   fruitful, it'd be EU or Switzerland-based and not US-based as it
>   is right now.
>

No. Location is not important.


>
> > I will need significant help with the operational aspects of the
> > migration.  I typically do a bunch of behind-the-scenes system
> > administration for twistedmatrix.com 
> > to keep the whole thing clunking along, but after my brain has
> > been fried by a year and a half's worth of pandemic stress, I
> > simply don't have the time, energy, or mental capacity to do
> > this.  I know I've asked for help before and rarely received
> > any, but if I have to do this entirely by myself, the most
> > likely outcome is that I'll migrate DNS to some free cloud thing
> > so at least my email address doesn't break (I migrated SMTP, and
> > thereby everything associated with personal email addresses to
> > Mailgun some years ago, so Tummy is not in that loop for that),
> > and the website (and this mailing list) will simply stop working
> > sometime between now and October.  So if you'd like to keep
> > www.twistedmatrix.com ,
> > speed.twistedmatrix.com , our
> > IRC bot (which is offline anyway right now due to Freenode
> > shutting down), https://twistedmatrix.com/highscores/
> > , this mailing list, or
> > our dogfooding instance for Twisted Names DNS, please volunteer
> > so we can start to coordinate.
>
>
> Thank you for both doing this so long and so far and for being
> straightforward about not being able to do it as much now.
>
> I'd like to help and would be particularly (aka more intensively)
> able to do so during August, even if that's just helping document
> what needs to be done and how.
> On a day-to-day basis I do some other sysadmin work and if
> necessary would be able to accommodate taking care of
> Twisted-related things into that.
>

To coordinate migration and also day-to-day Twisted project things we use
the IRC Libera channel #twisted-dev

If you don't like IRC ... we are also experiencing with Gitter/Matrix
https://gitter.im/twisted/twisted

I start this to-do list https://github.com/twisted-infra/braid/issues/277


> This is the documentation that I am aware of:
> https://twistedmatrix.com/trac/wiki/Infrastructure


There is not much documentation... we need to rely on memories :)
So don't bother to search for documentation, and just ask :)


>
> Particularly braid: https://github.com/twisted-infra/braid
>
>
Yes. braid is our deployment as code thing... is the best we have in terms
of documenting the infrastructure.

"braid" is (was?) the Twisted customization on top of fabric