I'm going to get a dinner, but I'll be on irc after, so if I can help somehow, I will be here. #openstack-infra mrmartin
M. On Fri, Feb 26, 2016 at 6:51 PM Paul Belanger <pbela...@redhat.com> wrote: > On phone but patch puppet-mediawiki and enable captcha for all pages. We > only did edit and create > On Feb 26, 2016 10:38 AM, Marton Kiss <marton.k...@gmail.com> wrote: > > I see a ton of incoming post requests: > > POST > /w/index.php?title=Special%3ARunJobs&tasks=jobs&maxjobs=1&sigexpiry=1456508270&signature=571cfb216f944b15d2eee1c0253d08b77003328e > > M. > > On Fri, Feb 26, 2016 at 6:35 PM Marton Kiss <marton.k...@gmail.com> wrote: > >> Oh, I can login. So what we need? >> >> M. >> >> On Fri, Feb 26, 2016 at 6:33 PM JP Maxwell <j...@tipit.net> wrote: >> >>> I think what Jimmy is referring to is what I was suggesting by removing >>> the extensions / making the question impossible to answer. Basically a >>> series of rapid fire changes while tailing the logs and seeing what stops >>> the spam. Once you know what worked then you can submit as an official >>> patch. But being able to quickly try these things on a server actually >>> under attack is the fastest path toward identifying the fix. >>> >>> *J.P. Maxwell* | tipit.net | fibercove.com <http://www.fibercove.com> >>> >>> On Fri, Feb 26, 2016 at 11:25 AM, Paul Belanger <pabelan...@redhat.com> >>> wrote: >>> >>> On Fri, Feb 26, 2016 at 11:08:18AM -0600, Jimmy McArthur wrote: >>> > Given the state of the wiki a the moment, I think taking the quickest >>> path >>> > to get it fixed would be prudent. Is there a way we can get JP root >>> access >>> > to this server, even temporarily? We get 25% of our website traffic (2 >>> > million visitors) to the wiki. I realize we're all after the same >>> thing, but >>> > spammers are not going to hit the dev environment, so there's really >>> no way >>> > to tell if teh problem is fixed without actually working directly on >>> the >>> > production machine. This should be a 30 minute fix. >>> > >>> I am still unclear what the 30min fix is. If really 30mins, then it >>> shouldn't be >>> hard to get the fix into our workflow. Could somebody please elaborate. >>> >>> If we are talking about deploying new versions of php or mediawiki >>> manually, I >>> not be in-favor of this. To me, while the attack sucks, we should be >>> working on >>> 2 fronts. Getting the help needed to mitigate the attack, then adding the >>> changes into -infra workflow in parallel. >>> >>> > I realize there is a lot of risk in giving ssh access to infra >>> machines, but >>> > I think it's worth taking a look at either putting this machine in a >>> place >>> > where a different level of admin could access it without giving away >>> the >>> > keys to the entire OpenStack infrastructure or figuring out a way to >>> set up >>> > credentials with varying levels of access. >>> > >>> As a note, all the work I've been doing to help with the attack hasn't >>> require >>> SSH access for me to wiki.o.o. I did need infra-root help to expose our >>> configuration safely. I'd rather take some time to see what the fixes >>> are, >>> having infra-root apply changes, then move them into puppet. >>> >>> It also has been discussed to simply disable write access to the wiki if >>> we >>> really want spamming to stop, obviously that will affect normal usage. >>> >>> > Jimmy >>> > >>> > Paul Belanger wrote: >>> > >On Fri, Feb 26, 2016 at 10:12:12AM -0600, JP Maxwell wrote: >>> > >>But if you wanted to upgrade everything, remove the mobile view >>> extension, >>> > >>test in a dev/staging environment then deploy to production fingers >>> > >>crossed, I think that would be a valid approach as well. >>> > >> >>> > >Current review up[1]. I'll launch a node tonight / tomorrow locally >>> to see how >>> > >puppet reacts. I suspect there will be some issues. >>> > > >>> > >If infra-roots are fine with this approach, we can use that box to >>> test against. >>> > > >>> > >[1] https://review.openstack.org/#/c/285405/ >>> > > >>> > >>J.P. Maxwell | tipit.net | fibercove.com >>> > >>On Feb 26, 2016 10:08 AM, "JP Maxwell"<j...@tipit.net> wrote: >>> > >> >>> > >>>Plus one except in this case it is much easier to know if our >>> efforts are >>> > >>>working on production because the spam either stops or not. >>> > >>> >>> > >>>J.P. Maxwell | tipit.net | fibercove.com >>> > >>>On Feb 26, 2016 9:48 AM, "Paul Belanger"<pabelan...@redhat.com> >>> wrote: >>> > >>> >>> > >>>>On Fri, Feb 26, 2016 at 09:18:00AM -0600, JP Maxwell wrote: >>> > >>>>>I really think you might consider the option that there is a >>> > >>>>vulnerability >>> > >>>>>in one of the extensions. If that is the case black listing IPs >>> will be >>> > >>>>an >>> > >>>>>ongoing wild goose chase. >>> > >>>>> >>> > >>>>>I think this would be easily proven or disproven by making the >>> questy >>> > >>>>>question impossible and see if the spam continues. >>> > >>>>> >>> > >>>>We'll have to let an infra-root make that call. Since nobody would >>> be >>> > >>>>able to >>> > >>>>use the wiki. Honestly, I'd rather spend the time standing up a >>> mirror dev >>> > >>>>instance for us to work on, rather then production. >>> > >>>> >>> > >>>>>J.P. Maxwell | tipit.net | fibercove.com >>> > >>>>>On Feb 26, 2016 9:12 AM, "Paul Belanger"<pabelan...@redhat.com> >>> wrote: >>> > >>>>> >>> > >>>>>>On Thu, Feb 25, 2016 at 08:10:34PM -0800, Elizabeth K. Joseph >>> wrote: >>> > >>>>>>>On Thu, Feb 25, 2016 at 6:35 AM, Jeremy Stanley< >>> fu...@yuggoth.org> >>> > >>>>>>wrote: >>> > >>>>>>>>On 2016-02-25 02:46:13 -0600 (-0600), JP Maxwell wrote: >>> > >>>>>>>>>Please be aware that you can now create accounts under the >>> mobile >>> > >>>>>>>>>view in the wiki native user table. I just created an account >>> for >>> > >>>>>>>>>JpMaxMan. Not sure if this matters but wanted to make sure you >>> > >>>>>>>>>were aware. >>> > >>>>>>>>Oh, yes I think having a random garbage question/answer was in >>> > >>>>fact >>> > >>>>>>>>previously preventing account creation under the mobile view. >>> We >>> > >>>>>>>>probably need a way to disable mobile view account creation as >>> it >>> > >>>>>>>>bypasses OpenID authentication entirely. >>> > >>>>>>>So that's what it was doing! We'll have to tackle the mobile >>> view >>> > >>>>issue. >>> > >>>>>>>Otherwise, quick update here: >>> > >>>>>>> >>> > >>>>>>>The captcha didn't appear to help stem the spam tide. We'll >>> want to >>> > >>>>>>>explore and start implementing some of the other solutions. >>> > >>>>>>> >>> > >>>>>>>I did some database poking around today and it does seem like >>> all >>> > >>>>the >>> > >>>>>>>users do have launchpad accounts and email addresses. >>> > >>>>>>> >>> > >>>>>>So, I have a few hours before jumping on my plane and checked >>> into >>> > >>>>this. >>> > >>>>>>We are >>> > >>>>>>using QuestyCaptcha which according to docs, should almost be >>> > >>>>impossible >>> > >>>>>>for >>> > >>>>>>spammers to by pass in an automated fashion. So, either our >>> captcha >>> > >>>>is too >>> > >>>>>>easy, or we didn't set it up properly. I don't have SSH on >>> wiki.o.o >>> > >>>>so >>> > >>>>>>others >>> > >>>>>>will have to check logs. I did test new pages and edits, and was >>> > >>>>promoted >>> > >>>>>>by >>> > >>>>>>captcha. >>> > >>>>>> >>> > >>>>>>As a next step, we might need to add additional apache2 >>> configuration >>> > >>>>to >>> > >>>>>>blacklist IPs. I am reading up on that now. >>> > >>>>>> >>> > >>>>>>>-- >>> > >>>>>>>Elizabeth Krumbach Joseph || Lyz || pleia2 >>> > >>>>>>> >>> > >>>>>>>_______________________________________________ >>> > >>>>>>>OpenStack-Infra mailing list >>> > >>>>>>>OpenStack-Infra@lists.openstack.org >>> > >>>>>>> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >>> > >>>>>>_______________________________________________ >>> > >>>>>>OpenStack-Infra mailing list >>> > >>>>>>OpenStack-Infra@lists.openstack.org >>> > >>>>>> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >>> > >>>>>> >>> > > >>> > >_______________________________________________ >>> > >OpenStack-Infra mailing list >>> > >OpenStack-Infra@lists.openstack.org >>> > >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >>> > >>> >>> > _______________________________________________ >>> > OpenStack-Infra mailing list >>> > OpenStack-Infra@lists.openstack.org >>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >>> >>> >>> _______________________________________________ >>> OpenStack-Infra mailing list >>> OpenStack-Infra@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >>> >>> _______________________________________________ >>> OpenStack-Infra mailing list >>> OpenStack-Infra@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >>> >>
_______________________________________________ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
