On Fri, Feb 26, 2016 at 11:29:31AM -0600, JP Maxwell wrote: > I think what Jimmy is referring to is what I was suggesting by removing the > extensions / making the question impossible to answer. Basically a series of > rapid fire changes while tailing the logs and seeing what stops the spam. > Once > you know what worked then you can submit as an official patch. But being > able to > quickly try these things on a server actually under attack is the fastest > path > toward identifying the fix. > Right, and I don't have an issue with that approach. Based on the work we did yesterday, anybody can do that via our workflow. Please submit a patch to puppet-mediawiki[1] and ping an infra-root in #openstack-infra IRC.
We can then have somebody look at the logs. I think it is more about scheduling the task since more infra-root as travling back from the mid-cycle last night and today. Last email from me, just on a plane. Will follow up when I land. [1] https://git.openstack.org/cgit/openstack-infra/puppet-mediawiki > J.P. Maxwell | tipit.net [http://tipit.net] | fibercove.com > [http://www.fibercove.com] > On Fri, Feb 26, 2016 at 11:25 AM, Paul Belanger <pabelan...@redhat.com> > wrote: > On Fri, Feb 26, 2016 at 11:08:18AM -0600, Jimmy McArthur wrote: > > Given the state of the wiki a the moment, I think taking the quickest path > > to get it fixed would be prudent. Is there a way we can get JP root access > > to this server, even temporarily? We get 25% of our website traffic (2 > > million visitors) to the wiki. I realize we're all after the same thing, > but > > spammers are not going to hit the dev environment, so there's really no > way > > to tell if teh problem is fixed without actually working directly on the > > production machine. This should be a 30 minute fix. > > > I am still unclear what the 30min fix is. If really 30mins, then it > shouldn't be > hard to get the fix into our workflow. Could somebody please elaborate. > > If we are talking about deploying new versions of php or mediawiki manually, > I > not be in-favor of this. To me, while the attack sucks, we should be working > on > 2 fronts. Getting the help needed to mitigate the attack, then adding the > changes into -infra workflow in parallel. > > > I realize there is a lot of risk in giving ssh access to infra machines, > but > > I think it's worth taking a look at either putting this machine in a place > > where a different level of admin could access it without giving away the > > keys to the entire OpenStack infrastructure or figuring out a way to set > up > > credentials with varying levels of access. > > > As a note, all the work I've been doing to help with the attack hasn't > require > SSH access for me to wiki.o.o. I did need infra-root help to expose our > configuration safely. I'd rather take some time to see what the fixes are, > having infra-root apply changes, then move them into puppet. > > It also has been discussed to simply disable write access to the wiki if we > really want spamming to stop, obviously that will affect normal usage. > > > Jimmy > > > > Paul Belanger wrote: > > >On Fri, Feb 26, 2016 at 10:12:12AM -0600, JP Maxwell wrote: > > >>But if you wanted to upgrade everything, remove the mobile view > extension, > > >>test in a dev/staging environment then deploy to production fingers > > >>crossed, I think that would be a valid approach as well. > > >> > > >Current review up[1]. I'll launch a node tonight / tomorrow locally to > see > how > > >puppet reacts. I suspect there will be some issues. > > > > > >If infra-roots are fine with this approach, we can use that box to test > against. > > > > > >[1] https://review.openstack.org/#/c/285405/ > > > > > >>J.P. Maxwell | tipit.net | fibercove.com > > >>On Feb 26, 2016 10:08 AM, "JP Maxwell"<j...@tipit.net> wrote: > > >> > > >>>Plus one except in this case it is much easier to know if our efforts > are > > >>>working on production because the spam either stops or not. > > >>> > > >>>J.P. Maxwell | tipit.net | fibercove.com > > >>>On Feb 26, 2016 9:48 AM, "Paul Belanger"<pabelan...@redhat.com> wrote: > > >>> > > >>>>On Fri, Feb 26, 2016 at 09:18:00AM -0600, JP Maxwell wrote: > > >>>>>I really think you might consider the option that there is a > > >>>>vulnerability > > >>>>>in one of the extensions. If that is the case black listing IPs will > be > > >>>>an > > >>>>>ongoing wild goose chase. > > >>>>> > > >>>>>I think this would be easily proven or disproven by making the questy > > >>>>>question impossible and see if the spam continues. > > >>>>> > > >>>>We'll have to let an infra-root make that call. Since nobody would be > > >>>>able to > > >>>>use the wiki. Honestly, I'd rather spend the time standing up a mirror > dev > > >>>>instance for us to work on, rather then production. > > >>>> > > >>>>>J.P. Maxwell | tipit.net | fibercove.com > > >>>>>On Feb 26, 2016 9:12 AM, "Paul Belanger"<pabelan...@redhat.com> > wrote: > > >>>>> > > >>>>>>On Thu, Feb 25, 2016 at 08:10:34PM -0800, Elizabeth K. Joseph wrote: > > >>>>>>>On Thu, Feb 25, 2016 at 6:35 AM, Jeremy Stanley<fu...@yuggoth.org> > > >>>>>>wrote: > > >>>>>>>>On 2016-02-25 02:46:13 -0600 (-0600), JP Maxwell wrote: > > >>>>>>>>>Please be aware that you can now create accounts under the mobile > > >>>>>>>>>view in the wiki native user table. I just created an account for > > >>>>>>>>>JpMaxMan. Not sure if this matters but wanted to make sure you > > >>>>>>>>>were aware. > > >>>>>>>>Oh, yes I think having a random garbage question/answer was in > > >>>>fact > > >>>>>>>>previously preventing account creation under the mobile view. We > > >>>>>>>>probably need a way to disable mobile view account creation as it > > >>>>>>>>bypasses OpenID authentication entirely. > > >>>>>>>So that's what it was doing! We'll have to tackle the mobile view > > >>>>issue. > > >>>>>>>Otherwise, quick update here: > > >>>>>>> > > >>>>>>>The captcha didn't appear to help stem the spam tide. We'll want to > > >>>>>>>explore and start implementing some of the other solutions. > > >>>>>>> > > >>>>>>>I did some database poking around today and it does seem like all > > >>>>the > > >>>>>>>users do have launchpad accounts and email addresses. > > >>>>>>> > > >>>>>>So, I have a few hours before jumping on my plane and checked into > > >>>>this. > > >>>>>>We are > > >>>>>>using QuestyCaptcha which according to docs, should almost be > > >>>>impossible > > >>>>>>for > > >>>>>>spammers to by pass in an automated fashion. So, either our captcha > > >>>>is too > > >>>>>>easy, or we didn't set it up properly. I don't have SSH on wiki.o.o > > >>>>so > > >>>>>>others > > >>>>>>will have to check logs. I did test new pages and edits, and was > > >>>>promoted > > >>>>>>by > > >>>>>>captcha. > > >>>>>> > > >>>>>>As a next step, we might need to add additional apache2 > configuration > > >>>>to > > >>>>>>blacklist IPs. I am reading up on that now. > > >>>>>> > > >>>>>>>-- > > >>>>>>>Elizabeth Krumbach Joseph || Lyz || pleia2 > > >>>>>>> > > >>>>>>>_______________________________________________ > > >>>>>>>OpenStack-Infra mailing list > > >>>>>>>OpenStack-Infra@lists.openstack.org > > > >>>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra > > >>>>>>_______________________________________________ > > >>>>>>OpenStack-Infra mailing list > > >>>>>>OpenStack-Infra@lists.openstack.org > > > >>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra > > >>>>>> > > > > > >_______________________________________________ > > >OpenStack-Infra mailing list > > >OpenStack-Infra@lists.openstack.org > > >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra > > > > > _______________________________________________ > > OpenStack-Infra mailing list > > OpenStack-Infra@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra > > > _______________________________________________ > OpenStack-Infra mailing list > OpenStack-Infra@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra _______________________________________________ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
