And the mobile frontend will be disabled permanently with this patch: https://review.openstack.org/285672 Disable mobile frontend
M. On Sat, Feb 27, 2016 at 1:39 PM Marton Kiss <marton.k...@gmail.com> wrote: > I made some investigation, and it seems to be that the spam pages are > created by accounts registered with password accounts, and the launchpad > openid auth is not affected at all. > > So the spam script is creating accounts like this: > mysql> select * from user where user_name = 'CedricJamieson'\G; > *************************** 1. row *************************** > user_id: 7494 > user_name: CedricJamieson > user_real_name: Cedric Jamieson > user_password: > :pbkdf2:sha256:10000:128:Mlo9tdaP+38niZrrEka7Ow==:jEVnrTclkwIpE1RzJywDlrSvkY5G3idYwOwYRkv5O0J/MSHjY+gdhtKmArQ53v6/w7o8E1wXb2QOR6HdL5TPfOI1bswS/fYXVVYjPjkEEdxqZ8q9L5p2f3N6rEYpMfT5tk+wDiy+j5aimrHrGSga44hndAHgX9/SnqUyxlutDVY= > user_newpassword: > user_newpass_time: NULL > user_email: balashkina.evdok...@mail.ru > user_touched: 20160227052454 > user_token: 7c39e44e849fb0e2bfae8790d6cc1379 > user_email_authenticated: NULL > user_email_token: be963ac3bd43e70ff2f323063c61e320 > user_email_token_expires: 20160305052441 > user_registration: 20160227052441 > user_editcount: 2 > user_password_expires: NULL > > The user_password field is always filled with a value, meanwhile this > field of non-infected user accounts with openid logins is empty. > We have 423 total accounts with passwords: > mysql> select count(*) from user where user_password != ''; > +----------+ > | count(*) | > +----------+ > | 423 | > +----------+ > 1 row in set (0.00 sec) > > Mediawiki logs-in the newly created users without any preliminary email > confirmation, right after the registration. I disabled the standard user > login page, as described here: > https://www.mediawiki.org/wiki/Manual:Special_pages#Disabling_Special:UserLogin_and_Special:UserLogout_pages > > And I made this patch to make it permanent: > https://review.openstack.org/285669 Disable standard password based auth > > Just for the record, the last spam user account: > 7536 | EarthaChester22 > > Marton > > > On Sat, Feb 27, 2016 at 8:31 AM Marton Kiss <marton.k...@gmail.com> wrote: > >> Hi, >> >> I created the following patch, infra cores must approve that: >> https://review.openstack.org/285641 Add ssh key of JP Maxwell to wiki.o.o >> >> Marton >> >> On Sat, Feb 27, 2016 at 6:41 AM JP Maxwell <j...@tipit.net> wrote: >> >>> Marton has SSH access and applied a patch earlier today. It appears the >>> spam continues to flow: >>> >>> >>> https://wiki.openstack.org/wiki/40_Thoughts_Of_Using_Open_Shelves_On_A_Kitchen >>> >>> Marton let me know if you can look at it some more or Infra if you want >>> to give me SSH I'll do so as well in the morning (public key attached). >>> >>> >>> >>> ssh-rsa >>> AAAAB3NzaC1yc2EAAAABIwAAAQEA2b5I7Yff9FCrtRmSjpILUePi54Vbc8zqJTbzrIAQZGFLBi3xd2MLlhV5QVgpDBC9H3lGjbdnc81D3aFd3HwHT4dvvvyedT12PR3VDEpftdW84vw3jzdtALcayOQznjbGnScwvX5SgnRhNxuX9Rkh8qNvOsjYPUafRr9azkQoomJFkdNVI4Vb5DbLhTpt18FPeOf0UuqDt/J2tHI4SjZ3kjzr7Nbwpg8xGgANPNE0+2pJbwCA8YDt4g3bzfzvVafQs5o9Gfc9tudkR9ugQG1M+EWCgu42CleOwMTd/rYEB2fgNNPsZAWqwQfdPajVuk70EBKUEQSyoA09eEZX+xJN9Q== >>> jpmax...@tipit.net >>> >>> >>> >>> >>> J.P. Maxwell / tipit.net <http://www.tipit.net> >>> >>> >>> On Fri, Feb 26, 2016 at 12:09 PM, Jimmy McArthur <ji...@openstack.org> >>> wrote: >>> >>>> Super thankful for all the folks that have jumped in over the last >>>> couple of days to help with the puppetization, etc... I just feel like >>>> we're taking a very wrong approach here. >>>> >>>> Paul Belanger wrote: >>>> >>>> Right, and I don't have an issue with that approach. Based on the work we >>>> did >>>> yesterday, anybody can do that via our workflow. Please submit a patch to >>>> puppet-mediawiki[1] and ping an infra-root in #openstack-infra IRC. >>>> >>>> What I'm proposing is the workflow is really meant for software, not >>>> for web applications. It's tedious and time consuming when what's needed >>>> here is a set of tests on the server. Submitting a patch, waiting for a +1, >>>> then getting on IRC to find someone with access (and time) to paste the >>>> logs is a pretty time consuming process for what should be a series of >>>> rapid-fire changes/fixes on the server. Especially when we're dealign with >>>> an active attack. >>>> >>>> >>>> We can then have somebody look at the logs. I think it is more about >>>> scheduling >>>> the task since more infra-root as travling back from the mid-cycle last >>>> night >>>> and today. >>>> >>>> Right, this is my point. This has been going on for 3 weeks (or more). >>>> Tom Fifeldt was asking for help without response. And here we are through >>>> another week and no closer to stemming the flow. >>>> >>>> I'm fully aware what I'm proposing goes against what Infra and the >>>> OpenStack workflow is all about, but I'd ask you all to look at this from a >>>> web development perspective instead of a software development perspective. >>>> >>>> Jimmy >>>> >>>> >>>> Last email from me, just on a plane. Will follow up when I land. >>>> >>>> [1] https://git.openstack.org/cgit/openstack-infra/puppet-mediawiki >>>> >>>> J.P. Maxwell | tipit.net [http://tipit.net] | fibercove.com >>>> [http://www.fibercove.com] >>>> On Fri, Feb 26, 2016 at 11:25 AM, Paul Belanger <pabelan...@redhat.com> >>>> <pabelan...@redhat.com> >>>> wrote: >>>> On Fri, Feb 26, 2016 at 11:08:18AM -0600, Jimmy McArthur wrote: >>>> >>>> Given the state of the wiki a the moment, I think taking the quickest path >>>> to get it fixed would be prudent. Is there a way we can get JP root access >>>> to this server, even temporarily? We get 25% of our website traffic (2 >>>> million visitors) to the wiki. I realize we're all after the same thing, >>>> >>>> but >>>> >>>> spammers are not going to hit the dev environment, so there's really no >>>> >>>> way >>>> >>>> to tell if teh problem is fixed without actually working directly on the >>>> production machine. This should be a 30 minute fix. >>>> >>>> >>>> I am still unclear what the 30min fix is. If really 30mins, then it >>>> shouldn't be >>>> hard to get the fix into our workflow. Could somebody please elaborate. >>>> >>>> If we are talking about deploying new versions of php or mediawiki >>>> manually, >>>> I >>>> not be in-favor of this. To me, while the attack sucks, we should be >>>> working >>>> on >>>> 2 fronts. Getting the help needed to mitigate the attack, then adding the >>>> changes into -infra workflow in parallel. >>>> >>>> >>>> I realize there is a lot of risk in giving ssh access to infra machines, >>>> >>>> but >>>> >>>> I think it's worth taking a look at either putting this machine in a place >>>> where a different level of admin could access it without giving away the >>>> keys to the entire OpenStack infrastructure or figuring out a way to set >>>> >>>> up >>>> >>>> credentials with varying levels of access. >>>> >>>> >>>> As a note, all the work I've been doing to help with the attack hasn't >>>> require >>>> SSH access for me to wiki.o.o. I did need infra-root help to expose our >>>> configuration safely. I'd rather take some time to see what the fixes are, >>>> having infra-root apply changes, then move them into puppet. >>>> >>>> It also has been discussed to simply disable write access to the wiki if we >>>> really want spamming to stop, obviously that will affect normal usage. >>>> >>>> >>>> Jimmy >>>> >>>> Paul Belanger wrote: >>>> >>>> On Fri, Feb 26, 2016 at 10:12:12AM -0600, JP Maxwell wrote: >>>> >>>> But if you wanted to upgrade everything, remove the mobile view >>>> >>>> extension, >>>> >>>> test in a dev/staging environment then deploy to production fingers >>>> crossed, I think that would be a valid approach as well. >>>> >>>> >>>> Current review up[1]. I'll launch a node tonight / tomorrow locally to >>>> >>>> see >>>> how >>>> >>>> puppet reacts. I suspect there will be some issues. >>>> >>>> If infra-roots are fine with this approach, we can use that box to test >>>> >>>> against. >>>> >>>> [1] https://review.openstack.org/#/c/285405/ >>>> >>>> J.P. Maxwell | tipit.net | fibercove.com >>>> On Feb 26, 2016 10:08 AM, "JP Maxwell"<j...@tipit.net> <j...@tipit.net> >>>> wrote: >>>> >>>> >>>> Plus one except in this case it is much easier to know if our efforts >>>> >>>> are >>>> >>>> working on production because the spam either stops or not. >>>> >>>> J.P. Maxwell | tipit.net | fibercove.com >>>> On Feb 26, 2016 9:48 AM, "Paul Belanger"<pabelan...@redhat.com> >>>> <pabelan...@redhat.com> wrote: >>>> >>>> >>>> On Fri, Feb 26, 2016 at 09:18:00AM -0600, JP Maxwell wrote: >>>> >>>> I really think you might consider the option that there is a >>>> >>>> vulnerability >>>> >>>> in one of the extensions. If that is the case black listing IPs will >>>> >>>> be >>>> >>>> an >>>> >>>> ongoing wild goose chase. >>>> >>>> I think this would be easily proven or disproven by making the questy >>>> question impossible and see if the spam continues. >>>> >>>> >>>> We'll have to let an infra-root make that call. Since nobody would be >>>> able to >>>> use the wiki. Honestly, I'd rather spend the time standing up a mirror >>>> >>>> dev >>>> >>>> instance for us to work on, rather then production. >>>> >>>> >>>> J.P. Maxwell | tipit.net | fibercove.com >>>> On Feb 26, 2016 9:12 AM, "Paul Belanger"<pabelan...@redhat.com> >>>> <pabelan...@redhat.com> >>>> >>>> wrote: >>>> >>>> On Thu, Feb 25, 2016 at 08:10:34PM -0800, Elizabeth K. Joseph wrote: >>>> >>>> On Thu, Feb 25, 2016 at 6:35 AM, Jeremy Stanley<fu...@yuggoth.org> >>>> <fu...@yuggoth.org> >>>> >>>> wrote: >>>> >>>> On 2016-02-25 02:46:13 -0600 (-0600), JP Maxwell wrote: >>>> >>>> Please be aware that you can now create accounts under the mobile >>>> view in the wiki native user table. I just created an account for >>>> JpMaxMan. Not sure if this matters but wanted to make sure you >>>> were aware. >>>> >>>> Oh, yes I think having a random garbage question/answer was in >>>> >>>> fact >>>> >>>> previously preventing account creation under the mobile view. We >>>> probably need a way to disable mobile view account creation as it >>>> bypasses OpenID authentication entirely. >>>> >>>> So that's what it was doing! We'll have to tackle the mobile view >>>> >>>> issue. >>>> >>>> Otherwise, quick update here: >>>> >>>> The captcha didn't appear to help stem the spam tide. We'll want to >>>> explore and start implementing some of the other solutions. >>>> >>>> I did some database poking around today and it does seem like all >>>> >>>> the >>>> >>>> users do have launchpad accounts and email addresses. >>>> >>>> >>>> So, I have a few hours before jumping on my plane and checked into >>>> >>>> this. >>>> >>>> We are >>>> using QuestyCaptcha which according to docs, should almost be >>>> >>>> impossible >>>> >>>> for >>>> spammers to by pass in an automated fashion. So, either our captcha >>>> >>>> is too >>>> >>>> easy, or we didn't set it up properly. I don't have SSH on wiki.o.o >>>> >>>> so >>>> >>>> others >>>> will have to check logs. I did test new pages and edits, and was >>>> >>>> promoted >>>> >>>> by >>>> captcha. >>>> >>>> As a next step, we might need to add additional apache2 >>>> >>>> configuration >>>> >>>> to >>>> >>>> blacklist IPs. I am reading up on that now. >>>> >>>> >>>> -- >>>> Elizabeth Krumbach Joseph || Lyz || pleia2 >>>> >>>> _______________________________________________ >>>> OpenStack-Infra mailing listopenstack-in...@lists.openstack.org >>>> >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >>>> _______________________________________________ >>>> OpenStack-Infra mailing listopenstack-in...@lists.openstack.org >>>> >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >>>> >>>> _______________________________________________ >>>> OpenStack-Infra mailing >>>> listOpenStack-Infra@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >>>> >>>> _______________________________________________ >>>> OpenStack-Infra mailing >>>> listOpenStack-Infra@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >>>> >>>> _______________________________________________ >>>> OpenStack-Infra mailing >>>> listOpenStack-Infra@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >>>> >>>> >>>> >>> _______________________________________________ >>> OpenStack-Infra mailing list >>> OpenStack-Infra@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >>> >>
_______________________________________________ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
