Elizabeth
I hope you feel better.
Just FYI, this is going full force in IRC right now. I’ve bowed out as
the approach I was suggesting didn’t get traction.
I proposed to manually iterate on this to confirm precisely which change
solves the spam problem. Once that has been identified we can revert
and come up with a proper patch. Right now the assumption is that
disabling manual accounts will solve the problem (and it might). As a
result the team is trying to solve for the consequences of not having
manual accounts. Some bots currently use manual accounts among other
issues. If the assumption is correct, these efforts will be worth it.
However, if it isn’t it will have been a great waste of energy.
In any case have a good weekend everyone. I’m off to eat some delicious
central Texas BBQ!
*J.P. Maxwell* | tipit.net <http://tipit.net> | fibercove.com
<http://www.fibercove.com>
On Sat, Feb 27, 2016 at 10:15 AM, Elizabeth K. Joseph
<l...@princessleia.com> wrote:
We'll be getting together on Monday around 1700 UTC to work through
this together in a debug session in #openstack-infra (I'm too sick
this weekend, plus we need a time when more infra-root folks with
the institutional knowledge are around).
On Feb 27, 2016 05:37, "Marton Kiss" <marton.k...@gmail.com
<mailto:marton.k...@gmail.com>> wrote:
Yeah, the Settings.php was overriden by the latest puppet run.
We need to wait for some infra guys to approve my patches and
make it permanent:
https://review.openstack.org/285669 Disable standard password
based auth
https://review.openstack.org/285672 Disable mobile frontend
M.
On Sat, Feb 27, 2016 at 2:27 PM JP Maxwell <j...@tipit.net
<mailto:j...@tipit.net>> wrote:
FYI. Still seeing the mobile view...
J.P. Maxwell | tipit.net <http://tipit.net> | fibercove.com
<http://fibercove.com>
On Feb 27, 2016 6:53 AM, "Marton Kiss"
<marton.k...@gmail.com <mailto:marton.k...@gmail.com>> wrote:
Yes, applied them manually. Let's wait a few hours, and
check for new spam content / user accounts.
M.
JP Maxwell <j...@tipit.net <mailto:j...@tipit.net>>
(időpont: 2016. febr. 27., Szo, 13:50) ezt írta:
Cool. Are these applied? Any indication it has
stopped the spam? Should we clear out these non
launchpad accounts from the DB?
J.P. Maxwell | tipit.net <http://tipit.net> |
fibercove.com <http://fibercove.com>
On Feb 27, 2016 6:47 AM, "Marton Kiss"
<marton.k...@gmail.com
<mailto:marton.k...@gmail.com>> wrote:
And the mobile frontend will be disabled
permanently with this patch:
https://review.openstack.org/285672 Disable
mobile frontend
M.
On Sat, Feb 27, 2016 at 1:39 PM Marton Kiss
<marton.k...@gmail.com
<mailto:marton.k...@gmail.com>> wrote:
I made some investigation, and it seems to
be that the spam pages are created by
accounts registered with password accounts,
and the launchpad openid auth is not
affected at all.
So the spam script is creating accounts like
this:
mysql> select * from user where user_name =
'CedricJamieson'\G;
*************************** 1. row
***************************
user_id: 7494
user_name: CedricJamieson
user_real_name: Cedric Jamieson
user_password:
:pbkdf2:sha256:10000:128:Mlo9tdaP+38niZrrEka7Ow==:jEVnrTclkwIpE1RzJywDlrSvkY5G3idYwOwYRkv5O0J/MSHjY+gdhtKmArQ53v6/w7o8E1wXb2QOR6HdL5TPfOI1bswS/fYXVVYjPjkEEdxqZ8q9L5p2f3N6rEYpMfT5tk+wDiy+j5aimrHrGSga44hndAHgX9/SnqUyxlutDVY=
user_newpassword:
user_newpass_time: NULL
user_email: balashkina.evdok...@mail.ru
<mailto:balashkina.evdok...@mail.ru>
user_touched: 20160227052454
user_token: 7c39e44e849fb0e2bfae8790d6cc1379
user_email_authenticated: NULL
user_email_token:
be963ac3bd43e70ff2f323063c61e320
user_email_token_expires: 20160305052441
user_registration: 20160227052441
user_editcount: 2
user_password_expires: NULL
The user_password field is always filled
with a value, meanwhile this field of
non-infected user accounts with openid
logins is empty.
We have 423 total accounts with passwords:
mysql> select count(*) from user where
user_password != '';
+----------+
| count(*) |
+----------+
| 423 |
+----------+
1 row in set (0.00 sec)
Mediawiki logs-in the newly created users
without any preliminary email confirmation,
right after the registration. I disabled the
standard user login page, as described here:
https://www.mediawiki.org/wiki/Manual:Special_pages#Disabling_Special:UserLogin_and_Special:UserLogout_pages
And I made this patch to make it permanent:
https://review.openstack.org/285669 Disable
standard password based auth
Just for the record, the last spam user account:
7536 | EarthaChester22
Marton
On Sat, Feb 27, 2016 at 8:31 AM Marton Kiss
<marton.k...@gmail.com
<mailto:marton.k...@gmail.com>> wrote:
Hi,
I created the following patch, infra
cores must approve that:
https://review.openstack.org/285641 Add
ssh key of JP Maxwell to wiki.o.o
Marton
On Sat, Feb 27, 2016 at 6:41 AM JP
Maxwell <j...@tipit.net
<mailto:j...@tipit.net>> wrote:
Marton has SSH access and applied a
patch earlier today. It appears the
spam continues to flow:
https://wiki.openstack.org/wiki/40_Thoughts_Of_Using_Open_Shelves_On_A_Kitchen
Marton let me know if you can look
at it some more or Infra if you want
to give me SSH I'll do so as well in
the morning (public key attached).
ssh-rsa
AAAAB3NzaC1yc2EAAAABIwAAAQEA2b5I7Yff9FCrtRmSjpILUePi54Vbc8zqJTbzrIAQZGFLBi3xd2MLlhV5QVgpDBC9H3lGjbdnc81D3aFd3HwHT4dvvvyedT12PR3VDEpftdW84vw3jzdtALcayOQznjbGnScwvX5SgnRhNxuX9Rkh8qNvOsjYPUafRr9azkQoomJFkdNVI4Vb5DbLhTpt18FPeOf0UuqDt/J2tHI4SjZ3kjzr7Nbwpg8xGgANPNE0+2pJbwCA8YDt4g3bzfzvVafQs5o9Gfc9tudkR9ugQG1M+EWCgu42CleOwMTd/rYEB2fgNNPsZAWqwQfdPajVuk70EBKUEQSyoA09eEZX+xJN9Q==
jpmax...@tipit.net
<mailto:jpmax...@tipit.net>
J.P. Maxwell / tipit.net
<http://www.tipit.net>
On Fri, Feb 26, 2016 at 12:09 PM,
Jimmy McArthur <ji...@openstack.org
<mailto:ji...@openstack.org>> wrote:
Super thankful for all the folks
that have jumped in over the
last couple of days to help with
the puppetization, etc... I just
feel like we're taking a very
wrong approach here.
Paul Belanger wrote:
Right, and I don't have an issue
with that approach. Based on the work we did
yesterday, anybody can do that via
our workflow. Please submit a patch to
puppet-mediawiki[1] and ping an
infra-root in #openstack-infra IRC.
What I'm proposing is the
workflow is really meant for
software, not for web
applications. It's tedious and
time consuming when what's
needed here is a set of tests on
the server. Submitting a patch,
waiting for a +1, then getting
on IRC to find someone with
access (and time) to paste the
logs is a pretty time consuming
process for what should be a
series of rapid-fire
changes/fixes on the server.
Especially when we're dealign
with an active attack.
We can then have somebody look at
the logs. I think it is more about scheduling
the task since more infra-root as
travling back from the mid-cycle last night
and today.
Right, this is my point. This
has been going on for 3 weeks
(or more). Tom Fifeldt was
asking for help without
response. And here we are
through another week and no
closer to stemming the flow.
I'm fully aware what I'm
proposing goes against what
Infra and the OpenStack workflow
is all about, but I'd ask you
all to look at this from a web
development perspective instead
of a software development
perspective.
Jimmy
Last email from me, just on a
plane. Will follow up when I land.
[1]https://git.openstack.org/cgit/openstack-infra/puppet-mediawiki
J.P. Maxwell |tipit.net
<http://tipit.net> [http://tipit.net] |fibercove.com
<http://fibercove.com>
[http://www.fibercove.com]
On Fri, Feb 26, 2016 at 11:25 AM,
Paul Belanger<pabelan...@redhat.com>
<mailto:pabelan...@redhat.com>
wrote:
On Fri, Feb 26, 2016 at
11:08:18AM -0600, Jimmy McArthur wrote:
Given the state of the wiki
a the moment, I think taking the quickest path
to get it fixed would be
prudent. Is there a way we can get JP root access
to this server, even
temporarily? We get 25% of our website traffic (2
million visitors) to the
wiki. I realize we're all after the same thing,
but
spammers are not going to
hit the dev environment, so there's really no
way
to tell if teh problem is
fixed without actually working directly on the
production machine. This
should be a 30 minute fix.
I am still unclear what the
30min fix is. If really 30mins, then it
shouldn't be
hard to get the fix into our
workflow. Could somebody please elaborate.
If we are talking about
deploying new versions of php or mediawiki manually,
I
not be in-favor of this. To me,
while the attack sucks, we should be working
on
2 fronts. Getting the help
needed to mitigate the attack, then adding the
changes into -infra workflow in
parallel.
I realize there is a lot of
risk in giving ssh access to infra machines,
but
I think it's worth taking a
look at either putting this machine in a place
where a different level of
admin could access it without giving away the
keys to the entire
OpenStack infrastructure or figuring out a way to set
up
credentials with varying
levels of access.
As a note, all the work I've
been doing to help with the attack hasn't
require
SSH access for me to wiki.o.o.
I did need infra-root help to expose our
configuration safely. I'd
rather take some time to see what the fixes are,
having infra-root apply
changes, then move them into puppet.
It also has been discussed to
simply disable write access to the wiki if we
really want spamming to stop,
obviously that will affect normal usage.
Jimmy
Paul Belanger wrote:
On Fri, Feb 26, 2016 at
10:12:12AM -0600, JP Maxwell wrote:
But if you wanted
to upgrade everything, remove the mobile view
extension,
test in a
dev/staging environment then deploy to production fingers
crossed, I think
that would be a valid approach as well.
Current review up[1].
I'll launch a node tonight / tomorrow locally to
see
how
puppet reacts. I
suspect there will be some issues.
If infra-roots are fine
with this approach, we can use that box to test
against.
[1]https://review.openstack.org/#/c/285405/
J.P. Maxwell
|tipit.net
<http://tipit.net>
|fibercove.com <http://fibercove.com>
On Feb 26, 2016 10:08 AM, "JP
Maxwell"<j...@tipit.net>
<mailto:j...@tipit.net> wrote:
Plus one except
in this case it is much easier to know if our efforts
are
working on
production because the spam either stops or not.
J.P. Maxwell
|tipit.net <http://tipit.net> |fibercove.com
<http://fibercove.com>
On Feb 26, 2016 9:48 AM,
"Paul Belanger"<pabelan...@redhat.com>
<mailto:pabelan...@redhat.com> wrote:
On Fri, Feb
26, 2016 at 09:18:00AM -0600, JP Maxwell wrote:
I
really think you might consider the option that there is a
vulnerability
in one
of the extensions. If that is the case black listing IPs will
be
an
ongoing
wild goose chase.
I think
this would be easily proven or disproven by making the questy
question impossible and see if the spam continues.
We'll have
to let an infra-root make that call. Since nobody would be
able to
use the
wiki. Honestly, I'd rather spend the time standing up a mirror
dev
instance
for us to work on, rather then production.
J.P.
Maxwell |tipit.net
<http://tipit.net> |fibercove.com
<http://fibercove.com>
On Feb 26, 2016 9:12
AM, "Paul Belanger"<pabelan...@redhat.com>
<mailto:pabelan...@redhat.com>
wrote:
On
Thu, Feb 25, 2016 at 08:10:34PM -0800, Elizabeth K. Joseph wrote:
On
Thu, Feb 25, 2016 at 6:35 AM, Jeremy Stanley<fu...@yuggoth.org>
<mailto:fu...@yuggoth.org>
wrote:
On 2016-02-25 02:46:13 -0600 (-0600), JP Maxwell wrote:
Please be aware that you can now create accounts under the mobile
view in the wiki native user table. I just created an account for
JpMaxMan. Not sure if this matters but wanted to make sure you
were aware.
Oh, yes I think having a random garbage question/answer was in
fact
previously preventing account creation under the mobile view. We
probably need a way to disable mobile view account creation as it
bypasses OpenID authentication entirely.
So that's what it was doing! We'll have to tackle the mobile view
issue.
Otherwise, quick update here:
The captcha didn't appear to help stem the spam tide. We'll want to
explore and start implementing some of the other solutions.
I did some database poking around today and it does seem like all
the
users do have launchpad accounts and email addresses.
So,
I have a few hours before jumping on my plane and checked into
this.
We
are
using QuestyCaptcha which according to docs, should almost be
impossible
for
spammers to by pass in an automated fashion. So, either our captcha
is too
easy, or we didn't set it up properly. I don't have SSH on wiki.o.o
so
others
will have to check logs. I did test new pages and edits, and was
promoted
by
captcha.
As
a next step, we might need to add additional apache2
configuration
to
blacklist IPs. I am reading up on that now.
--
Elizabeth Krumbach Joseph || Lyz || pleia2
_______________________________________________
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
<mailto:OpenStack-Infra@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
_______________________________________________
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
<mailto:OpenStack-Infra@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
_______________________________________________
OpenStack-Infra mailing
list
OpenStack-Infra@lists.openstack.org
<mailto:OpenStack-Infra@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
_______________________________________________
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
<mailto:OpenStack-Infra@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
_______________________________________________
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
<mailto:OpenStack-Infra@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
_______________________________________________
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
<mailto:OpenStack-Infra@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
_______________________________________________
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
<mailto:OpenStack-Infra@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
_______________________________________________
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
