Quick update: -* I had another report of this happening during the night*. I have disabled everything past the main page until I can get to the bottom of this - Waiting for more information from the user. i.e. to get their platform and the link they clicked - I know that the user they were erroneously logged in as had been active on the site around the time. - I rolled back to stable before this happened, so it isn't a problem with trunk - Using webfaction/nginx/uwsgi (ver 1.2.4)
Neil On Tuesday, July 24, 2012 11:53:26 PM UTC+1, Craig Younkins wrote: > > What is the deployment configuration? What wsgi server and what web server? > > Craig Younkins > > > On Tue, Jul 24, 2012 at 5:55 PM, Massimo Di Pierro wrote: > >> Is there an open issue about this? If not, can you open one with more >> details? >> >> >> On Tuesday, 24 July 2012 16:31:52 UTC-5, howesc wrote: >> >>> one other scenario...... >>> >>> i reported a few months back that running web2py on GAE with python2.7 >>> and multi-threading had odd behaviors with the globals (request, response, >>> session). i have yet tracked down the issues i was having (might have been >>> a coding error on my part).....but if you are using GAE + multithreading >>> i'd be interested to know that. >>> >>> cfh >>> >>> On Tuesday, July 24, 2012 1:26:21 PM UTC-7, Massimo Di Pierro wrote: >>>> >>>> Perhaps it would be safe to block access to the site if request.client >>>> is "unknown". >>>> I think we should change web2py to block access to any web2py app if >>>> request.client does not validate as an IP address. >>>> >>>> Massimo >>>> >>>> On Tuesday, 24 July 2012 15:24:06 UTC-5, Massimo Di Pierro wrote: >>>>> >>>>> Here is a possible cause of the problem although I am not sure. >>>>> There are two possible issues which may conspire to create this >>>>> problem. >>>>> >>>>> Issue #1 >>>>> ======= >>>>> >>>>> There is a session file in the app you sent me called: >>>>> >>>>> unknown-c4571a37... >>>>> >>>>> session files should be >>>>> >>>>> <ip>-..... >>>>> >>>>> This means that one of the HEADERS http_x_forwarded_for or remote_addr >>>>> has a value "unknown". >>>>> >>>>> A first google search retuned: >>>>> http://nixforums.org/**about154671-Hacking-X-**Forwarded-For.html<http://nixforums.org/about154671-Hacking-X-Forwarded-For.html> >>>>> which opens the possibility the the web server, in your case nginx, is >>>>> not finding the client ip address (how is that possible) and setting it >>>>> to >>>>> unknown. This should never happen. The client_addr is a required field >>>>> for >>>>> WSGI. >>>>> >>>>> This could be the result of a hacking attempt but it would required >>>>> both parties doing the hacking for the sessions to be mixed up. >>>>> >>>>> Issue #2 >>>>> ======= >>>>> >>>>> There is a bug with may prevent urandom from working: >>>>> >>>>> http://community.webfaction.**com/questions/9333/** >>>>> importerror-cannot-import-**name-urandom<http://community.webfaction.com/questions/9333/importerror-cannot-import-name-urandom> >>>>> http://stackoverflow.com/**questions/10776797/error-when-** >>>>> importing-wsgihandler-with-**django<http://stackoverflow.com/questions/10776797/error-when-importing-wsgihandler-with-django> >>>>> >>>>> Can you check if you can import urandom on your version of python on >>>>> webfaction? >>>>> >>>>> >>>>> It is therefore theoretically possible that, given the concurrency >>>>> model of nginx, if two users visit the site very close to each other, >>>>> with >>>>> urandom missing, both declaring the same incorrect client ip (unknown), >>>>> they get assigned the same session id. This is because web2py has no way >>>>> of >>>>> distinguishing the two users and lacks a proper random number generator. >>>>> >>>>> TODO: >>>>> >>>>> 1) check if you can import urandom >>>>> 2) try understand how it possible to have an "unkown" client_addr in >>>>> the http headers. >>>>> >>>>> My google search returned nothing about 2. Has anybody ever seen this >>>>> before? >>>>> Please let us know. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Tuesday, 24 July 2012 14:50:04 UTC-5, Massimo Di Pierro wrote: >>>>>> >>>>>> Nothing stands out from your code. It is very good code. You have >>>>>> changed to gluon/tools.py but I do not think they can be causing this >>>>>> problem. >>>>>> >>>>>> On Tuesday, 24 July 2012 14:48:16 UTC-5, Massimo Di Pierro wrote: >>>>>>> >>>>>>> I should add that the conflict I mentioned below is not possible >>>>>>> unless there is a proxy in between. That is because the session id >>>>>>> includes >>>>>>> the client IP. >>>>>>> >>>>>>> I really do not see how this problem can be possible. Are you sure >>>>>>> they are not playing a prank on you? If they share a facebook page >>>>>>> perhaps >>>>>>> they know each other. I have to ask but we will keep investigating the >>>>>>> issue very seriously nevertheless. >>>>>>> >>>>>>> For now I suggest you add this to your code: >>>>>>> >>>>>>> if auth.user: >>>>>>> session.clients = session.clients or [] >>>>>>> if not request.client in session.clients: session.clients.append( >>>>>>> **request.client) >>>>>>> if len(session.clients)>1: print auth.user.email, session.clients >>>>>>> >>>>>>> log the output and check how often you have multiple session.clients >>>>>>> for the same email from different network top level domains (xxx.*.*.*) >>>>>>> If >>>>>>> you do, email the user and check what is going on with them. >>>>>>> >>>>>>> Massimo >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tuesday, 24 July 2012 14:26:35 UTC-5, Massimo Di Pierro wrote: >>>>>>>> >>>>>>>> The only time I have seen something like this was long age. Web2py >>>>>>>> was running on replicated VMs behing a load balancer. If two requests >>>>>>>> from >>>>>>>> new users arrived within a short time frame (do not remember if a >>>>>>>> millisecond or a second), they were assigned the same session uuid >>>>>>>> because >>>>>>>> uuid.uuid4() could not discriminate between the VMs. We fixed it by >>>>>>>> make >>>>>>>> uuid dependent on the os entropy source urandom and initializing it >>>>>>>> differently on different VMs using the IP address. The fix works on >>>>>>>> linux/unix but not on Windows. Replicated windows machine may suffer >>>>>>>> from >>>>>>>> this problem still. >>>>>>>> >>>>>>>> What is the web server and configuration in your case? >>>>>>>> Do you know what was the link that caused the problem? >>>>>>>> Which page she was directed too? >>>>>>>> >>>>>>>> massimo >>>>>>>> >>>>>>>> On Tuesday, 24 July 2012 10:18:46 UTC-5, Jonathan Lundell wrote: >>>>>>>>> >>>>>>>>> On 24 Jul 2012, at 6:41 AM, Neil wrote: >>>>>>>>> >>>>>>>>> Good point about trunk. There are some features that I liked and >>>>>>>>> got used to, but nothing essential. >>>>>>>>> >>>>>>>>> I'll try to summarize any relevant settings in the hope that >>>>>>>>> someone can spot something. >>>>>>>>> >>>>>>>>> In 0.py I have: >>>>>>>>> >>>>>>>>> ... >>>>>>>>> settings.login_method = 'local' >>>>>>>>> settings.login_config = '' >>>>>>>>> ... >>>>>>>>> >>>>>>>>> in db.py: >>>>>>>>> >>>>>>>>> ... >>>>>>>>> auth = Auth(db, hmac_key=Auth.get_or_create_**key()) >>>>>>>>> crud, service, plugins = Crud(db), Service(), PluginManager() >>>>>>>>> auth.define_tables() >>>>>>>>> db.auth_user.last_name.**requires = None >>>>>>>>> auth.settings.actions_**disabled.append('register') >>>>>>>>> auth.settings.registration_**requires_verification = False >>>>>>>>> auth.settings.registration_**requires_approval = True >>>>>>>>> auth.settings.reset_password_**requires_verification = False >>>>>>>>> auth.settings.login_next = URL("social_anxiety", "user_main") >>>>>>>>> auth.settings.logout_next = URL("default", "index") >>>>>>>>> ... >>>>>>>>> >>>>>>>>> and in default.py: >>>>>>>>> >>>>>>>>> >>>>>>>>> def index(): >>>>>>>>> session.forget(response) >>>>>>>>> if auth.is_logged_in(): >>>>>>>>> redirect(URL(c='social_**anxiety', f='user_main')) >>>>>>>>> else: >>>>>>>>> return dict() >>>>>>>>> >>>>>>>>> def user(): >>>>>>>>> if request.args(0) == 'register': >>>>>>>>> db.auth_user.first_name.commen**t = '(or an anonymous >>>>>>>>> user name)' >>>>>>>>> elif request.args(0) == 'profile': >>>>>>>>> redirect(URL(c='default', f='user_profile')) >>>>>>>>> >>>>>>>>> return dict(form = auth()) >>>>>>>>> >>>>>>>>> and in layout.html to create the navbar: >>>>>>>>> >>>>>>>>> {{try:}} >>>>>>>>> {{=auth.navbar(referrer_**actions=None)}} >>>>>>>>> {{except:pass}} >>>>>>>>> >>>>>>>>> Anything stand out? In particular, anything that would apply one >>>>>>>>> user's session to another user on a different computer? >>>>>>>>> >>>>>>>>> Now that I look at it, "session.forget" >>>>>>>>> in application/default/index seems like a bad idea. I put it in to >>>>>>>>> see if I >>>>>>>>> could speed up the main page and kind of forgot about it... Just >>>>>>>>> removed it. >>>>>>>>> >>>>>>>>> >>>>>>>>> That jumped out at me too, but it's not obvious how it could >>>>>>>>> result in the reported symptom. >>>>>>>>> >>>>>>>>> Does the forget() call affect the is_logged_in() call one way or >>>>>>>>> the other? Even if it did, in order to appear logged in as user X, a >>>>>>>>> browser would have to present a cookie with session id of a user X >>>>>>>>> session. >>>>>>>>> How could that happen? Weird. >>>>>>>>> >>>>>>>>> >>>>>>>>> Neil >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tuesday, July 24, 2012 2:11:25 PM UTC+1, Richard wrote: >>>>>>>>>> >>>>>>>>>> For sure using trunk is not very safe in production >>>>>>>>>> environnement, not because it not secure, but because sometimes >>>>>>>>>> things >>>>>>>>>> brake when new features are added. If you don't need edge feature, >>>>>>>>>> better >>>>>>>>>> to stick with stable. >>>>>>>>>> >>>>>>>>>> For the problem you describe, I think if you show us the way you >>>>>>>>>> activate auth could help. I mean it is not just a matter of using >>>>>>>>>> decorator... >>>>>>>>>> >>>>>>>>>> I am not the best one to help you fix this issue, but if you give >>>>>>>>>> us more information like what's in you db.py and all the auth >>>>>>>>>> setting you >>>>>>>>>> set, I am sure there is more knowledge users that will be kind and >>>>>>>>>> will >>>>>>>>>> help. >>>>>>>>>> >>>>>>>>>> Richard >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Jul 24, 2012 at 8:18 AM, Neil: >>>>>>>>>> >>>>>>>>>>> I just heard from someone who had never been to my site before. >>>>>>>>>>> When she visited (on her phone), it was already logged on as >>>>>>>>>>> another user. >>>>>>>>>>> This other user (she told me his name) is located on the other side >>>>>>>>>>> of the >>>>>>>>>>> world, and may or may not have logged out. I'm rather worried - she >>>>>>>>>>> was >>>>>>>>>>> accessing functions decorated with @auth.requires_login() without >>>>>>>>>>> even >>>>>>>>>>> having an account, let alone logging in! Once she clicked "logout" >>>>>>>>>>> she was >>>>>>>>>>> no longer able to access any user pages. >>>>>>>>>>> >>>>>>>>>>> I understand this will be tough to debug with so little >>>>>>>>>>> information. Furthermore, I've never observed this behaviour >>>>>>>>>>> personally. >>>>>>>>>>> However, it's concerning enough that I thought I'd see if anyone >>>>>>>>>>> else >>>>>>>>>>> has experienced such a thing. If not, any ideas how such a thing >>>>>>>>>>> could even >>>>>>>>>>> happen? >>>>>>>>>>> >>>>>>>>>>> I'm using trunk - I suppose I should roll back to stable? >>>>>>>>>>> >>>>>>>>>>> Neil >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> On Tuesday, 24 July 2012 14:50:04 UTC-5, Massimo Di Pierro wrote: >>>>>> >>>>>> Nothing stands out from your code. It is very good code. You have >>>>>> changed to gluon/tools.py but I do not think they can be causing this >>>>>> problem. >>>>>> >>>>>> On Tuesday, 24 July 2012 14:48:16 UTC-5, Massimo Di Pierro wrote: >>>>>>> >>>>>>> I should add that the conflict I mentioned below is not possible >>>>>>> unless there is a proxy in between. That is because the session id >>>>>>> includes >>>>>>> the client IP. >>>>>>> >>>>>>> I really do not see how this problem can be possible. Are you sure >>>>>>> they are not playing a prank on you? If they share a facebook page >>>>>>> perhaps >>>>>>> they know each other. I have to ask but we will keep investigating the >>>>>>> issue very seriously nevertheless. >>>>>>> >>>>>>> For now I suggest you add this to your code: >>>>>>> >>>>>>> if auth.user: >>>>>>> session.clients = session.clients or [] >>>>>>> if not request.client in session.clients: session.clients.append( >>>>>>> **request.client) >>>>>>> if len(session.clients)>1: print auth.user.email, session.clients >>>>>>> >>>>>>> log the output and check how often you have multiple session.clients >>>>>>> for the same email from different network top level domains (xxx.*.*.*) >>>>>>> If >>>>>>> you do, email the user and check what is going on with them. >>>>>>> >>>>>>> Massimo >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tuesday, 24 July 2012 14:26:35 UTC-5, Massimo Di Pierro wrote: >>>>>>>> >>>>>>>> The only time I have seen something like this was long age. Web2py >>>>>>>> was running on replicated VMs behing a load balancer. If two requests >>>>>>>> from >>>>>>>> new users arrived within a short time frame (do not remember if a >>>>>>>> millisecond or a second), they were assigned the same session uuid >>>>>>>> because >>>>>>>> uuid.uuid4() could not discriminate between the VMs. We fixed it by >>>>>>>> make >>>>>>>> uuid dependent on the os entropy source urandom and initializing it >>>>>>>> differently on different VMs using the IP address. The fix works on >>>>>>>> linux/unix but not on Windows. Replicated windows machine may suffer >>>>>>>> from >>>>>>>> this problem still. >>>>>>>> >>>>>>>> What is the web server and configuration in your case? >>>>>>>> Do you know what was the link that caused the problem? >>>>>>>> Which page she was directed too? >>>>>>>> >>>>>>>> massimo >>>>>>>> >>>>>>>> On Tuesday, 24 July 2012 10:18:46 UTC-5, Jonathan Lundell wrote: >>>>>>>>> >>>>>>>>> On 24 Jul 2012, at 6:41 AM, Neil wrote: >>>>>>>>> >>>>>>>>> Good point about trunk. There are some features that I liked and >>>>>>>>> got used to, but nothing essential. >>>>>>>>> >>>>>>>>> I'll try to summarize any relevant settings in the hope that >>>>>>>>> someone can spot something. >>>>>>>>> >>>>>>>>> In 0.py I have: >>>>>>>>> >>>>>>>>> ... >>>>>>>>> settings.login_method = 'local' >>>>>>>>> settings.login_config = '' >>>>>>>>> ... >>>>>>>>> >>>>>>>>> in db.py: >>>>>>>>> >>>>>>>>> ... >>>>>>>>> auth = Auth(db, hmac_key=Auth.get_or_create_**key()) >>>>>>>>> crud, service, plugins = Crud(db), Service(), PluginManager() >>>>>>>>> auth.define_tables() >>>>>>>>> db.auth_user.last_name.**requires = None >>>>>>>>> auth.settings.actions_**disabled.append('register') >>>>>>>>> auth.settings.registration_**requires_verification = False >>>>>>>>> auth.settings.registration_**requires_approval = True >>>>>>>>> auth.settings.reset_password_**requires_verification = False >>>>>>>>> auth.settings.login_next = URL("social_anxiety", "user_main") >>>>>>>>> auth.settings.logout_next = URL("default", "index") >>>>>>>>> ... >>>>>>>>> >>>>>>>>> and in default.py: >>>>>>>>> >>>>>>>>> >>>>>>>>> def index(): >>>>>>>>> session.forget(response) >>>>>>>>> if auth.is_logged_in(): >>>>>>>>> redirect(URL(c='social_**anxiety', f='user_main')) >>>>>>>>> else: >>>>>>>>> return dict() >>>>>>>>> >>>>>>>>> def user(): >>>>>>>>> if request.args(0) == 'register': >>>>>>>>> db.auth_user.first_name.commen**t = '(or an anonymous >>>>>>>>> user name)' >>>>>>>>> elif request.args(0) == 'profile': >>>>>>>>> redirect(URL(c='default', f='user_profile')) >>>>>>>>> >>>>>>>>> return dict(form = auth()) >>>>>>>>> >>>>>>>>> and in layout.html to create the navbar: >>>>>>>>> >>>>>>>>> {{try:}} >>>>>>>>> {{=auth.navbar(referrer_**actions=None)}} >>>>>>>>> {{except:pass}} >>>>>>>>> >>>>>>>>> Anything stand out? In particular, anything that would apply one >>>>>>>>> user's session to another user on a different computer? >>>>>>>>> >>>>>>>>> Now that I look at it, "session.forget" >>>>>>>>> in application/default/index seems like a bad idea. I put it in to >>>>>>>>> see if I >>>>>>>>> could speed up the main page and kind of forgot about it... Just >>>>>>>>> removed it. >>>>>>>>> >>>>>>>>> >>>>>>>>> That jumped out at me too, but it's not obvious how it could >>>>>>>>> result in the reported symptom. >>>>>>>>> >>>>>>>>> Does the forget() call affect the is_logged_in() call one way or >>>>>>>>> the other? Even if it did, in order to appear logged in as user X, a >>>>>>>>> browser would have to present a cookie with session id of a user X >>>>>>>>> session. >>>>>>>>> How could that happen? Weird. >>>>>>>>> >>>>>>>>> >>>>>>>>> Neil >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tuesday, July 24, 2012 2:11:25 PM UTC+1, Richard wrote: >>>>>>>>>> >>>>>>>>>> For sure using trunk is not very safe in production >>>>>>>>>> environnement, not because it not secure, but because sometimes >>>>>>>>>> things >>>>>>>>>> brake when new features are added. If you don't need edge feature, >>>>>>>>>> better >>>>>>>>>> to stick with stable. >>>>>>>>>> >>>>>>>>>> For the problem you describe, I think if you show us the way you >>>>>>>>>> activate auth could help. I mean it is not just a matter of using >>>>>>>>>> decorator... >>>>>>>>>> >>>>>>>>>> I am not the best one to help you fix this issue, but if you give >>>>>>>>>> us more information like what's in you db.py and all the auth >>>>>>>>>> setting you >>>>>>>>>> set, I am sure there is more knowledge users that will be kind and >>>>>>>>>> will >>>>>>>>>> help. >>>>>>>>>> >>>>>>>>>> Richard >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Jul 24, 2012 at 8:18 AM, Neil: >>>>>>>>>> >>>>>>>>>>> I just heard from someone who had never been to my site before. >>>>>>>>>>> When she visited (on her phone), it was already logged on as >>>>>>>>>>> another user. >>>>>>>>>>> This other user (she told me his name) is located on the other side >>>>>>>>>>> of the >>>>>>>>>>> world, and may or may not have logged out. I'm rather worried - she >>>>>>>>>>> was >>>>>>>>>>> accessing functions decorated with @auth.requires_login() without >>>>>>>>>>> even >>>>>>>>>>> having an account, let alone logging in! Once she clicked "logout" >>>>>>>>>>> she was >>>>>>>>>>> no longer able to access any user pages. >>>>>>>>>>> >>>>>>>>>>> I understand this will be tough to debug with so little >>>>>>>>>>> information. Furthermore, I've never observed this behaviour >>>>>>>>>>> personally. >>>>>>>>>>> However, it's concerning enough that I thought I'd see if anyone >>>>>>>>>>> else >>>>>>>>>>> has experienced such a thing. If not, any ideas how such a thing >>>>>>>>>>> could even >>>>>>>>>>> happen? >>>>>>>>>>> >>>>>>>>>>> I'm using trunk - I suppose I should roll back to stable? >>>>>>>>>>> >>>>>>>>>>> Neil >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >> >> >> >> > > --
