Thanks, creating the venv in a container did the trick. May I suggest
adding virtualenv to the Python 3.7 image? python -m venv works, but
virtualenv is more commonly used.
On Mon, Jan 13, 2020 at 2:09 AM Brooke Storm <bst...@wikimedia.org> wrote:
> You have to create the venv in a container using 'webservice shell of the
> right runtime'. We support Python versions from Debian Jessie, Stretch and
> Buster by building in containers, so we cannot sync more than one of those
> to the bastion. We have moved a lot of Python tools back and forth without
> issue, but we have rebuilt the container image, which can introduce
> issues. Tomorrow, I will try a few things to be sure, but a venv can easily
> cause problems.
>
> On Sun, Jan 12, 2020, 16:30 Alex Monk <kren...@gmail.com> wrote:
>
>> Interesting, uwsgi had Python 3.7.3 but `./www/python/venv/bin/python
>> --version` says 3.7.6. Is that a big enough difference to cause problems?
>>
>> On Sun, 12 Jan 2020 at 23:19, Chico Venancio <chicocvenan...@gmail.com>
>> wrote:
>>
>>> Maybe a venv created in a different python version?
>>>
>>> Chico Venancio
>>>
>>> Em dom, 12 de jan de 2020 20:14, Alex Monk <kren...@gmail.com> escreveu:
>>>
>>>> I think I've seen that particular error that you see in stdout/stderr
>>>> (via kubectl logs) before - on pods that in fact were working.
>>>>
>>>> Meanwhile, uwsgi.log says:
>>>>
>>>> Python version: 3.7.3 (default, Apr 3 2019, 05:39:12) [GCC 8.3.0]
>>>> Set PythonHome to /data/project/countcounttest/www/python/venv
>>>> Fatal Python error: initfsencoding: Unable to get the locale encoding
>>>> ModuleNotFoundError: No module named 'encodings'
>>>>
>>>> Current thread 0x00007fe50490e780 (most recent call first):
>>>> !!! uWSGI process 1 got Segmentation Fault !!!
>>>>
>>>> followed by a backtrace. Suggests the problem is related to something
>>>> inside the image/application code rather than the cluster itself anyway.
>>>> I notice the pod on the new cluster seems to be using the sssd variant
>>>> of the toolforge-python37-web image, which pods in the old cluster are not
>>>> using. I doubt it's the source problem as uwsgi shouldn't be segfaulting
>>>> over some problem talking to LDAP...
>>>> Needs further investigation by someone during the week I think.
>>>>
>>>> On Sun, 12 Jan 2020 at 23:00, Count Count <
>>>> countvoncount123...@gmail.com> wrote:
>>>>
>>>>> Your pod started and container and it crashed, I see a uwsgi.log file
>>>>>> with a python module problem and a uwsgi segfault.
>>>>>>
>>>>>
>>>>> Yes. It was working fine with the legacy cluster.
>>>>> The service ist started via webservice --backend=kubernetes
>>>>> python3.7 start
>>>>>
>>>>> Apparently it cannot load the uwsgi shared library if deployed on the
>>>>> new cluster?
>>>>> tools.countcounttest@tools-sgebastion-07:~$ kubectl logs
>>>>> countcounttest-6b58f5c547-785mr
>>>>> open("/usr/lib/uwsgi/plugins/python_plugin.so"): No such file or
>>>>> directory [core/utils.c line 3724]
>>>>> !!! UNABLE to load uWSGI plugin:
>>>>> /usr/lib/uwsgi/plugins/python_plugin.so: cannot open shared object file:
>>>>> No
>>>>> such file or directory !!!
>>>>>
>>>>> On Sun, Jan 12, 2020 at 11:42 PM Alex Monk <kren...@gmail.com> wrote:
>>>>>
>>>>>> Hi Count Count, I believe I may have sorted out an issue that
>>>>>> prevented some pods (depending partially on luck) from creating
>>>>>> containers.
>>>>>> Your pod started and container and it crashed, I see a uwsgi.log file
>>>>>> with
>>>>>> a python module problem and a uwsgi segfault.
>>>>>>
>>>>>> On Sun, 12 Jan 2020 at 22:12, Alex Monk <kren...@gmail.com> wrote:
>>>>>>
>>>>>>> Thanks Count Count. I have identified a new issue with the new k8s
>>>>>>> cluster and am looking into it.
>>>>>>>
>>>>>>> On Sun, 12 Jan 2020 at 21:43, Count Count <
>>>>>>> countvoncount123...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Yes, I switched back to the old cluster. This is a new tool that
>>>>>>>> was used in production even if only rarely. I can't leave it offline
>>>>>>>> for
>>>>>>>> hours.
>>>>>>>>
>>>>>>>> I have created a test tool as a copy with which I can reproduce the
>>>>>>>> issue:
>>>>>>>> tools.countcounttest@tools-sgebastion-07:~$ kubectl get pods
>>>>>>>> NAME READY STATUS
>>>>>>>> RESTARTS AGE
>>>>>>>> countcounttest-6b58f5c547-mf4jx 0/1 ContainerCreating 0
>>>>>>>> 77s
>>>>>>>>
>>>>>>>> I will leave that running. If the container gets created I might
>>>>>>>> also be able to reproduce the segfault.
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>>
>>>>>>>> Count Count
>>>>>>>>
>>>>>>>> On Sun, Jan 12, 2020 at 10:20 PM Alex Monk <kren...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Count Count,
>>>>>>>>>
>>>>>>>>> I'm afraid you seem to have no pods on the new cluster to look at:
>>>>>>>>>
>>>>>>>>> # kubectl get -n tool-flaggedrevspromotioncheck pod
>>>>>>>>> No resources found.
>>>>>>>>>
>>>>>>>>> Alex
>>>>>>>>>
>>>>>>>>> On Sun, 12 Jan 2020 at 21:07, Count Count <
>>>>>>>>> countvoncount123...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi!
>>>>>>>>>>
>>>>>>>>>> I don't have much luck with a webservice based on the python3.7
>>>>>>>>>> image. It is running fine on the legacy K8s cluster.
>>>>>>>>>>
>>>>>>>>>> On the new cluster I got a segfault. After stopping the
>>>>>>>>>> webservice and trying again to get an empty log the pod is now stuck
>>>>>>>>>> in
>>>>>>>>>> ContainerCreating.
>>>>>>>>>>
>>>>>>>>>> A few minutes ago:
>>>>>>>>>> tools.flaggedrevspromotioncheck@tools-sgebastion-08:~$ kubectl
>>>>>>>>>> get pods
>>>>>>>>>> NAME READY STATUS
>>>>>>>>>> RESTARTS AGE
>>>>>>>>>> flaggedrevspromotioncheck-7cbfff44fc-jnhmq 0/1
>>>>>>>>>> ContainerCreating 0 2m48s
>>>>>>>>>>
>>>>>>>>>> ...and just now:
>>>>>>>>>> tools.flaggedrevspromotioncheck@tools-sgebastion-08:~$ kubectl
>>>>>>>>>> get pods
>>>>>>>>>> NAME READY STATUS
>>>>>>>>>> RESTARTS AGE
>>>>>>>>>> flaggedrevspromotioncheck-7cbfff44fc-q55gm 0/1
>>>>>>>>>> ContainerCreating 0 5m18s
>>>>>>>>>>
>>>>>>>>>> Best regards,
>>>>>>>>>>
>>>>>>>>>> Count Count
>>>>>>>>>>
>>>>>>>>>> On Thu, Jan 9, 2020 at 10:58 PM Bryan Davis <bd...@wikimedia.org>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> I am happy to announce that a new and improved Kubernetes
>>>>>>>>>>> cluster is
>>>>>>>>>>> now available for use by beta testers on an opt-in basis. A page
>>>>>>>>>>> has
>>>>>>>>>>> been created on Wikitech [0] outlining the self-service migration
>>>>>>>>>>> process.
>>>>>>>>>>>
>>>>>>>>>>> Timeline:
>>>>>>>>>>> * 2020-01-09: 2020 Kubernetes cluster available for beta testers
>>>>>>>>>>> on an
>>>>>>>>>>> opt-in basis
>>>>>>>>>>> * 2020-01-23: 2020 Kubernetes cluster general availability for
>>>>>>>>>>> migration on an opt-in basis
>>>>>>>>>>> * 2020-02-10: Automatic migration of remaining workloads from
>>>>>>>>>>> 2016
>>>>>>>>>>> cluster to 2020 cluster by Toolforge admins
>>>>>>>>>>>
>>>>>>>>>>> This new cluster has been a work in progress for more than a year
>>>>>>>>>>> within the Wikimedia Cloud Services team, and a top priority
>>>>>>>>>>> project
>>>>>>>>>>> for the past six months. About 35 tools, including
>>>>>>>>>>> https://tools.wmflabs.org/admin/, are currently running on what
>>>>>>>>>>> we are
>>>>>>>>>>> calling the "2020 Kubernetes cluster". This new cluster is
>>>>>>>>>>> running
>>>>>>>>>>> Kubernetes v1.15.6 and Docker 19.03.4. It is also using a newer
>>>>>>>>>>> authentication and authorization method (RBAC), a new ingress
>>>>>>>>>>> routing
>>>>>>>>>>> service, and a different method of integrating with the Developer
>>>>>>>>>>> account LDAP service. We have built a new tool [1] which makes
>>>>>>>>>>> the
>>>>>>>>>>> state of the Kubernetes cluster more transparent and on par with
>>>>>>>>>>> the
>>>>>>>>>>> information that we already expose for the grid engine cluster
>>>>>>>>>>> [2].
>>>>>>>>>>>
>>>>>>>>>>> With a significant number of tools managed by Toolforge
>>>>>>>>>>> administrators
>>>>>>>>>>> already migrated to the new cluster, we are fairly confident
>>>>>>>>>>> that the
>>>>>>>>>>> basic features used by most Kubernetes tools are covered. It is
>>>>>>>>>>> likely
>>>>>>>>>>> that a few outlying issues remain to be found as more tools
>>>>>>>>>>> move, but
>>>>>>>>>>> we have confidence that we can address them quickly. This has
>>>>>>>>>>> led us
>>>>>>>>>>> to propose a fairly short period of voluntary beta testing,
>>>>>>>>>>> followed
>>>>>>>>>>> by a short general availability opt-in migration period, and
>>>>>>>>>>> finally a
>>>>>>>>>>> complete migration of all remaining tools which will be done by
>>>>>>>>>>> the
>>>>>>>>>>> Toolforge administration team for anyone who has not migrated
>>>>>>>>>>> their
>>>>>>>>>>> self.
>>>>>>>>>>>
>>>>>>>>>>> Please help with beta testing if you have some time and are
>>>>>>>>>>> willing to
>>>>>>>>>>> get help on irc, Phabricator, and the cloud@lists.wikimedia.org
>>>>>>>>>>> mailing list for early adopter issues you may encounter.
>>>>>>>>>>>
>>>>>>>>>>> I want to publicly praise Brooke Storm and Arturo Borrero
>>>>>>>>>>> González for
>>>>>>>>>>> the hours that they have put into reading docs, building proof of
>>>>>>>>>>> concept clusters, and improving automation and processes to make
>>>>>>>>>>> the
>>>>>>>>>>> 2020 Kubernetes cluster possible. The Toolforge community can
>>>>>>>>>>> look
>>>>>>>>>>> forward to more frequent and less disruptive software upgrades
>>>>>>>>>>> in this
>>>>>>>>>>> cluster as a direct result of this work. We have some other
>>>>>>>>>>> feature
>>>>>>>>>>> improvements in planning now that I think you will all be
>>>>>>>>>>> excited to
>>>>>>>>>>> see and use later this year!
>>>>>>>>>>>
>>>>>>>>>>> [0]:
>>>>>>>>>>> https://wikitech.wikimedia.org/wiki/News/2020_Kubernetes_cluster_migration
>>>>>>>>>>> [1]: https://tools.wmflabs.org/k8s-status/
>>>>>>>>>>> [2]: https://tools.wmflabs.org/sge-status/
>>>>>>>>>>>
>>>>>>>>>>> Bryan (on behalf of the Toolforge admins and the Cloud Services
>>>>>>>>>>> team)
>>>>>>>>>>> --
>>>>>>>>>>> Bryan Davis Technical Engagement Wikimedia
>>>>>>>>>>> Foundation
>>>>>>>>>>> Principal Software Engineer Boise,
>>>>>>>>>>> ID USA
>>>>>>>>>>> [[m:User:BDavis_(WMF)]]
>>>>>>>>>>> irc: bd808
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Wikimedia Cloud Services announce mailing list
>>>>>>>>>>> cloud-annou...@lists.wikimedia.org (formerly
>>>>>>>>>>> labs-annou...@lists.wikimedia.org)
>>>>>>>>>>> https://lists.wikimedia.org/mailman/listinfo/cloud-announce
>>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Wikimedia Cloud Services mailing list
>>>>>>>>>> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>>>>>>>>>> https://lists.wikimedia.org/mailman/listinfo/cloud
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Wikimedia Cloud Services mailing list
>>>>>>>>> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>>>>>>>>> https://lists.wikimedia.org/mailman/listinfo/cloud
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Wikimedia Cloud Services mailing list
>>>>>>>> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>>>>>>>> https://lists.wikimedia.org/mailman/listinfo/cloud
>>>>>>>
>>>>>>> _______________________________________________
>>>>>> Wikimedia Cloud Services mailing list
>>>>>> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>>>>>> https://lists.wikimedia.org/mailman/listinfo/cloud
>>>>>
>>>>> _______________________________________________
>>>>> Wikimedia Cloud Services mailing list
>>>>> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>>>>> https://lists.wikimedia.org/mailman/listinfo/cloud
>>>>
>>>> _______________________________________________
>>>> Wikimedia Cloud Services mailing list
>>>> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>>>> https://lists.wikimedia.org/mailman/listinfo/cloud
>>>
>>> _______________________________________________
>>> Wikimedia Cloud Services mailing list
>>> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>>> https://lists.wikimedia.org/mailman/listinfo/cloud
>>
>> _______________________________________________
>> Wikimedia Cloud Services mailing list
>> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>> https://lists.wikimedia.org/mailman/listinfo/cloud
>
> _______________________________________________
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud
_______________________________________________
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud
