[Cloud] Quarry slower than before

I feel like Quarry is slower than before (before being last week or last month). Queries almost always get queued, and once executed, simple queries take longer to result. I have no idea how to investigate this though. Any thoughts? Huji ___ Wikimedia C

[Cloud] Help with a query

I wrote this query to find all page moves done in fawiki in 2017, and determine how many edits the performing user had prior to that page move. The query tries to use indexes, as much as I could think of, and yet it runs for a very long time (more than 20 min, at which point it gets killed). Any i

Re: [Cloud] Quarry slower than before

> On Sat, Dec 30, 2017 at 9:51 AM, Huji Lee wrote: > > I feel like Quarry is slower than before (before being last week or last > > month). Queries almost always get queued, and once executed, simple > queries > > take longer to result. > > > > I have no idea how

Re: [Cloud] Help with a query

All fantastic suggestions. The EXPLAIN results showed that changing to logging_userindex had potentially the largest impact (reducing the rows to be retrieved from 10mil to 5mil). I am rerunning the query with all suggested recommendations, and will report back. On Sat, Dec 30, 2017 at 1:32 PM, De

Re: [Cloud] Help with a query

There is a logging_logindex table. I incorporated all suggested changes into https://quarry.wmflabs.org/query/23829 Since I still think they query will take more than 20 minutes, I think quarry is not the way to go. I saved the query on the grid and tried to run it using this command: jsub -once

Re: [Cloud] Help with a query

The query took more than an hour, so I killed it. I think a better strategy is to first run the query on logging, and get a list of logs, then run the subquery for retrieving the edit counts. My new query is shown below; however, I cannot create temporary tables with my user on Toolforge. I tried

[Cloud] Changing my shell preference

Is there a way to change my preferred shell on the Clouds to zsh? When I try chsh -s `which zsh` it asks for a password, which I don't have of course. Thanks, Huji ___ Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly lab...@li

Re: [Cloud] Changing my shell preference

On Sun, Dec 31, 2017 at 6:41 AM, Martin Urbanec wrote: > Hello, > > The only one think that comes to my mind is to add /usr/bin/zsh to the .bashrc > file (and probably delete the rest of its content as it won't be needed). > But this is a hack > > ​It seems like .bashrc is managed by the puppet,

Re: [Cloud] Changing my shell preference

; LDAP field that can be used for the same. I have no way how I can examine >>> the LDAP thing because I'm not an admin :). >>> >>> Martin >>> >>> ne 31. 12. 2017 v 5:00 odesílatel Huji Lee napsal: >>> >>>> Is there a way to chang

Re: [Cloud] Help with a query

Sat, Dec 30, 2017 at 10:55 PM, Huji Lee wrote: > >> and log_id > 9406768; >> > > As I said earlier, it was bad advice to use log_id rather than > log_timestamp for this query. > > > -- > Brad Jorsch (Anomie) > Se

Re: [Cloud] Changing my shell preference

The method proposed above works with my own username (huji). But when I do "become huji" (my bot's name is also huji), the new "tools.huji" account still uses bash. How can I make that one use another shell? On Sun, Dec 31, 2017 at 3:14 PM, Bryan Davis wrote: > On Sun, Dec 31, 2017 at 10:02 AM,

[Cloud] Wikidata query request

Hello, I need help writing a query that I would like to run on the Clouds. The goal of the query is to retrieve the following information from wikidatawiki_p: * Find all pages that have a claim for the property P1566, for example see https://www.wikidata.org/wiki/Q2113430 * Find out what is the v

Re: [Cloud] Wikidata query request

ds (by > retrieving its contents), as it could include false positives or be behind > on updates. > > On Sun, Mar 11, 2018 at 3:44 PM, Huji Lee wrote: > >> Hello, >> >> I need help writing a query that I would like to run on the Clouds. The >> goal of the quer

Re: [Cloud] Wikidata query request

n HTML, RDF XML, Turtle, LD-JSON, … depending on Accept > header). > > Cheers, > Lucas > > 2018-03-14 1:03 GMT+01:00 Huji Lee : > >> Thanks, Jaime, for your recommendation. >> >> If I understand the result of [1] correctly, there are around 3.5 million &

Re: [Cloud] Wikidata query request

/wiki/Wikidata:Database_download and parse it myself. Thanks again! Huji On Wed, Mar 14, 2018 at 12:12 PM, Huji Lee wrote: > Lucas, > > No I don't need the page_id. The other two are enough. > > Wikidata Query Service seems very slow (it'll take about one day of > c

Re: [Cloud] Wikidata query request

fast the query service > can pump out the data. (If it was really computationally expensive, it > would be killed after sixty seconds anyways.) > > 2018-03-14 17:34 GMT+01:00 Huji Lee : > >> Actually, never mind. I reviewed the Java code behind it and it doesn't >> su

[Cloud] Upgrade to MariaDB 10.2

Hi all, Does anyone know when we will be upgrading the Clouds replica DBs to MariaDB 10.2? I am asking mainly because we are on 10.1.33, 10.3 is now out, and since 10.2 support for CTEs (WITH statements) has been added which is very handy. Thanks, Huji ___

Re: [Cloud] Upgrade to MariaDB 10.2

CTEs are not an absolute must-have, they are just very nice to have. Thanks for explaining the decision process. I can certainly wait longer. Huji On Tue, Jul 17, 2018 at 3:43 AM, Jaime Crespo wrote: > Hi, > > There is no timeline at the moment for an upgrade. We are finishing at the > moment

Re: [Cloud] [Cloud-announce] Addshore and Legoktm granted Toolforge admin rights

Fantastic! Congrats, and thanks for taking the new responsibilities. Huji On Sun, Jul 29, 2018 at 10:21 AM, Bryan Davis wrote: > I am happy to announce that Addshore and Legoktm have been granted > admin (root) level privileges in the Toolforge project. Both are long > time users of Toolforge an

[Cloud] pip

Hi all, What is the best way to install pip on the Clouds servers? I have a pywikibot program that depends on a pip-based package, and would like to be able to install that dependency using *pip install*. Thanks, Huji ___ Wikimedia Cloud Services maili

Re: [Cloud] pip

ll --upgrade pip*$* pip install > whatever > > Cheers, > M > > > 2018-08-11 2:51 GMT+02:00 Huji Lee : > >> Hi all, >> >> What is the best way to install pip on the Clouds servers? I have a >> pywikibot program that depends on a pip-based package, and

Re: [Cloud] pip

the python3-venv package using the following command. apt-get install python3-venv You may need to use sudo with that command. After installing the python3-venv package, recreate your virtual environment. On Sat, Aug 11, 2018 at 8:57 AM Huji Lee wrote: > Perfect, thanks! > > On Sa

Re: [Cloud] pip

I think I found the solution actually. I found it on https://wikitech.wikimedia.org/wiki/ORES/Deployment#Update_wheels and am pasting it below as well: virtualenv -p python3 venv source venv/bin/activate pip install --upgrade pip pip install whatever On Sat, Aug 11, 2018 at 10:20 AM Huji Lee

Re: [Cloud] pip

Pywikibot > Perhaps you could add some details of what you got to work, in the > appropriate place? (I assume the latter?) Thanks! :) > > > On Sat, Aug 11, 2018 at 7:28 AM Huji Lee wrote: > >> I think I found the solution actually. I found it on >> https://wikitech.wi

Re: [Cloud] pip

t; po 13. 8. 2018 v 3:39 odesílatel Huji Lee napsal: > >> I plan to do so, but I want to figure out all the nuances first. >> >> For instance, the method I discovered and mentioned above does not seem >> to support all pip packages (i tried `pip install webcite` and it c

[Cloud] Python gives "invalid syntax" error only on labs

Hello, When I run this script[1] on my own machine it runs without any problem. When I run it on Labs, I get an invalid syntax error, pasted below. Can you please help me figure out why? Thanks! (venv)tools.huji@tools-bastion-03:~$ python core/pwb.py webcite -page:"SOME PAGE" Traceback (most rec

Re: [Cloud] Python gives "invalid syntax" error only on labs

t sure. On Sun, Aug 19, 2018 at 4:48 PM Alex Monk wrote: > What are the versions of Python involved? > > On Sun, 19 Aug 2018 at 21:34, Huji Lee wrote: > >> Hello, >> >> When I run this script[1] on my own machine it runs without any problem. >> When I run it o

Re: [Cloud] Python gives "invalid syntax" error only on labs

t; probably just use *r'...'* for your regular expression and settle on > Python 3 for both local development and on Toolforge. > > Hope this helps! > > 2018-08-19 21:57 GMT+01:00 Huji Lee : > >> Excellent question. On my own computer, I am using 2.7.12 but on labs

[Cloud] List of users who have access to certain references

This is an idea that came up on fawiki, and there is some merit to it. I just want to figure out the best approach to implement it and would love your input. *TL;DR: *We want to sweep through the recent edits in articles, look at each diff, see if it contains the addition of a "{{cite book}}" temp

Re: [Cloud] List of users who have access to certain references

Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for > Windows 10 > > > > *From: *Huji Lee > *Sent: *Thursday, December 27, 2018 11:42 AM > *To: *Labs > *Subject: *[Cloud] List of users who have access to certain references > > > > This is an idea that

Re: [Cloud] List of users who have access to certain references

more than 1-2 > seconds especially if you optimize the logic. I’m just spitballing ideas at > this point, but the logic should be easy > > On Thu, Dec 27, 2018 at 12:37 PM Huji Lee wrote: > >> We will never know who "owns" which book. We only know that they have >

Re: [Cloud] Advice about scheduling cronjobs

Out of curiosity, did you intentionally leave out the other hours in the day? I am asking because my account has four jobs and they are all either on hour 11 or hour 12 of the day (either daily or weekly), but I didn't see any numbers for hour 11 in your email. On Wed, Mar 20, 2019 at 8:46 PM Gio

Re: [Cloud] Upgrade to MariaDB 10.2

I wanted to follow-up on this question from 10 months ago and ask if there are any plans to upgrade to MariaDB 10.2 or higher Besides CTEs, another important feature added in 10.2 is the windowed functions LEAD() and LAG() On Tue, Jul 17, 2018 at 6:29 PM Huji Lee wrote: > CTEs are not

Re: [Cloud] Toolforge SQL performance issue with new comment table

Out of curiosity, Jesse, can you share the query after that change, so it is all stored in this thread for future reference? On Sun, Jun 2, 2019 at 11:09 PM Jesse Plamondon-Willard < pathosch...@gmail.com> wrote: > Thanks! Performance is much better with that change. > > > On Sun, 2 Jun 2019 at 2

Re: [Cloud] Toolforge SQL performance issue with new comment table

It may help to put things in context and remind everyone that among the most visited websites in the world (of which, Wikipedia is #5 per Alexa [0]) the vast majority do not even use a relational database, making real-time large-scale analytic queries something that is not easily available to the u

[Cloud] Another SQL optimization request

Hi all, I have a query that used to do a GROUP BY on rev_user and now I have migrated it to do so using rev_actor instead. The latest version looks like this and its performance is not good

Re: [Cloud] Another SQL optimization request

All great answers; thank you! Let's wait and see what comes out of T224850. On Tue, Jun 4, 2019 at 8:21 PM Bryan Davis wrote: > On Tue, Jun 4, 2019 at 4:54 PM Huji Lee wrote: > > > > Hi all, > > > > I have a query that used to do a GROUP BY on rev_user and n

Re: [Cloud] Another SQL optimization request

but one that >> obtains equal or similar results. >> >> On Wed, Jun 5, 2019 at 2:21 AM Bryan Davis wrote: >> > >> > On Tue, Jun 4, 2019 at 4:54 PM Huji Lee wrote: >> > > >> > > Hi all, >> > > >> > > I have a query

[Cloud] Question about NOT EXISTS

Hi all, The execution plan for this query [1] indicates that it will be entirely based on existing indexes. Yet in practice, the query takes a long time to run. Why is that? Also, interestingly, the execution plan for this modified version [2] is identical, but it runs almost instantaneously. La

[Cloud] Issue with pip

Hi all, I am trying to run a pywikibot script on the Cloud servers. Please find the script at [1]. The script depends on a few pip packages include cidr_trie. I created a virtual environment under my tools account and installed these dependencies using "pip3 install ..." commands. When I ran the b

Re: [Cloud] Issue with pip

res python 3.6 or greater. You > can check what python is available in Toolforge and use a suitable version > for your environment or change the module to be compatible with earlier > versions. > > Cheers, > > Chico Venancio > > Em ter, 30 de jul de 2019 21:55, Huji Lee esc

Re: [Cloud] Issue with pip

Indeed, I modified the module on my venv and now it works fine. But if there is a better way, I am still eager to know. Thanks! Huji On Tue, Jul 30, 2019 at 9:30 PM Huji Lee wrote: > Good catch! It seems like we have 3.5.3 on Toolforge. Being a light user > of pip, I have no clue how to

Re: [Cloud] Issue with pip

there with pip install git+https://github.com/Huji/cidr-trie.git See < > https://pip.pypa.io/en/stable/reference/pip_install/#vcs-support > for > details. > Chico Venancio > > > Em qua, 31 de jul de 2019 às 11:03, Bryan Davis > escreveu: > >> On Tue, Jul 30,

[Cloud] SQL Question

I have a question for the MySQL/MariaDB experts. *Short and sweet* How is that Query 1 runs in seconds and so does Query 2 , but Query 3 -- which is essentially the same thing

Re: [Cloud] SQL Question

ly all run equally fast. > > Cyberpower678 > English Wikipedia Account Creation Team > English Wikipedia Administrator > Global User Renamer > > On Aug 7, 2019, at 22:15, Huji Lee wrote: > > I have a question for the MySQL/MariaDB experts. > > *Short an

Re: [Cloud] SQL Question

Certainly doable! But I didn't ask this here because of not having a way to achieve my goal; I asked it because I am perplexed as to why the query never finishes, despite its relatively simple structure and row number of rows returned. On Tue, Aug 13, 2019 at 7:43 PM Platonides wrote: > Why not

Re: [Cloud] SQL Question

bly whats > slowing you down > > On Tue, Aug 13, 2019 at 7:46 PM Huji Lee wrote: > >> Certainly doable! But I didn't ask this here because of not having a way >> to achieve my goal; I asked it because I am perplexed as to why the query >> never finishes, despite its

Re: [Cloud] SQL Question

Amazing! It did not occur to me that I should try it outside of Quarry (I just love and trust Quarry so much ... ;) Thank you all for your help. I am going to move this to Phabricator as a task for Quarry. On Wed, Aug 14, 2019 at 9:30 AM Brad Jorsch (Anomie) wrote: > On Wed, Aug 14, 2019 at 9:1

[Cloud] Can this query be optimized further?

I have a query that tries to find pages in article namespace that contain a link to some page in the user/user talk namespace. Many of these result from users signing in the articles (which is obviously not appropriate). The query also tries to exclude pages that have a legitimate link to user page

[Cloud] Loading wikipedia dump onto Clouds

Hi all, We have a set of database reports (on users, articles, etc.) that we used to generate on a weekly basis.[1] Ever since the introduction of the *actor* table,[2] many of the reports that have to do with users have become so slow that the SQL query cannot finish within a reasonable time and

Re: [Cloud] Toolforge: new domain toolforge.org

Thanks for sharing the news! When I log into login.toolforge.org the login message that is shown still references the old domain name. Can we update that too? On Mon, Apr 13, 2020 at 7:25 AM Arturo Borrero Gonzalez < aborr...@wikimedia.org> wrote: > Hi! > > We are happy to announce the new domain

Re: [Cloud] Toolforge: new domain toolforge.org

13, 2020 at 11:52 AM Huji Lee wrote: > Thanks for sharing the news! > When I log into login.toolforge.org the login message that is shown still > references the old domain name. Can we update that too? > > On Mon, Apr 13, 2020 at 7:25 AM Arturo Borrero Gonzalez < > aborr...

Re: [Cloud] Loading wikipedia dump onto Clouds

for now) would be a reasonable starting point, no? On Mon, Apr 13, 2020 at 4:42 PM Bryan Davis wrote: > On Sun, Apr 12, 2020 at 7:48 AM Huji Lee wrote: > > > > One possible solution is to create a script which is scheduled to run > once a month; the script would download the

Re: [Cloud] Loading wikipedia dump onto Clouds

re before. > > ~ MA > > [0] > https://wikitech.wikimedia.org/wiki/News/Actor_storage_changes_on_the_Wiki_Replicas#Advanced_use_cases_of_specialized_views > > On Mon, Apr 13, 2020 at 5:03 PM Huji Lee wrote: > >> I understand. However, I think that the use case we are look

Re: [Cloud] Loading wikipedia dump onto Clouds

b tickets on which I will seek advice about how to optimize those super-slow queries (if at all possible). Thank you for your attention! On Tue, Apr 14, 2020 at 8:49 AM Huji Lee wrote: > Yes. If you go to the source of all those pages, there is a hidden HTML > element ( kind) that has the SQL

Re: [Cloud] Toolforge: new domain toolforge.org

Quick question: what does it take for a JS script to be allowed to interact with a toolforge tool? When I run something like this, I get a Cross-Origin Request Blocked error in the console: On Tue, Apr 14, 2020 at 6:40 AM billinghurst wrote: > Checking our interwiki map, we have the following:

Re: [Cloud] Toolforge: new domain toolforge.org

that are being referenced through JS. On Thu, Apr 16, 2020 at 4:55 PM Huji Lee wrote: > Quick question: what does it take for a JS script to be allowed to > interact with a toolforge tool? > > When I run something like this, I get a Cross-Origin Request Blocked error > in the conso

Re: [Cloud] Toolforge: new domain toolforge.org

That was a general question. Thanks for clarifying. On Thu, Apr 16, 2020 at 6:21 PM Bryan Davis wrote: > On Thu, Apr 16, 2020 at 2:56 PM Huji Lee wrote: > > > > Oops, I don't know how I pressed the Send button, sorry! > > > > When I run something like this from

Re: [Cloud] Loading wikipedia dump onto Clouds

53 AM Huji Lee wrote: > I completely appreciate the points you are making, Bryan and Jaime. And I > would very much enjoy "dealing with you" if we end up going to "Cloud VPS > project" route! If anything, I keep learning new things from you all. > > Let's

[Cloud] User tables

Hi all, Is it possible to store data into user tables through queries on Wikireplica DBs? Or is it only possible by mysqldump'ing from the replica DB and loading into the user table in a separate step? I am thinking of aggregate data. Thanks! ___ Wikimedi

Re: [Cloud] User tables

> SRE > Wikimedia Cloud servicesbst...@wikimedia.org > IRC: bstorm_ > > On 5/8/20 12:31 PM, Huji Lee wrote: > > Hi all, > Is it possible to store data into user tables through queries on > Wikireplica DBs? Or is it only possible by mysqldump'ing from the replica >

Re: [Cloud] User tables

trying to understand my options. On Fri, May 8, 2020 at 5:07 PM Martin Urbanec wrote: > Hmm, does create temporary table work? If not, can it? > > Martin > > On Fri, May 8, 2020, 11:04 PM Huji Lee wrote: > >> I was thinking of running a query, temporarily store its outpu

Re: [Cloud] User tables

Super informative! Thank you Bryan and Jaime. I will take a look at that task and see if I can be of any help. On Sat, May 9, 2020 at 1:38 AM Jaime Crespo wrote: > One clarification, > > While everything Bryan says is right, I have to say that the main factor > in not allowing writes on wikirepl

Re: [Cloud] User tables

base 'fawiki_p' On Fri, May 8, 2020 at 5:07 PM Martin Urbanec wrote: > Hmm, does create temporary table work? If not, can it? > > Martin > > On Fri, May 8, 2020, 11:04 PM Huji Lee wrote: > >> I was thinking of running a query, temporarily store its output on &g

Re: [Cloud] [Cloud-announce] Fwd: Phasing out the .wmflabs tld on September 8th

This is also a good time remind everyone to update their ProxyJump configuration [1] to include the new TLDs. At least I would have forgotten it if I had not seen this remidner. [1] https://wikitech.wikimedia.org/wiki/Help:Accessing_Cloud_VPS_instances#ProxyJump_(recommended) On Mon, Sep 7, 202

Re: [Cloud] [Cloud-announce] Wiki Replicas 2020 Redesign

Cross-wiki JOINS are used by some of the queries we run regularly for fawiki. One of those queries looks for articles that don't have an image in their infobox in fawiki, but do have one on enwiki, so that we can use/import that image. Another one JOINs fawiki data with commons data to look for red

Re: [Cloud] [Cloud-announce] Wiki Replicas 2020 Redesign

One some level, the real issue here is that different wikis are living on different slices (s1, s2, s3). One possible solution is to replicate "shared" wikis (Wikidata and Commons) and possibly a few other "mother" wikis (at least En WP) into *every* slice. The uses cases that need to join enwikti

Re: [Cloud] [Cloud-announce] Wiki Replicas 2020 Redesign

I like the idea of dumps as an alternative too. But I think this should be a service that is offered via the WM Clouds. Some might remember me asking related questions on this very mailing list several months ago. Having a DB called "latest_dump" which actually has the latest dump of all wikis wou

Re: [Cloud] [Cloud-announce] Wiki Replicas 2020 Redesign

ok like. If you have looked > at this and done changes to your queries and code it would be super helpful > if you can point me to the repo/code to show real examples in the > documentation. > > If you can share real use cases of your use cases like Huji Lee and > MusikAnimal, it

Re: [Cloud] [Cloud-announce] New Wikireplicas available, timeline update, and Quarry migration

I said it before, and I say it again: *some* databases should be available for cross-wiki JOIN everywhere. This would at least include commons_p and centralauth_p but perhaps also enwiki_p and meta_p I know that we discussed it before and better long-term solutions can be imagined (such as a data

Re: [Cloud] [Cloud-announce] New Wikireplicas available, timeline update, and Quarry migration

media Foundation and they do quite a lot to support us with the > little resources they have. Your frustrations are perfectly valid though. > This is a very frustrating situation. > > On Wed, Mar 31, 2021 at 7:01 AM Huji Lee wrote: > >> I said it before, and I say it again:

Re: [Cloud] [Cloud-announce] Wikireplicas: old cluster migrations start in 2 weeks. Please test your code with the new cluster

Pardon my potentially dumb question, but how do the shorter DB names get resolved? I have python code that looks like this: *conn = mysqldb.connect(host="enwiki.labsdb", db="enwiki_p", read_default_file="~/replica.my.cnf")* How does *enwiki.labsdb* get resolved to, I am assuming, *enwiki.web.db.

Re: [Cloud] [Cloud-announce] Wikireplicas: old cluster migrations start in 2 weeks. Please test your code with the new cluster

Using the _p at .labsdb created a simple >> uniform way to connect without having to keep track of what database moved >> to which slice (when that was added mid size wikis were being moved >> regularrly >> >> On Tue, Apr 6, 2021 at 3:11 PM Brooke Storm wrote: >>

[Cloud] IP WHOIS tool on Cloud

Hi all, I opened https://phabricator.wikimedia.org/T280726 moments ago. It was un-assigned from the Cloud-Services tag but I think it is critical for our Clouds gurus to see it and opine on it. This is just an FYI email. I recommend that we keep the discussion on the Phab ticket as much as possib

Re: [Cloud] Wiki replicas: Compilation of input and logged cross-DB queries

Hi, Joaquin! I was looking at the page the other day. I agree with your summary above, except I would add one more prominent case which somehow did not appear in the random query analysis but I think is critical: - Querying CentralAuth and another DB An example is this query

[Cloud] Porting the output of qstat to a web page

Hi, Some of the jobs I submit to the grid take a long time (say, 30-60 minutes) and I would like to check on their status without having to log back into the labs. I was hoping I could run shell_exec('qstat') in PHP and display its output on a web page. While shell_exec() works with other command

[Cloud] Re: Porting the output of qstat to a web page

command line, both of those return an XML as expected. Any ideas as to how that could be the case? On Tue, May 11, 2021 at 6:18 PM Bryan Davis wrote: > On Tue, May 11, 2021 at 2:16 PM Huji Lee wrote: > > > > Hi, > > > > Some of the jobs I submit to the grid take a lon

[Cloud] Re: Porting the output of qstat to a web page

That explains it! Thanks! On Tue, May 11, 2021 at 10:29 PM YiFei Zhu wrote: > On Tue, May 11, 2021 at 8:45 PM Huji Lee wrote: > > > > @Bryan: this is great! In fact, https://sge-jobs.toolforge.org/tool/huji > completely serves my current needs. I added the links to the

[Cloud] Python upgrade on the Toolforge

Hi all, When I run python3 --version on Toolforge I see version 3.5.3 is installed. Because python 3.5 reached the end of its life in September 2020, pip is really unhappy about that. Is there a way to use a later version of python3 on Toolforge? If not, are there plans to upgrade the OS and upgr

[Cloud] Re: Python upgrade on the Toolforge

If you've done that kind of thing before, > it's a reasonable way to go, but it's not for everybody. > > Of course, at this point, even 3.7 is pretty old. At some point, I'll > probably break down and build 3.9. I really want the better f-strings that > came out

[Cloud] Source command not working with zsh

Hi all, Starting Nov 7, a number of the jobs I would run through Toolforge grid have stopped working. Each job consists of a .sh file like this on the first line of which I use the source command to activate a p

[Cloud] Re: Source command not working with zsh

e right executable, the shell > won't figure it out for you. > > On Thu, Nov 11, 2021 at 6:57 PM Huji Lee wrote: > > > > Hi all, > > > > Starting Nov 7, a number of the jobs I would run through Toolforge grid > have stopped working. Each job consists of

[Cloud] Re: Source command not working with zsh

;t see any reason why it shouldn't work, but > > > On Nov 11, 2021, at 7:44 PM, Huji Lee wrote: > > And also about the proper way to change the shell for tools accounts. > > > ___ > Cloud mailing list -- cloud@

[Cloud] Re: Source command not working with zsh

gt; > In truly bizarre cases, there might be some permission problem, which you > can solve with: > > touch /tmp/this-is-my-environment > chmod 0666 /tmp/this-is-my-environment > > > On Nov 12, 2021, at 10:21 AM, YiFei Zhu wrote: > > On Thu, Nov 11, 2021 at 3:57 PM Huji

[Cloud] Re: Source command not working with zsh

I had the chance to try out these options. *YiFei *was right in that the scripts, when invoked by crontab after zsh was activated, were being invoked by sh and that was why the *source* command was not working. Using . (dot) would resolve the issue on command line, but I cannot tell jsub to use d

[Cloud] Re: Source command not working with zsh

Again, good advice by both of you! Let me explore more and get back with any potential questions. On Sun, Nov 14, 2021 at 9:24 AM Roy Smith wrote: > This is really good advice. Any time you've got a process being run by > some tool on your behalf (cron, initd, remote job execution, etc), you're

[Cloud] Re: Source command not working with zsh

am still curious as to the root cause. Is it really that zsh is not available on the grid, and the grid tries to replicate my environment first and reaches the "exec zsh" command and falls apart somehow? On Sun, Nov 14, 2021 at 10:54 AM Huji Lee wrote: > Again, good advice by both

[Cloud] Re: Source command not working with zsh

PM Roy Smith wrote: > Submit a job that does: > > ls -l /usr/bin/zsh > > > and see what it says. > > > On Nov 16, 2021, at 9:37 PM, Huji Lee wrote: > > Is it really that zsh is not available on the grid, and the grid tries to > replicate my environment first and

[Cloud] Re: Source command not working with zsh

2021 at 6:38 PM Huji Lee wrote: > > > > > > I went back and reactivated the line in .bash_profile which enabled > zsh ("exec zsh" as the last line of .bash_profile) > > > > > > Then I submitted the job to the grid, using a command like this: &g

[Cloud] Mounting Toolforge on local Linux machine

Hi all, I had previously found a way to mount the home directory of my tool on Toolforge onto my local Linux machine using a *mount* command. It was so handy that I turned it into a bash alias and used it on the daily. Sadly, due to a hardware malfunction, I lost my machine and my .bash_aliases f

[Cloud] Re: Mounting Toolforge on local Linux machine

permutations and once I find it, I will certainly put it on the Wikitech wiki. On Sun, Dec 26, 2021 at 1:32 AM YiFei Zhu wrote: > This? > > $ sshfs login.toolforge.org:/data/project/toolname /path/to/mountpoint > > YiFei Zhu > > On Sat, Dec 25, 2021 at 6:39 PM Huji Lee wrote: >

[Cloud] Re: Mounting Toolforge on local Linux machine

The missing parameter was -o allow_other I documented this at https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Tool_Accounts#Mount_your_tools_home_directory_onto_your_local_machine On Sun, Dec 26, 2021 at 8:49 AM Huji Lee wrote: > Something like that, except I am pretty sure I had to

[Cloud] Re: [Cloud-announce] [IMPORTANT] Announcing Toolforge Debian Stretch Grid Engine deprecation

I too use bash files (see https://github.com/PersianWikipedia/fawikibot/tree/master/HujiBot/grid/jobs) and am planning to move them from grid to k8s in the next few weeks. I have found the process well-documented and am hoping for the transition to be easy, like Amir said. Since at least two peopl

[Cloud] Re: Admin Samyar

Thank you for this message. However, this listserv is not a place to send an article to. You may want to consider editing the Persian Wikipedia instead. (Persian translation below) به فارسی: از پیام شما ممنونیم. اما این فهرست ایمیلی، مکانی برای ارسال مقاله‌ها نیست. ممکن است بخواهید که در ویکی‌پدیا

[Cloud] Re: [Cloud-announce] Call for Quarry maintainers

That is sad, Vivian. I love Quarry. But then again, at this point, is there any feature that Quarry offers which Superset doesn't? Maybe this is a natural time for Quarry to be sunset? On Tue, Oct 3, 2023 at 4:59 PM Vivian Rook wrote: > Hello, I'm writing to notify any interested party that I a

[Cloud] Re: Changes in schema of pagelinks tables

Yes, a communication would be great (including, on this thread). As it stands right now, the linktargets table is incomplete on the wikis I checked. For instance, this fawiki query returns 388 results but the equivalent query

[Cloud] zsh is slow on login.toolforge.net

Hi, Today, when I tried to log into login.toolforge.net after about a couple weeks of not having logged in, I got an error about the fingerprint for the server having changed. Did we upgrade servers? Anyway, I updated my known_hosts file and was able to login but then I noticed that opening zsh (

[Cloud] Re: zsh is slow on login.toolforge.net

nen wrote: > > On Wed, Apr 24, 2024 at 4:27 AM Huji Lee wrote: > > > Today, when I tried to log into login.toolforge.net after about a > couple weeks of not having logged in, I got an error about the fingerprint > for the server having changed. Did we upgrade servers? > > &

[Cloud] LLM services

Hi all, Are there any LLMs available on Cloud services, or are there any plans for them? I think there are many possible use cases. Even free, lightweight LLMs (like LLaMa) could be helpful, e.g. in bots that review edits, categorize pages, etc. Thanks, Huji __