[PATCH] ob-shell-test, test-ob-shell and introduction
Hi, I'm interested in getting async into ob-shell.el. Since I've never contributed before, I figure it'd be good to start with a few easy tasks. It looks to me like the stdin and cmdline header args aren't documented anywhere (at least I couldn't find anything). To make sure I'm using them correctly before making a patch for the manual, here are some tests. Please let me know if things look okay. It wasn't clear to me how to send along a message with git send-email, so I formatted these patches and included them as an attachment. Is that fine? Thanks, Matt 0001-ob-shell-test.org-Add-example-for-stdin.patch Description: Binary data 0002-test-ob-shell.el-Add-tests-for-stdin-and-cmdline-hea.patch Description: Binary data
[PATCH] ob-doc-shell.org (was [PATCH] ob-shell-test, test-ob-shell and introduction)
> Unfortunately, there is no ob-doc-shell.org at > worg/org-contrib/babel/languages/. > > There is an ob-doc-template.org in case you'd like to contribute > ob-doc-shell.org :) I created one, along with a sourcehut account. However, it looks like I need to be granted some permissions to write to the worg website. Is Bastien the contact for that? I've attached a patch in the mean time. 0001-org-contrib-babel-languages-ob-doc-shell.org-Create-.patch Description: Binary data
Re: [PATCH] ob-shell-test, test-ob-shell and introduction
> [FSF copyright assignment]. Have you done that yet? I just verified with my employer that my contract grants an exception for this project. Just emailed the request to ass...@gnu.org. Also, got access from Bastien for worg. I figure it's probably best to reserve any more changes 'til the paper work is done?
Re: [PATCH] ob-shell-test, test-ob-shell and introduction
> > I just verified with my employer that my contract grants an exception for > > this > > project. Just emailed the request to ass...@gnu.org. Not surprisingly the FSF hasn't resources to verify my contract's exception and needs a written disclaimer from my employer. I'm waiting on this now. > Feel free to develop and > share patches before the assignment is complete, we’ll just wait till it’s > gone > through before merging Given the contract exception, I'll start moving forward again with the assumption that I'll eventually get that written disclaimer. > Hope that helps. It does, thank you. It's nice to know that my words aren't going into the void. :)
Re: [PATCH] ob-shell-test, test-ob-shell and introduction
> Contributions to Worg aren't similarly restricted. Feel free to > push material there in the meantime. Looks like the email finally got sent to the right person at my company. Who knows how long it will take for them to get the FSF disclaimer back to me... Now that I'm on holiday, I've got a little more time to contribute to Worg. Here's a few patches showing what I've come up with. I'll finish it up in the next few days, hopefully. Not sure if the patches are formed correctly, sorry about that if not. It's late here and I wanted to show progress. I'll worry about those kinds of details later. :) 0001-Draft-introduction-and-setup.patch Description: Binary data 0001-org-contrib-babel-languages-ob-doc-shell.org-Create-.patch Description: Binary data
Re: [PATCH] ob-shell-test, test-ob-shell and introduction
> Wow. Nice work! Thanks. I pushed things to Worg, if you haven't seen already (https://orgmode.org/worg/org-contrib/babel/languages/ob-doc-shell.html). If you read it and find anything missing or unclear, please let me know. I'm still waiting on work to sign the FSF disclaimer in order to work on ob-shell.el. I might just implement something on the side and use that to inform any future contributions.
Re: [PATCH] ob-shell-test, test-ob-shell and introduction
Apologies for the book. I've been sitting on this stuff for two months and am wondering how to proceed. IANAL but AFAIK/CT, my contract contains an exception for making contributions to projects like Org. I've gotten confirmation from my manager and by HR. However, until the CEO signs the FSF disclaimer, I can't officially contribute. I'm confident that I can publish changes (e.g. to a personal website); the FSF just can't accept my changes (yet). I could start working on ob-shell.el separately now and publish that myself. It's not clear how I would then contribute those changes back once I'm cleared by the FSF. I'm inclined towards some refactoring and I'm not sure how I could break that down in such a way that if it took two more months to get the copyright stuff worked out that anyone could make sense of the changes. I would much prefer to gradually submit patches, discuss them, and then eventually have them merged in turn. If I have a heap of changes elsewhere, I feel like it would be harder to have a conversion about them. Regardless, I think I should define test cases first. If those are considered valid, then any refactoring would be moot if they pass, right? With (agreed upon) test cases, it shouldn't matter if I refactor as long as functionality remains the same? Overall, what advice do you have? It looks to me like ob-shell.el could use some love in other respects besides async evaluation. I've detailed where below, partly for my own organization and partly for posterity, but mainly because this isn't my house, so to speak, and I don't want to barge in and start rearranging the furniture and eating whatever's in the fridge. Or, is it like Worg in that once I have the keys I can drive where I like, so long as there're no crashes? I'm interested in people's thoughts on these notes on ob-shell.el: * Tests There are several code paths, like shebang, cmdline, and basic execution, which don't always have something to do with one another in the code. Having tests would be really helpful to make sure everything jives. Before doing anything with the code base, I intend to create tests for all the functionality. * 2D-array I documented two options for the =:var= header[fn:1]. The ob-shell.el code actually defines a third option for 2D-arrays. I couldn't get it to work. It was one of several things not documented anywhere, at least as far as I could find, and which I had to figure out straight from the code. Between not being great at shell scripting and having a hard time deciphering that ob-shell.el code, I'm not sure 2D-arrays are actually fully or correctly implemented. * M-up behavior <> The =org-babel-load-session:shell= code path only works when M-up is used on a code block[fn:2]. Is M-up actually documented anywhere? Furthermore, the =org-babel-load-session:shell= only works for the "shell" language, which is not actually a "proper" shell (i.e. it's not listed in =org-babel-shell-names=). The M-up defaults to using $ESHELL or shell-file-name through the =shell= command. For example, try calling M-up on these: #+comment: (opaquely) calls the system shell #+begin_src shell :session my-shell echo "hello, world!" #+end_src #+comment: fails #+begin_src sh :session my-sh echo "hello, world!" #+end_src #+comment: fails #+begin_src bash :session my-bash echo "hello, world!" #+end_src To fix this, there needs to be an =org-babel-load-session:= for each language in =org-babel-shell-names=. This would probably make the most sense in =org-babel-shell-initialize=. However, that function [[org-babel-shell-initialize][needs attention]]. * Refactoring <> The ob-shell.el code appears inconsistent to me. Clearly, this is somewhat subjective. I've tried to give a rationale for each point to make it less so. My goal is to be maintainer of ob-shell.el, but that's not forever. These things were an obstacle for me and my aim is to remove them for the next person. ** =org-babel-shell-initialize= <> As alluded to elsewhere, =org-babel-shell-initialize= doesn't appear to adhere to the (elisp) Coding Conventions, #+begin_quote • Constructs that define a function or variable should be macros, not functions, and their names should start with ‘define-’. The macro should receive the name to be defined as the first argument. That will help various tools find the definition automatically. Avoid constructing the names in the macro itself, since that would confuse these tools. #+end_quote The =org-babel-shell-initialize= function defines =org-babel-execute:=, =org-babel-variable-assignments:=, and =org-babel-default-header-args:= for the "languages" in =org-babel-shell-names=. As it stands, that =org-babel-shell-initialize= is a function does no harm (aside from being confusing by way of straying from convention). However, if the [[M-up][M-up]] issue is to be resolved, it seems to me that =org-babel-shell-initialize= should be update
Re: [PATCH] ob-shell-test, test-ob-shell and introduction
> Thank you, I was not aware of ":cmdline" argument and of ":shebang" as a > means to avoid stray prompts (I have seen mentions of similar problem > for other languages in this list). You're welcome! I was surprised to find those in the source. Glad to have documented them and even happier to hear that it's news to others. > - https://www.mirbsd.org/pics/logo-grey.png works only with http: > protocol for me. I updated (downgraded?) it to http. I'm also seeing that the kornshell logo is intermittently broken. Not quite sure the best way to go about fixing that. Maybe commit copies? Use something like archive.org? Sounds like a problem for someone else (e.g. future me). I'm out of time for tonight. > - I am unsure concerning example after "…or as a standlone script.", > maybe it should be wrapped into #+begin_example. Good call. That example was wonky. I've given a different one. Not sure it's better, though. I find the shebang behavior tricky to demo in a simple way. > - In some cases "sh" is used despite bashisms in the code like "declare" > or "echo -ne". Actually "printf" may be more convenient instead of "echo -n" > - "export" in session example is unnecessary. It is matter of taste > though. Bash and dash is not the case, but some shells require that > assignment and export should be separate commands. Good catch! Since arrays are only defined in ob-shell using bash, I've updated that example to use bash. Otherwise, I've removed the exports. > - In your examples variable values are simple. Often it is safer to add > double quotes around variable or command expansion. I would consider > adding quotes just to encourage people to do the same by default with > hope that less scripts will give strange results in response to a file > name containing a space. Actually my first impression was that > backslashes before quotes in some cases were added by mistake. Another > unescaped pair of quotes may make your intention more clear. However it > is related to code style where everybody has opinion, so I do not insist. I appreciate these clarifications. Admittedly, I'm not great with shell scripting. Your recommendations appear sound to me and I've tried to incorporate them. The updates have been made and are pushed. Thanks for your feedback!
Re: Shell SRC blocks under Windows
On Sat, 05 Feb 2022 10:48:02 -0500 H. Dieter Wilhelm
wrote
> >> Is is it possible to extend org-mode's src blocks with
> >> org-babel-shell-names to use either MSYS2 or MinGW shells (instead of
> >> cmd.exe)? Unfortunately ob-shell.el doesn't guide me further and I
> >> can't find examples or the respective lisp files on how to define the
> >> execution functions.
>
> Yes, you are right, when using
>
> > #+begin_src sh
>
> It seems that I'm also running some bash shell! :
Yes, it's likely possible to run those shells. Whether or not it will be
straight forward or not depends. It's not clear to me what you're trying to do.
First off, you're looking in the right place. Unfortunately, ob-shell.el can
be a little tricky to read. All the shells are initialized through the
=org-babel-shell-initialize= function. This creates a
"org-babel-execute:" for each of the shells given in the
=org-babel-shell-names= variable (where is given by #+begin_src )
. The default shells are: ("sh" "bash" "zsh" "fish" "csh" "ash" "dash" "ksh"
"mksh" "posh"). All execute calls make their way through the God function
=org-babel-sh-evaluate= (note: the "sh" in the function name appears to be
cruft from when the ob-shell.el module was called "ob-sh.el"; it is not tied
specifically to the sh shell). Technically, which shell is used to evaluate
the block depends on the source block parameters used. There are several
possible paths. Unless something weird is being done, however, the block
should be evaluated by . It looks like you're using a simple one-off
shell, not a session or a shebang, so I'll speak to that.
For #+begin_src sh, the block will we evaluated by whatever "sh" corresponds
to in the environment Emacs is running in. Under the hood, a call is made to
the =shell= command and the =shell-file-name= variable gets set to whatever
the block specifies. The an binding happens in
=org-babel-shell-initialize=. So, open a command line environment that would
be used to start Emacs and see what "sh" corresponds to:
which sh
That's the shell used by #+begin_src sh. Similarly, whatever "which bash"
shows is what #+begin_src bash uses.
This is why it would be surprising that cmd.exe would be used.
I know that was a lot. Depending, it may be as simple as updating the
=org-babel-shell-names= and rerunning =org-babel-shell-initialize=. Check the
doc strings on those. Otherwise, I hope this helps ob-shell.el guide you
better.
> But I'm not sure if I'm halfway there with "sh"? I need to run the
> following MSYS2 command AND switch between two arguments (for building
> Emacs).
>
> \MSYS2\msys2_shell.cmd -mingw64
>
> and
>
> \MSYS2\msys2_shell.cmd -msys.
>
> How could I achieve this within a src block?
I'm not sure I follow you here. Can you please describe what you're trying to
do and the system you're using (e.g. Windows, GNU/Linux, Haiku) as well the
shell used to run the Emacs instance you're doing this in?
Re: Shell SRC blocks under Windows
On Sat, 05 Feb 2022 17:22:29 -0500 Matt wrote > > But I'm not sure if I'm halfway there with "sh"? I need to run the > > following MSYS2 command AND switch between two arguments (for building > > Emacs). > > > > \MSYS2\msys2_shell.cmd -mingw64 > > > > and > > > > \MSYS2\msys2_shell.cmd -msys. > > > > How could I achieve this within a src block? > > I'm not sure I follow you here. Can you please describe what you're trying > to do and the system you're using (e.g. Windows, GNU/Linux, Haiku) as well > the shell used to run the Emacs instance you're doing this in? > Excuse me. Obviously you're using Windows. :)
Re: Shell SRC blocks under Windows
On Sun, 06 Feb 2022 12:51:49 -0500 H. Dieter Wilhelm wrote > Matt writes: > > > On Sat, 05 Feb 2022 17:22:29 -0500 Matt wrote > > > > > > > > But I'm not sure if I'm halfway there with "sh"? I need to run the > > > > following MSYS2 command AND switch between two arguments (for > > building > > > > Emacs). > > > > > > > > \MSYS2\msys2_shell.cmd -mingw64 > > > > > > > > and > > > > > > > > \MSYS2\msys2_shell.cmd -msys. > > > > > > > > How could I achieve this within a src block? > > > > > > I'm not sure I follow you here. Can you please describe what > > > you're trying to do and the system you're using (e.g. Windows, > > > GNU/Linux, Haiku) as well the shell used to run the Emacs instance > > > you're doing this in? > > > > > Excuse me. Obviously you're using Windows. :) > > Yes I have to, but please don't mind, thank you for your interest. :-) > I don't mind. I had asked about the system to help troubleshoot yet that info was clearly given in the thread title. :) You said you weren't sure if you're halfway there with "sh". Is that feeling resolved for you?
Re: Shell SRC blocks under Windows
On Tue, 08 Feb 2022 04:47:04 -0500 H. Dieter Wilhelm wrote > Not yet, I'm still grappling with your message > <17ecbfb46e9.e16df906236239.6523271905911672...@excalamus.com> and > ob-shell.el. I'll come back to you when I'm through. Okay, feel free to let me know if something I wrote is unclear or to ask any questions.
Re: How do you manage complex project with Org-mode
On Mon, 28 Feb 2022 20:43:47 -0500 Sébastien Gendre wrote > And I don't know how to manage this kind of projects with Org-mode. How > to do it, without failing a 6 days project because I spent to much time > on something else and I have only 3 days left with 3 half-day important > appointment I cannot cancel. I can't risk failing a single one of these > project by trying. So, when I am in a period with a lot of these > projects, I stop using Org-mode and concentrate on doing these project > as fast as I can. And because I often have this kind of project, I spend > most of the year without being able to use Org-mode. It sounds like you have a lot going on! If none of what you need to do explicitly requires Org, you may have to scale back what you learn about Org to fit the time you've got. Give yourself permission to accept that the time you have right now for Org isn't what you want. (I hope that's because you're learning lots of other cool things in school.) In situations like these, I like to do just a little each day. Maybe that means reading one paragraph a night before bed. It sounds like you're really excited about Org. (If you are, you've come to the right place. The people here love Org :) Reading about Org would be something fun to look forward to each night. You might be surprised at how motivating that one little paragraph can be! Working through the manual in this way will give you a good overview of how Org can be used and what you personally might use Org it for. > So, if you have any suggestion on how to manage, in Org-mode, projects > with: > * Lot of work to do (many days) > * Short deadline (not enough time) > * High importance (disastrous consequences in my future in case of fail) > * Many of them in the same time > * Progression need to be followed to chose where to sacrifice time to > limit the damages > > I will be happy to read them. :) My suggestion is to not try something new on anything that has a tight deadline. Org is new to you and learning things takes time. This is all normal. Life will throw a lot at you. Some times all I get is 20 minutes at the end of the day. Some days, I get nothing at all. This is because I've filled my life with other cool things, like a partner, a house, friends, etc. If I get to spend some time doing something I think is worthwhile (like trying to help a fellow Org enthusiast), that's time well spent in my book. Isn't there some saying that goes like, "every avalance starts as a snowflake?" Read a little, experiment a little, and over time, you'll be surprised at how much you've learned. You'll get there!
Thanks for good docs
Hi, I'm spending more time writing this email than took me to look up what the source block switch is for line numbering. I had an idea of what I wanted and didn't know where to look. Jump into info, go to the org manual, do a few searches across menu and node titles. Then I check the index and there's something promising...takes me right to what I was looking for. Thanks for indexing. Thanks for including examples. So much better than internet search engines. Y'all are doing good things. :) Thanks, Matt
Re: Org as a workspace (an impromptu reflection)
On Fri, 29 Apr 2022 11:03:55 -0400 Juan Manuel Macías wrote > I don't know if anyone has had a similar experience... I tell people that Emacs changed my life. I feel it's that profound. My story is different from yours, yet similar in that it started with Org. I love that it has this affect on people. It brings a smile to my face. This really is a great community. Thanks for sharing your experience. :)
Re: C-c $ missing?
On Thu, 18 Aug 2022 12:46:23 -0400 Summer Emacs wrote --- > > Hi everyone, first time posting. I hope it's okay to ask this here (I'm > totally new): Welcome! > Since this morning (I updated a few packages but not sure which), C-c $ > is missing in org mode? It's how I used to archive tasks when they were > done. I know that I can use C-c C-x C-s but I used to just use C-c $ and > it worked. Did something change with an update? > > (I think it was C-c $ or maybe C-x $? Sometimes I get the two confused) I don't know if anything changed with Org, I'll let others speak to that. However, in the mean time, check C-h k C-c C-x C-s to see what command is bound to that sequence (is it org-archive-subtree?). Then you can bind it with something like: (eval-after-load 'org-mode (define-key org-mode-map (kbd "C-c C-x C-s") #'org-archive-subtree)) This says, "after loading org-mode for the first time, define the key sequence within the variable telling Emacs what keys do what for org-mode to call the function org-archive-subtree when it sees C-c C-x C-s." The C-h f and C-h v commands describe-function and describe-variable can probably explain that better. > -- > Summer, > Sent from MU4E in Emacs, > Because I'm becoming a NEWBIE at this! We all start somewhere! I hope this helps. Just saw your message as I was passing by. If it's unclear, that's totally my fault!
Re: Bash results broken?
On Mon, 19 Dec 2022 19:44:12 -0500 Rudolf Adamkovič wrote ---
> Ihor Radchenko yanta...@posteo.net> writes:
>
> > Fixed on bugfix.
>
> I can confirm that the fix works. Thank you!
I can confirm it works, too, for the 1 2 3 example. However, it puts double
quotes around text:
#+BEGIN_SRC sh :results list
echo hello
echo world
echo !
#+END_SRC
#+RESULTS:
- "hello"
- "world"
- "!"
I was working on other ob-shell related stuff and my Emacs got into a mixed
state where =ob-shell/results-list= was defined, but the fix wasn't (or wasn't
loaded). I fixed the failing test before I saw this thread and Ihor's fix.
The way I solved it was to wrap either =org-babel-import-elisp-from-file= or
=org-babel-sh-evaluate= like,
(mapcar #'car (org-babel-import-elisp-from-file tmp-file))
or
(mapcar #'car (org-babel-sh-evaluate session full-body params stdin cmdline))
Doing this (without Ihor's fix in place) returns:
#+BEGIN_SRC sh :results list
echo "hello"
echo "world"
echo "!"
#+END_SRC
#+RESULTS:
- hello
- world
- !
The underlying issue, so far as I understand it, is that
=org-babel-import-elisp-from-file= returns a list-of-lists. For example, ((1)
(2) (3)) or (("hello") ("world") ("!")). I thought it appropriate to flatten
it out to (1 2 3) or ("hello" "world" "!").
Independently from how I addressed the bug, I feel it raises an important
question: how should :results list handle text/strings?
I'm not sure which approach is more "appropriate". If I echo a string without
quotes, as I did in the first example, should it return a list that's got
quotes? The hello world example is a little contrived–it's good practice to
use double-quotes. I think a more realistic use case would be listing a
directory. In that case, I'd probably want the list of items to be unquoted:
Original:
#+BEGIN_SRC sh :results list
ls /home/ahab/.emacs.d/straight/repos/org/testing/lisp/ | head -n 3
#+END_SRC
#+RESULTS:
- ("test-ob-awk.el")
- ("test-ob-C.el")
- ("test-ob-clojure.el")
Ihor solution:
#+BEGIN_SRC sh :results list
ls /home/ahab/.emacs.d/straight/repos/org/testing/lisp/ | head -n 3
#+END_SRC
#+RESULTS:
- "test-ob-awk.el"
- "test-ob-C.el"
- "test-ob-clojure.el"
MAPCAR solution:
#+BEGIN_SRC sh :results list
ls /home/ahab/.emacs.d/straight/repos/org/testing/lisp/ | head -n 3
#+END_SRC
#+RESULTS:
- test-ob-awk.el
- test-ob-C.el
- test-ob-clojure.el
For :results list, the manual says, "Interpret the results as an Org list. If
the result is a single value, create a list of one element". I don't find that
clarifies what would be best.
Thoughts?
ob-shell intentions and paperwork (was Bash results broken?)
On Fri, 16 Dec 2022 12:41:45 -0500 Ihor Radchenko wrote --- > We really need more tests. I'm working on giving ob-shell a little bit of love. I wrote the worg documentation for it earlier this year. I tried to include examples of all coded functionality, including previously undocumented features. I think the examples would make for good tests since they document the present and expected behavior. Currently, though, I'm refactoring the ob-shell tests to remove dependency on ob-shell-test.org and to stop the suite from littering. After that, I intend to incorporate the worg examples as tests. I'd like to adopt ob-shell, life permitting. Sadly, I'm not confident I'm able to commit to that right now. So, if I can get the FSF paperwork cleared, hopefully, I can make little contributions here and there. I've included patches of what I've been doing. They're not ready for actual submission yet, both in quality and legality. However, I'm including them for feedback, to make sure I'm not getting way off base. Overall, I'd *really* like to implement async for ob-shell similar to ob-python. However, I feel like it'd be more responsible and respectful to clean up the tests and code a little first before proposing such a big change (looking at you, org-babel-shell-initialize). Can someone please assist me with the FSF paperwork off list? 0001-Remove-mixed-tabs-and-spaces.patch Description: Binary data 0002-Split-test-ob-shell-dont-error-on-empty-results.patch Description: Binary data 0003-Refactor-test-ob-shell-session.patch Description: Binary data 0004-Refactor-ob-shell-generic-uses-no-arrays.patch Description: Binary data 0005-Refactor-ob-shell-bash-uses-arrays.patch Description: Binary data 0006-Refactor-ob-shell-generic-uses-no-assoc-arrays.patch Description: Binary data 0007-Refactor-ob-shell-bash-uses-assoc-arrays.patch Description: Binary data
Re: Bash results broken?
On Sun, 25 Dec 2022 06:23:53 -0500 Ihor Radchenko wrote --- > As you see, "%S" have been used previously for non-string results. > I cannot find explanation in git log. > > That said, I think that it will be more consistent to leave strings > specifically as is. See the attached patch. That makes sense and works for me.
Re: ob-shell intentions and paperwork (was Bash results broken?)
On Wed, 21 Dec 2022 01:17:50 -0500 Matt wrote --- > Currently, though, I'm refactoring the ob-shell tests to remove dependency > on ob-shell-test.org and to stop the suite from littering. Done. Branched off bugfix, 12e10eb0d, and refactored test-ob-shell.el. See attached patches. The main changes spanning the patches are: - tests are now standalone; they don't rely on ob-shell-test.org - tests now share a common prefix, "test-ob-shell". The entire suite runs with (ert "test-ob-shell") - buffers and processes created during a test are killed when the test passes > After that, I intend to incorporate the worg examples as tests. This is probably a good place for me to pause and get feedback before writing more code. I just got the signed paperwork back from Craig and am waiting to be admitted to the Emacs group on Savannah. 0001-test-ob-shell.el-Remove-mixed-tabs-and-spaces.patch Description: Binary data 0002-test-ob-shell.el-Split-cases-into-two-tests.patch Description: Binary data 0003-test-ob-shell.el-Refactor-test-ob-shell-session.patch Description: Binary data 0004-test-ob-shell.el-Refactor-ob-shell-generic-uses-no-a.patch Description: Binary data 0005-test-ob-shell.el-Refactor-ob-shell-bash-uses-arrays.patch Description: Binary data 0006-test-ob-shell.el-Refactor-ob-shell-generic-uses-no-a.patch Description: Binary data 0007-test-ob-shell.el-Refactor-ob-shell-bash-uses-assoc-a.patch Description: Binary data 0008-test-ob-shell.el-Refactor-ob-shell-simple-list.patch Description: Binary data 0009-test-ob-shell.el-Refactor-ob-shell-remote-with-stdin.patch Description: Binary data 0010-test-ob-shell.el-Refactor-ob-shell-results-table.patch Description: Binary data 0011-test-ob-shell.el-Refactor-ob-shell-results-list.patch Description: Binary data 0012-test-ob-shell.el-Refactor-ob-shell-standard-output-a.patch Description: Binary data 0013-test-ob-shell.el-Refactor-ob-shell-standard-output-a.patch Description: Binary data 0014-test-ob-shell.el-Refactor-ob-shell-error-output-afte.patch Description: Binary data 0015-test-ob-shell.el-Refactor-ob-shell-error-output-afte.patch Description: Binary data 0016-test-ob-shell.el-Refactor-ob-shell-error-output-afte.patch Description: Binary data 0017-test-ob-shell.el-Refactor-ob-shell-exit-code.patch Description: Binary data 0018-test-ob-shell.el-Refactor-ob-shell-exit-code-multipl.patch Description: Binary data 0019-ob-shell-test.org-Remove-ob-shell-test.org.patch Description: Binary data 0020-test-ob-shell.el-Organize-tests.patch Description: Binary data
Re: ob-shell intentions and paperwork (was Bash results broken?)
On Thu, 29 Dec 2022 06:08:59 -0500 Ihor Radchenko wrote --- > Does it mean that you are willing to maintain lisp/ob-shell.el? > We usually give write access to the maintainers and regular > contributors. AFAIR, you previously contributed to WORG but not to Org > core. You're correct, I've not contributed to core. I would love to maintain lisp/ob-shell.el. I'm expecting life changes in the coming months and can't anticipate how that will affect my time. Would it be a problem if I need to step down as maintainer for a period? > > From: Matt Trzcinski m...@excalamus.com> > > > > [fn:1] Session name is a string and not a variable (such as > > `session-name'). This is because `org-babel-execute:sh' produces a > > type error when the session name is defined with a variable. The > > source of the error appears to be the `params' symbol in > > `org-babel-execute:sh'. `params' does not evaluate as a variable as > > expected–it evaluates as a symbol. However, `org-babel-execute:sh' is > > defined within a function factory which makes it difficult to debug. > > Hard code the test name for now and refactor it later once > > `org-babel-execute:sh` is refactored. > > There is nothing wrong here. `org-babel-execute-src-block' takes care > about parameter processing making sure that :session value is always a > string. Originally the test used "yes" for a comparison string and the shell it ran in. I changed "yes" to be the test name because when it was "yes", I didn't know what test had produced it. However, the problem with the test name being hard coded as a string is that if the function name changes, the test string may get out of sync. More on this below. > > +;; TODO refactor into macro. Currently violates (elisp) Coding > > +;; Conventions and is hard to debug. > > (defun org-babel-shell-initialize () > >"Define execution functions associated to shell names. > > Could you please elaborate? Which particular convention does it violate? > What is hard to debug? (elisp) Coding Conventions says, "• Constructs that define a function or variable should be macros, not functions, and their names should start with ‘define-’. The macro should receive the name to be defined as the first argument. That will help various tools find the definition automatically. Avoid constructing the names in the macro itself, since that would confuse these tools." The `org-babel-shell-initialize' function defines *all* the `org-babel-execute:XXX' functions given by `org-babel-shell-names' (sh, bash, zsh, etc.). Because `org-babel-shell-initialize' is a function factory, you can't easily examine or modify their definitions. `C-h f org-babel-execute:sh' jumps to the top of lisp/ob-shell.el. Changing the definition requires reevaluating the definition for all the execute functions (or first changing `org-babel-shell-names'). This was a problem for me when I wanted to make the session name string for `test-ob-shell/session' the test name (mentioned above). In the test, when I replaced the session name string with a variable containing the string, `org-babel-execute:sh' failed with a type error. I couldn't get the variable to evaluate (with backquote and comma or otherwise). Without an explicit function definition or a macro to expand, I found it hard to debug/experiment with (and so left the test name as a hard coded string). > > (ert-deftest test-ob-shell/dont-insert-spaces-on-expanded-bodies () > >"Expanded shell bodies should not start with a blank line unless > > -the body of the tangled block does." > > +the strings of the tangled block does." > > What does "strings of the tangled block" refer to? The previous variant > is a lot more clear for me. I believe this is a query-replace error. Good catch! > > +(ert-deftest test-ob-shell/generic-uses-no-arrays () > > + "Test generic serialization of array into a single string." > > + (org-test-with-temp-text > > + (test-ob-shell-multiline-string > > + "#+NAME: sample_array" > > + "| one |" > > + "| two |" > > + "| three |" > > Why do you need `test-ob-shell-multiline-string' here? > Can simply type-in the string directly, as the rest of tests do. I probably don't need it and am happy to remove it.An older version of the function was more complex and made sense as a separate function (or so I thought). My aim was to make the test strings easy to read so that it&#x
Re: ob-shell intentions and paperwork (was Bash results broken?)
On Fri, 30 Dec 2022 00:34:38 -0500 Matt wrote --- > > On Thu, 29 Dec 2022 06:08:59 -0500 Ihor Radchenko wrote --- > > > From: Matt Trzcinski m...@excalamus.com> > > > +(require 'org-test (expand-file-name "../org-test.el")) > > > > I am unsure here. What will happen if you run this file from > > default-directory not the same with file location? I was mistaken. Loading a new Emacs instance and running `(load "/path/to/test-ob-shell.el")' fails with file-missing error. I'm content to leave that alone for now. I've backed out the `require' change and adjusted everything else based on your feedback. There is a separate patch for each refactor that created a new test. The remaining refactors are in a single patch. I was also able to resolve the issue I had with inserting the test name for the session (the "yes" string in `test-ob-shell/session'). The issue with `org-babel-shell-initialize' still stands but is outside the scope of these patches and isn't dealt with here. 0001-test-ob-shell.el-Split-test-ob-shell-dont-error-on-e.patch Description: Binary data 0002-test-ob-shell.el-Refactor-test-ob-shell-session.patch Description: Binary data 0003-test-ob-shell.el-Refactor-ob-shell-generic-uses-no-a.patch Description: Binary data 0004-Refactor-ob-shell-bash-uses-assoc-arrays.patch Description: Binary data 0005-test-ob-shell.el-Refactor-test-names-and-kill-test-b.patch Description: Binary data 0006-ob-shell-test.org-Remove-ob-shell-test.org.patch Description: Binary data 0007-test-ob-shell.el-Organize-tests.patch Description: Binary data
Re: ob-shell intentions and paperwork (was Bash results broken?)
On Sat, 31 Dec 2022 09:31:16 -0500 Ihor Radchenko wrote --- > Matt m...@excalamus.com> writes: > > > I've backed out the `require' change and adjusted everything else based on > > your feedback. There is a separate patch for each refactor that created a > > new test. The remaining refactors are in a single patch. I was also > > able to resolve the issue I had with inserting the test name for the > > session (the "yes" string in `test-ob-shell/session'). > > Thanks! > > I can see that you still prefer to use `string-join' for Org body. > Why not a bare string? Because I often program in Python where double quoted strings typically don't span lines. One of the original tests had the string "#+BEGIN_SRC sh :results list\necho 1\necho 2\necho 3\n#+END_SRC" in it (that is, used newline characters instead of inserting new lines), so my brain assumed strings Emacs Lisp behaved similarly to Python. Silly brain :) Thanks for continuing to follow up on that. I've updated the patches to use bare strings. If this set of patches look good, I can push them to main. 0001-test-ob-shell.el-Split-test-ob-shell-dont-error-on-e.patch Description: Binary data 0002-test-ob-shell.el-Refactor-test-ob-shell-session.patch Description: Binary data 0003-test-ob-shell.el-Refactor-ob-shell-generic-uses-no-a.patch Description: Binary data 0004-Refactor-ob-shell-bash-uses-assoc-arrays.patch Description: Binary data 0005-test-ob-shell.el-Refactor-test-names-and-kill-test-b.patch Description: Binary data 0006-ob-shell-test.org-Remove-ob-shell-test.org.patch Description: Binary data 0007-test-ob-shell.el-Organize-tests.patch Description: Binary data
Refactor org-babel-shell-initialize? (was Re: ob-shell intentions and paperwork (was Bash results broken?))
On Sat, 31 Dec 2022 07:56:10 -0500 Ihor Radchenko wrote --- > As for being a macro, there will be not much gain - the convention is > mostly designed for things like `cl-defun' aimed to be used in the code. > `org-babel-shell-initialize' is only used by `org-babel-shell-names'. I'm not sure what you mean by "to be used in code". Do you mean called within the global scope? > I do not have objections if it were a macro though. (But I do not see > how it would help debugging). > > > Because `org-babel-shell-initialize' is a function factory, you can't > > easily examine or modify their definitions. `C-h f org-babel-execute:sh' > > jumps to the top of lisp/ob-shell.el. Changing the definition requires > > reevaluating the definition for all the execute functions (or first > > changing `org-babel-shell-names'). > > This is indeed a downside. Any better ideas? > ob-core dictates that we must have org-babel-execute:lang functions to > make things work. My apologies, I feel there are too many separate issues I've brought up. Since I've already brought them up, let me try to be more clear about them. I observe four issues with the current form of `org-babel-shell-initialize': 1. The source is not explicit for a given `org-babel-execute:some-shell', making it difficult to debug 2. Source navigation gets confused when looking up help for a generated function. For example, `C-h f org-babel-execute:sh' goes to the top of ob-shell.el 3. It does not adhere to the coding convention 4. Except for using the customize interface, changing `org-babel-shell-names' requires reevaluating the function generator, currently `org-babel-shell-initialize' Here's my perspective on each point. 1. The source is not explicit for a given `org-babel-execute:some-shell', making it difficult to debug The benefit of using a macro is clarity of the defined functions. Here's the core `org-babel-shell-initialize' functionality as a macro. Note that it doesn't loop through `org-babel-shell-names'. (defmacro define-babel-shell-1 (shell-name) (declare (indent nil) (debug t)) (let ((function-name (intern (concat "my-org-babel-execute:" shell-name `(progn (defun ,function-name (body params) ,(format "Execute a block of %s commands with Babel." shell-name) (let ((shell-file-name ,shell-name) (org-babel-prompt-command (or (alist-get ,shell-name org-babel-shell-set-prompt-commands) (alist-get t org-babel-shell-set-prompt-commands (org-babel-execute:shell body params))) (defalias ',(intern (concat "org-babel-variable-assignments:" shell-name)) 'org-babel-variable-assignments:shell ,(format "Return list of %s statements assigning to the block's variables." shell-name)) (defvar ,(intern (concat "org-babel-default-header-args:" shell-name)) nil You can expand to see the definitions: (pp-macroexpand-expression '(define-babel-shell-1 "csh")) Is there a way to see the definition of`org-babel-execute:csh' using the current `org-babel-shell-initialize', that is, when generated by a function? Looking at the expansion, I see what appears to be an error: (alist-get "csh" org-babel-shell-set-prompt-commands) `org-babel-shell-set-prompt-commands' is an alist keyed by string shell names and whose values are shell commands to set the prompt. `alist-get' uses `eq' by default which always returns nil for string comparisons. That is, (alist-get "csh" org-babel-shell-set-prompt-commands) returns nil, not because the corresponding alist value is nil. Rather, because the (eq "csh" "csh") comparison evaluates as nil. The TESTFN probably should be `equal' or `string=': (alist-get "csh" org-babel-shell-set-prompt-commands nil nil 'equal) Then, it will return the prompt associated with "csh". All this is visible from the function version of `org-babel-shell-initialize', of course. However, it requires mentally tracking that ,name resolves to a string. Using a macro along with `pp-macroexpand-expression' makes the defined function explicit. The forms generated by the macro expansion are readily available to eval whereas the function version makes the definitions inaccessible (AFAICT). 2. Source navigation gets confused when looking up help for a generated function. For example, `C-h f org-babel-execute:sh' goes to the top of ob-shell.el Generating the execute functions with a macro also has this problem. I'm not sure there's a way around it other than defining each function with `defun'. Doing that would be a bad idea because of the massive code duplication/maintenance it would create. 3. It does not adhere to the coding convention. I'll requote the convention here for convenience. > (elisp) Coding Conventions says, > > "• Constructs that define a function or variable should be macros, not > functions, and their names should start wit
Re: ob-shell intentions and paperwork (was Bash results broken?)
On Mon, 02 Jan 2023 04:47:10 -0500 Ihor Radchenko wrote --- > They will not be reliable when tests are executed interactively. > If the `should' condition fails, `kill-buffer' will never be executed > leaving dirty state, especially for sessions. >From my perspective, that's the point and a good thing. That "dirty state" >information may be crucial to understanding why the failure happened. Can you explain why you would want a failed test to remove failure related information?
Re: ob-shell intentions and paperwork (was Bash results broken?)
On Tue, 03 Jan 2023 05:50:17 -0500 Ihor Radchenko wrote --- > I was mostly worried about session states affecting subsequent test > invocations. But I do agree that it may be better to keep them. That makes sense. I tend to run tests one at a time unless I'm about to submit patches or push... > Feel free to push upstream. I'm not able to push to git://git.sv.gnu.org/emacs/org-mode.git. I've read through the following and, as far as I can tell, I've followed the directions. - https://orgmode.org/worg/org-contribute.html - https://orgmode.org/worg/org-maintenance.html - https://www.emacswiki.org/emacs/GitForEmacsDevs
Re: ob-shell intentions and paperwork (was Bash results broken?)
On Thu, 05 Jan 2023 06:21:16 -0500 Bastien Guerry wrote --- > My bad: I did not warn Emacs maintainers in time. Now it is done, > I will let you know when they grand you access to the Emacs project. I got an email from Eli on Thursday saying I was added. I've still been getting an error when trying to push: fatal: remote error: access denied or repository not exported: /emacs/org-mode.git
Re: ob-shell intentions and paperwork (was Bash results broken?)
On Wed, 11 Jan 2023 06:53:41 -0500 Ihor Radchenko wrote --- > I have the following Git configuration: > > u remote.origin.url > yanta...@git.savannah.gnu.org:/srv/git/emacs/org-mode.git > > Are you using the same? I was not, thank you. I've since pushed.
Re: ob-shell intentions and paperwork (was Bash results broken?)
On Wed, 11 Jan 2023 12:02:42 -0500 Ihor Radchenko wrote --- > It looks like you lost all the individual commits and commit messages in > the process. > > Could you please revert 4f319088ba5 and re-push in such a way that > individual commits do appear on main? Goodness! Sorry! Thanks for catching that. I reverted the merge and then applied the patches.
Re: ob-shell intentions and paperwork (was Bash results broken?)
On Fri, 13 Jan 2023 04:36:40 -0500 Ihor Radchenko wrote --- > Cgit displays our bugfix merges with all the required commits. > So, what happened what not ideal either way. Would you like me to correct how I've incorporated my changes?
Re: [BUG] ob-shell doesn't evaluate last line on Windows (cmd/cmdproxy) [9.6.1 ( @ c:/Users/Osher/AppData/Roaming/.emacs.d/elpa/org-9.6.1/)]
Thank you for your report, Osher! Windows shells aren't currently supported by ob-shell, AFAIK. I'm open to including them. Unfortunately, I don't have a Windows machine to test against. On Mon, 16 Jan 2023 11:27:52 -0500 Osher Jacob wrote --- > Expected behaviour: > On Windows, all lines of the babel shell block should be evaluated, with > full output printed. What's does `C-h v shell-file-name' say? That should tell us what shell is being used. > #+begin_src shell > echo 1 > echo 2 > > > #+end_src > > #+RESULTS: > | Microsoft | Windows | [Version | 10.0.19044.2364] | > | | > | (c) | Microsoft | Corporation. | All | > rights | reserved. | > | | | | | > | | > | c:\Users\osherj>echo | 1 | | | > | | > | 1 | | | | > | | > | | | | | > | | > | c:\Users\osherj>More? | | | | > | | > > The hacky way I solved it was to change this line: > (t (org-babel-eval shell-file-name (org-trim body)) > to this: > (t (org-babel-eval shell-file-name (concat (org-trim body) "\n")) I don't think org-trim is the issue. Running the block (eventually) calls the default shell command, `org-babel--shell-command-on-region'. This calls `org-babel--get-shell-file-name' on a temp file containing the block source using the "-c" flag. I assume the shell used is cmdproxy.exe (https://git.savannah.gnu.org/cgit/emacs.git/tree/nt/cmdproxy.c). It looks like this converts a "-c" to a "/c", among other things. MSDN says for cmd.exe, Parameter Description /c Carries out the command specified by string and then stops. /k Carries out the command specified by string and continues. So, the way I reason it, ob-shell tries calling cmdproxy.exe using /c which basically calls cmd.exe /c, the block temp file is executed, and the shell stops. I see that `org-babel--shell-command-on-region' calls `process-file' using the `shell-command-switch'. It doesn't appear to be set anywhere else in `ob-eval' (see ob-eval:112). > Let me know if there's any other information you need, or if I can help in > any other way. I wonder if changing `shell-command-switch' to "/k" would make a difference? That is, go through the steps to reproduce and before executing the block, run `M-: (setq shell-command-switch "/k")'.
Re: [BUG] ob-shell doesn't evaluate last line on Windows (cmd/cmdproxy) [9.6.1 ( @ c:/Users/Osher/AppData/Roaming/.emacs.d/elpa/org-9.6.1/)]
On Mon, 16 Jan 2023 16:40:27 -0500 Matt wrote --- > That is, go through the steps to reproduce and before executing the block, > run `M-: (setq shell-command-switch "/k")'. Whoa, just re-read this after stepping away and it sounds super demanding! Please excuse me, I've been trying to use active voice while writing documentation. Let me try that again: What happens if you go through the steps to reproduce and run `M-: (setq shell-command-switch "/k") before executing the block?
Re: [BUG] ob-shell doesn't evaluate last line on Windows (cmd/cmdproxy) [9.6.1 ( @ c:/Users/Osher/AppData/Roaming/.emacs.d/elpa/org-9.6.1/)]
On Tue, 17 Jan 2023 14:53:39 -0500 Osher Jacob wrote ---
> changing shell-command-switch to "/k" or "-k", I get a similar output:
Thanks for checking that.
> You also mentioned the source code block is being passed through the "-c"
> flag as a command-line argument.I might be misunderstanding something here,
> but it seems like it is being passed through the stdin of the shell process
> when the calls process-file -> call-process are being made.
That the block source is passed through "-c" is an educated guess. I keep
finding myself in C land where it's less clear to me what precisely happens.
You're correct that `call-process' acts on the block source. It's not clear to
me whether that's done through stdin or whether being passed through stdin
would even help us. Here's what I'm seeing.
Evaluating the block goes through 4 lisp commands and then into C:
1. Calls `(org-babel-eval shell-file-name (org-trim body))'
This evaluates as...
(org-babel-eval
"cmdproxy.exe" ; command, whatever `shell-file-name' is
(org-trim body) ; query, the block body
)
The query parameter (the block body) gets inserted into a temp buffer on which
`org-babel--shell-command-on-region' is called.
2. Calls `(org-babel--shell-command-on-region command error-buffer)' on a temp
buffer containing block body.
This evaluates as...
(org-babel--shell-command-on-region
command ; "cmdproxy.exe"
error-buffer ; #
)
This in turn calls `process-file' where INPUT-FILE is a temp file containing
the contents of the buffer `org-babel--shell-command-on-region' was called on
(that is, the temp file contains the block body).
3. Calls `(process-file shell-file-name input-file (if error-file (list t
error-file) t) nil shell-command-switch command)'
This evaluates as...
(process-file
"cmdproxy.exe" ; shell-file-name
"/path/to/temp/containing/block/source" ; input-file
(t "/tmp/babel jTCHe/ob-error-yHOivA"); output destination (like
call-process's DESTINATION)
nil ;
display
"-c"; args
"cmdproxy.exe" ; args
)
Of course, I imagine the paths would be Windows paths when run on Windows.
According to the documentation, the args are passed to the process
(cmdproxy.exe) "verbatim without filename handling, as `call-process' does."
The `call-process' documentation says, "remaining arguments ARGS are strings
passed as command arguments to PROGRAM."
To me, that sounds like cmdproxy.exe gets "passed into" cmdproxy.exe. This
would explain the nested shell calls. If all this were written out, I imagine
it would look something like:
cmdproxy.exe -c cmdproxy.exe
What's not clear to me is how the INPUT-FILE is handled.
The `process-file' documentation says "normally". I assume it's how
`call-process' handles INPUT-FILE, but the documentation doesn't really address
it.
Finally, `process-file' finally calls `call-process'.
4. Calls `(apply 'call-process program (or lc infile) (if stderr-file (list
(car buffer) stderr-file) buffer) display args)'
This evaluates as...
(apply 'call-process
program
; cmdproxy
(or lc infile)
; local-copy of the INFILE, "/path/to/temp/containing/block/source"
(if stderr-file (list (car buffer) stderr-file) buffer) ; (t
"/tmp/babel jTCHe/ob-error-yHOivA" )
display
; nil
args)
; ("-c" "cmdproxy.exe")
How this actually works is non-trivial
(https://git.savannah.gnu.org/cgit/emacs.git/tree/src/callproc.c#n250) and not
something I understand at the moment. We can see here that indeed a call like
`cmdproxy.exe -c cmdproxy.exe' is being made. Still, I'm not sure how the
INPUT-FILE gets processed. For example, is it passed in the second cmdproxy
call? Or, maybe it gets called first and then the second call to cmdproxy
happens and hangs? I don't know.
> Any ideas on how to proceed from here?
I have two ideas.
1. Another naive work around attempt. Again, I'm going from memory,
documentation, and what I have previously written.
I have in my init a command to open a terminal when working on Windows that
looks like:
(start-process "cmd" nil "cmd.exe" "/C" "start" "cmd.exe" "/K" "cd" dir)
This starts a cmd process with /c which terminates the prompt after the command
that follows is run. The command That follows starts another cmd process with
/k which changes the directory and leaves the terminal open.
I have another command to open QGIS that does the same thing:
(start-pro
Re: [BUG] ob-shell doesn't evaluate last line on Windows (cmd/cmdproxy) [9.6.1 ( @ c:/Users/Osher/AppData/Roaming/.emacs.d/elpa/org-9.6.1/)]
On Thu, 19 Jan 2023 11:28:09 -0500 Osher Jacob wrote --- > If we're insistent on passing the input through the command line arguments, > I can think of two ways to go about this, but both seem undesirable They're good ideas and, I agree, aren't ideal. > I think it could be enough to make sure the input ends with a newline, in > which case it could be done the way I proposed in the first message, that is > changing:(t (org-babel-eval shell-file-name (org-trim body)) > to(t (org-babel-eval shell-file-name (concat (org-trim body) "\n")) Your original proposal was reasonable and looks even more so now. > I think as long as this change doesn't break the code in non-Windows > operating systems (How would we go about verifying this?), it would be > preferable due to its simplicity and minimality. Agreed, I like its simplicity. Tests are a first step and they all pass with your suggested change. > If for any other reason you would rather not change the > org-babel-sh-evaluate, and would like to implement a separate function for > Windows, that's also a viable option.I am willing to try and go down that > path, although I'm not a very experienced lisper. > Do any of these options sound like they could work well? I agree, putting in the newline is simple and it would be easy enough to create a separate path for Windows/cmd if it were a problem. I wonder if it might make sense to move where the trimming happens so that the body passed into `org-babel-sh-evaluate' is ready to go and we could handle the cmd case separately and earlier (by inserting the newline there)? I appreciate you taking the time to explore and for being open to what feels like me over-thinking it :) I'll be away from the keyboard for a few days and I look forward to getting this wrapped up for you. Meanwhile, I'm curious if Ihor or others have advice.
Re: [BUG] ob-shell doesn't evaluate last line on Windows (cmd/cmdproxy) [9.6.1 ( @ c:/Users/Osher/AppData/Roaming/.emacs.d/elpa/org-9.6.1/)]
On Fri, 20 Jan 2023 04:27:18 -0500 Ihor Radchenko wrote --- > I think `org-babel--shell-command-on-region' will be more appropriate. > Because similar issues might appear when attempting to evaluate other > code blocks on Windows, where `shell-file-name' is set to cmdproxy.exe. > > It will also be useful to leave a comment in the code explaining why we > need this newline. Works for me. How should I update bugfix to match main? Since we're considering this a bug, I figured I'd do this on bugfix. I notice, however, that bugfix doesn't have the latest updates to test-ob-shell. When I rebase main onto bugfix I get a ton of conflicts. If I instead merge main into bugfix, that shows many merge commits with bugfix. Since I don't see any "Merge with 'bugfix'" commits on bugfix currently, I assume that's not the way to do it.
Re: [BUG] ob-shell doesn't evaluate last line on Windows (cmd/cmdproxy) [9.6.1 ( @ c:/Users/Osher/AppData/Roaming/.emacs.d/elpa/org-9.6.1/)]
> On Fri, 20 Jan 2023 04:27:18 -0500 Ihor Radchenko wrote --- > I think `org-babel--shell-command-on-region' will be more appropriate. > Because similar issues might appear when attempting to evaluate other > code blocks on Windows, where `shell-file-name' is set to cmdproxy.exe. Is something like this what you're thinking? Or, do we want to check explicitly for "cmdproxy.exe"? Also, this bug isn't currently tracked in Woof!. I'm not able to confirm it, but since we're putting in a fix, should be add it anyway? ob-eval-handle-windows-shell.patch Description: Binary data
Re: [BUG] Org-Babel shell problems and crashes with images [9.6.1 (9.6.1-ga6c882 @ /home/joe/.emacs.d/straight/build/org/)]
On Wed, 25 Jan 2023 14:34:19 -0500 Josep Jesus Bigorra Algaba wrote --- > This is my first time posting to the mailing list Welcome and thanks for taking the time to send us a message! I love hearing how much you've enjoyed your experience with Org. We love it, too! > 2- Org babel tangle for shell scripts seems to exhibit strange behaviour > in latest build, > and it adds a strange BOM character at beggining of the script. This > makes it that the script is not runnable anymore. I can > fix this by manually going to exported file and doing a > =set-buffer-file-coding-system RET utf-8= . That seems to remove that > BOM character. > > My literate scripts use these options, thought sometimes shebang as > =#!/usr/bin/zsh= : > > #+begin_example > > #+title: Joe's spin on openSUSE Tumbleweed > #+property: header-args :tangle install :comments org :shebang > #!/usr/bin/bash > #+auto_tangle: t > > #+end_example I'm the ob-shell maintainer, so I'm most interested in your second point, the one related to shell things. 1. Your first point mentions that you've reproduced the bug in Emacs 28 through 30. Is the same true for this second, shell/tangle related issue? 2. Does the problem happen with both bash and zsh or just zsh? 3. Can you please give steps to reproduce the problem? I'm wondering if something like this is enough to reproduce it or if you're doing something different. (Note that I'm using #!/usr/bin/env zsh instead of #!/usr/bin/zsh because of how my system is set up.) #+property: header-args :tangle install :comments org :shebang #!/usr/bin/env zsh #+begin_src emacs-lisp :export no :tangle no (org-babel-do-load-languages 'org-babel-load-languages '((shell . t))) #+end_src #+begin_src sh readlink /proc/$$/exe #+end_src 4. Also, are you running into the issue when starting Emacs using `emacs -Q'?
Re: [BUG] ob-shell doesn't evaluate last line on Windows (cmd/cmdproxy) [9.6.1 ( @ c:/Users/Osher/AppData/Roaming/.emacs.d/elpa/org-9.6.1/)]
On Thu, 26 Jan 2023 04:51:19 -0500 Ihor Radchenko wrote --- > This will add a newline to "cmdproxy.exe" command -> "cmdproxy.exe\n". > You should instead look into `org-babel--write-temp-buffer-input-file'. I made the change in `org-babel--shell-command-on-region' based on your recommendation in https://list.orgmode.org/orgmode/87bkmttv2h.fsf@localhost/ I'd be happy to put it in `org-babel--write-temp-buffer-input-file'. Unfortunately, I don't understand why you suggest the change be made there. Wouldn't it make more sense to insert the newline into the temp buffer that `org-babel--write-temp-buffer-input-file' writes from? I'm confused because I don't see a good way write a newline from `org-babel--write-temp-buffer-input-file'. All the file writing functions[fn:1], with the exception of `with-temp-file', require a region's start and end and `with-temp-file' appears to overwrite the file. ;; 99.999% sure this isn't what you meant (defun org-babel--write-temp-buffer-input-file (input-file) "Write the contents of the current temp buffer into INPUT-FILE." (let ((start (point-min)) (end (point-max))) (goto-char start) (push-mark (point) 'nomsg) (write-region start end input-file) +(with-temp-buffer + (insert "\n") + (write-region (point-min) (point-max) input-file)) (delete-region start end) (exchange-point-and-mark))) AFAICT, `org-babel--write-temp-buffer-input-file' is only called by `org-babel--shell-command-on-region' and that's only ever called by `org-babel-eval'. That's where the query (the block source) is inserted into a temp buffer. Maybe something like this? modified lisp/ob-eval.el @@ -64,7 +64,7 @@ Writes QUERY into a temp-buffer that is processed with (let ((error-buffer (get-buffer-create " *Org-Babel Error*")) exit-code) (with-current-buffer error-buffer (erase-buffer)) (with-temp-buffer - (insert query "\n") + (insert (concat query "\n")) (setq exit-code (org-babel--shell-command-on-region command error-buffer)) [fn:1] https://www.gnu.org/software/emacs/manual/html_node/elisp/Writing-to-Files.html#Writing-to-Files
Re: [BUG] ob-shell doesn't evaluate last line on Windows (cmd/cmdproxy) [9.6.1 ( @ c:/Users/Osher/AppData/Roaming/.emacs.d/elpa/org-9.6.1/)]
Pushed the change to `org-babel-eval'.
Re: [BUG] ob-shell doesn't evaluate last line on Windows (cmd/cmdproxy) [9.6.1 ( @ c:/Users/Osher/AppData/Roaming/.emacs.d/elpa/org-9.6.1/)]
On Wed, 01 Feb 2023 07:05:45 -0500 Ihor Radchenko wrote --- > This and another commit are rather trivial - they are eligible for > bugfix. See https://orgmode.org/worg/org-maintenance.html#branches > I cherry-picked the commits for bugfix. Noted for the future. > I also added a comment clarifying the purpose of the newline. > https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=b4a1133a1 My apology for not carrying over the note. Thank you for correcting it.
[PATCH] Async evaluation in ob-shell
I'm excited to share that I've got async evaluation working (crudely) with ob-shell. A rough implementation is attached. It has clear issues, such as the prompt being present in the output: #+begin_src sh :session tester :async t echo "By sending delimiters separately..." sleep 3 slep 1 echo "typos don't cause problems--^" #+end_src #+RESULTS: : org_babel_sh_prompt> By sending delimiters separately... : org_babel_sh_prompt> org_babel_sh_prompt> sh: slep: command not found : org_babel_sh_prompt> typos don't cause problems--^ : org_babel_sh_prompt> It's not clear to me if that's something that a better regexp would handle or if it should be cleaned up in the callback function. I'm still figuring out how it's done in ob-python and ob-R. Any feedback or advice is welcome. ob-shell-async-separate-calls.patch Description: Binary data
Re: [PATCH] Async evaluation in ob-shell
I've attached two patches which replace the previous. I found cleaning the output was dramatically helped by calling `buffer-substring-no-properties' instead of `buffer-substring' in `org-babel-comint-async-filter'. I'm not sure why `buffer-substring' was originally used. `make test' shows no failures, so I assume it doesn't make a difference...? On Tue, 07 Feb 2023 06:40:51 -0500 Ihor Radchenko wrote --- > That's likely because of the same reasons why prompt did not get cleaned > in synchronous blocks earlier. See `org-babel-comint-with-output'. That, my friend, was a wild ride. I'm curious about people's feelings on `org-babel-comint-with-output'. Personally, I get the heebie-jeebies. I can't shake feeling that there's a better way, especially since `org-babel-comint-async-filter' achieves similar ends. My hunch is that other Babel languages may want async and that now would be a good time to consolidate the common functionalities of `org-babel-comint-with-output' and `org-babel-comint-async-filter' . Maybe even unify the API. So far, `org-babel-comint-with-output' touches 9 languages and `org-babel-comint-async-filter' appears to touch 2 (soon to be 3). I suspect those numbers will only grow. I also can't shake the feeling that I might become ob-comint maintainer at some point...(not yet!). I'm curious what people's thoughts are. 01-ob-shell-remove-properties-from-callback-string.patch Description: Binary data 02-ob-shell-async-non-file.patch Description: Binary data
Re: [PATCH] Async evaluation in ob-shell
On Thu, 09 Feb 2023 06:23:42 -0500 Ihor Radchenko wrote --- > Could you please elaborate? What was the problem with > `buffer-substring'? I am asking because one of the previous versions of > `org-babel-comint-wait-for-output' relied upon 'face text property. See > a35d16368. The problem is I got mixed up about the printed string representation and thought having properties changed how functions operated on the string. The 02-ob-shell-async-non-file.patch works fine with `buffer-substring'. No need for the other patch. It seems to me like 02-ob-shell-async-non-file.patch is all that's needed for basic async in ob-shell. I'm able to run long processes like `guix pull' and `guix package -u' calls without issue and the results look like I expect. Similarly for running a web server, such as `python3 -m http.server' and killing it. Unless there's something you or others think needs to be done, I can get it committed (and try to write a test or two for it).
Re: [PATCH] Async evaluation in ob-shell
On Sat, 11 Feb 2023 15:56:00 -0500wrote ---
> org-babel-comint-async-filter is capable of taking a similar approach,
> and reading/writing to tempfile.
There is some precedence in ob-shell for this. Currently, the cmdline, stdin,
and shebang headers use temp files. It may be that implementing async versions
of these could use this part of the async API. However, cmdline, stdin, and
shebang each use a temporary shell process rather than a dedicated comint, even
when the :session header is present. The async implementation requires a
comint buffer.
> I believe this approach would be
> generally more robust, but the major weakness is that it would break if
> you SSH'd to a remote host in the middle of the session.
Good point.
> There was an interesting patch to ob-shell that was never applied, that
> took the approach of wrapping the code block in a bash function before
> executing; I think it might be a promising approach:
>
> https://lists.gnu.org/archive/html/emacs-orgmode/2020-02/msg00923.html
This is interesting. Thanks for sharing! Instead of writing to a temp file,
it could use a here document. Here documents, AFAIK, are POSIX portable,
although function definitions aren't. Of course, if we're considering Windows
cmd (as we did in a recent bug report), there are all sorts of problems: batch
syntax (executed from a file) is different than CLI syntax, functions will
implemented using labels (I'm not sure how robust that is), and no here
documents. A comint would probably be best for the Windows use-case.
ahab@pequod ~$ guix shell dash
ahab@pequod ~ [env]$ dash <<- EOF
my_function() { echo "hello, world!"; echo "the end"; }
my_function;
EOF
hello, world!
the end
ahab@pequod ~$ guix shell fish
ahab@pequod ~ [env]$ fish <<- EOF
> function my_function
> echo "hello, world!"
> echo "the end"
> end
> my_function
> EOF
hello, world!
the end
> By the way, I took a look at ob-shell for the first time in awhile, and
> noticed that ob-shell now forces the prompt to be like:
>
> org_babel_sh_prompt>
>
> Which I think makes cleaning up the prompt markers a lot more
> robust. But it is a bit ugly, and also seems to break working with shell
> sessions started outside of Babel with M-x shell.
Is this something you consider a bug?
Re: [PATCH] Async evaluation in ob-shell
On Sat, 11 Feb 2023 06:44:56 -0500 Ihor Radchenko wrote ---
> 1. You should provide all the docstrings.
> 2. I generally feel that separate async and separate session code are
>either duplicating the same code or edge cases considered by session
>code may popup in async code unexpectedly.
Excellent points. Thank you.
As part of the commit, I want to include tests.
How to test an async block is non-obvious. The initial evaluation returns a
uuid which returns immediately and can be checked using a regex:
(defconst test-ob-shell/uuid-regex
"[0-9a-fA-F]\\{8\\}\\b-[0-9a-fA-F]\\{4\\}\\b-[0-9a-fA-F]\\{4\\}\\b-[0-9a-fA-F]\\{4\\}\\b-[0-9a-fA-F]\\{12\\}")
Technically, this is a ob-comint aspect and may be more rightly included in
(the currently non-existant) tests for that module.
Checking the final result from the callback is trickier. The following works,
but requires advice (which could potentially persist beyond the test) and a
delay (which slows testing overall and whose duration likely depends on the
hardware the test runs on). Without the delay, I could not get the callback to
execute within the test. It would execute when running manually in an Org
buffer, however. I'm not sure why.
(ert-deftest test-ob-shell/session-async-evaluation ()
(let ((session-name "test-ob-shell/session-async-evaluation")
(kill-buffer-query-functions nil)
result)
;; perform check after the callback executes which looks for the
;; expected result
(advice-add
'ob-shell-async-chunk-callback
:filter-return
(lambda (&rest r)
(let ((result (car r)))
(if (not (string= result "1\n2\n"))
(ert-fail (format "Expected 1\n2\n: %s" result)))
result))
`((name . ,session-name)))
;; always remove the advice, regardless of test outcome
(unwind-protect
(org-test-with-temp-text
(concat "#+begin_src sh :session " session-name " :async t
echo 1
echo 2
#+end_src")
;; execute the block; delay momentarily so that the callback
;; executes
(setq result (org-trim (org-babel-execute-src-block)))
(if (should
(and
;; if the block runs...
(string-match
test-ob-shell/uuid-regex
result)
;; ...and the callback executes without fail
(not (sleep-for 0.1
;; clean up buffer on success
(kill-buffer session-name)))
(advice-remove 'ob-shell-async-chunk-callback session-name
This works for me using the last patch.
Thoughts?
[BUG] shell sessions started outside of Babel broken
On Sat, 11 Feb 2023 15:56:00 -0500 Jack Kamm wrote --- > By the way, I took a look at ob-shell for the first time in awhile, and > noticed that ob-shell now forces the prompt to be like: > > org_babel_sh_prompt> > > Which I think makes cleaning up the prompt markers a lot more > robust. But it is a bit ugly, and also seems to break working with shell > sessions started outside of Babel with M-x shell. Can you please elaborate? The following does what I expect it to: 1. M-x shell 2. Call `org-ctrl-c-ctrl-c' on the following block in a separate Org buffer: #+begin_src sh :session *shell* echo "hello, world!" #+end_src #+RESULTS: | hello | world! | Are you doing something different?
[BUG] conda doesn't work in ob-shell sessions
On Sun, 12 Feb 2023 22:16:16 -0500 Jack Kamm wrote --- > But I also noticed another prompt-related issue: conda doesn't seem to > work in ob-shell sessions anymore. That is a bigger problem for my use > case. Could you elaborate? It looks like conda has a new init (from what I remember) which requires users to run a `conda init ' command in order to create a new environment. For example, I needed to run `conda init bash.' This added garbage to my .bashrc which automatically enabled the base environment. I was able to do this from ob-shell, however it required me to reload the shell (I reloaded Emacs to be sure I got the latest environment variables). After this, I was able to create and activate a new conda environment. 1. M-x shell 2. Executing the following in sequence: #+begin_src emacs-lisp (org-version nil t) #+end_src #+RESULTS: : Org mode version 9.6.1 (release_9.6.1-250-ge6353d @ /home/user/Projects/org-mode/lisp/) #+begin_src emacs-lisp (org-babel-do-load-languages 'org-babel-load-languages '((shell . t))) #+end_src #+begin_src sh :session *shell* :results output conda create --yes --name myenv python=3.9 #+end_src #+RESULTS: #+begin_example Downloading and Extracting Packages Preparing transaction: done Verifying transaction: done Executing transaction: done To activate this environment, use conda activate myenv To deactivate an active environment, use conda deactivate #+end_example #+begin_src sh :session *shell* :results output conda activate myenv #+end_src #+RESULTS: #+begin_src sh :session *shell* :results output which python #+end_src #+RESULTS: : /home/user/miniconda3/envs/myenv/bin/python The *shell* buffer's prompt also showed the expected [myenv] prefix.
Re: [PATCH] Async evaluation in ob-shell
On Wed, 15 Feb 2023 10:08:47 -0500 Ihor Radchenko wrote --- > Matt m...@excalamus.com> writes: > > > Checking the final result from the callback is trickier. The following > > works, but requires advice (which could potentially persist beyond the > > test) and a delay (which slows testing overall and whose duration likely > > depends on the hardware the test runs on). Without the delay, I could not > > get the callback to execute within the test. It would execute when > > running manually in an Org buffer, however. I'm not sure why. > > Doesn't (while ... (sleep-for ...)) work? I'm afraid I don't follow what you mean. My biggest concern is the sleep-for itself. Aside from the concerns above, the test doesn't work without it. For example, the check made by the advice looks for "1\n2\n". I've changed the source block from "echo 1" to "echo 2" to induce a failure (it will now receive "2\n2\n"). I re-evaluate the test and call M-: (ert "test-ob-shell/session-async-evaluation"). The test passes! I can put a message or debug in the advice and see that it's never called. However, if I uncomment the sleep-for, the advice runs, and the test fails as expected. (ert-deftest test-ob-shell/session-async-evaluation () (let ((session-name "test-ob-shell/session-async-evaluation") (kill-buffer-query-functions nil) result) ;; check callback return for correct results (advice-add 'ob-shell-async-chunk-callback :filter-return (lambda (&rest r) (let ((result (car r))) (should (string= result "1\n2\n")) ; this was previously an if statement result)) `((name . ,session-name))) ;; always remove advice, regardless of test outcome (unwind-protect (org-test-with-temp-text (concat "#+begin_src sh :session " session-name " :async t echo 2 echo 2 #+end_src") ;; execute block; callback only executes when delay (setq result (org-trim (org-babel-execute-src-block))) (if (should (and (string-match test-ob-shell/uuid-regex result) ; block runs (not (sleep-for 0.1)) ; callback doesn't fail )) (kill-buffer session-name))) (advice-remove 'ob-shell-async-chunk-callback session-name
Re: Template for ob- packages?
On Thu, 16 Feb 2023 16:10:51 -0500 Galaxy Being wrote --- > Is there a generic starter template for writing an ob- package, some sort of > example code? Yes: https://git.sr.ht/~bzg/worg/tree/master/item/org-contrib/babel/ob-template.el When I started learned Babel, I took notes with the hope of expanding it into a Worg article. Here are the notes: https://excalamus.com/2021-11-03-org_babel.html If you're interested in fleshing out the notes into proper documentation, I'd be happy to help.
Re: Template for ob- packages?
On Fri, 17 Feb 2023 17:32:18 -0500 Leo Butler wrote --- > Matt, thanks for sharing those notes. I would suggest that they be added > to worg in their current state. Thanks for your vote of confidence. I've attached the source, in case someone wants to do that. Otherwise, I'll get to it when I can. 2021-11-03-org_babel.org Description: Binary data
Re: [PATCH] Async evaluation in ob-shell
On Fri, 17 Feb 2023 05:44:20 -0500 Ihor Radchenko wrote --- > What I mean is... I follow you now. Thank you. I've attached a patch (with commit message) for adding async to ob-shell. If it looks good, I can apply it to main. 0001-ob-shell-Add-async-evaluation.patch Description: Binary data
Re: [PATCH] Async evaluation in ob-shell
On Mon, 20 Feb 2023 06:24:52 -0500 Ihor Radchenko wrote --- > Why not simply doing the `should' test when the > `test-ob-shell/uuid-regex' match fails? Instead of returning `t'. Then, > we will not need to use advise. Great point. I had originally used advice to avoid a loop. However, when it became apparent that the loop was necessary, I overlooked that the advice was no longer needed. I've rewritten the test and updated the patch. 0001-ob-shell-Add-async-evaluation.patch Description: Binary data
Re: [PATCH] Async evaluation in ob-shell
On Wed, 22 Feb 2023 05:29:59 -0500 Ihor Radchenko wrote --- > > +(defun ob-shell-async-chunk-callback (string) > > + "Filter applied to results before insertion. > > +See `org-babel-comint-async-chunk-callback'." > > + (replace-regexp-in-string (concat org-babel-sh-prompt "*") "" string)) > > Why not using `comint-prompt-regexp'? > > > +(ert-deftest test-ob-shell/session-async-valid-header-arg-values () > > + "Test that session runs asynchronously for certain :async values." > > + (let ((session-name > > "test-ob-shell/session-async-valid-header-arg-values") > > +(kill-buffer-query-functions nil)) > > +(cl-loop > > A simple `dolist' would do here. There is no reason to use `cl-loop'. Great points! Changed. 0002-ob-shell-Add-async-evaluation.patch Description: Binary data
Re: [PATCH] Async evaluation in ob-shell
On Fri, 03 Mar 2023 09:52:09 -0500 Ihor Radchenko wrote --- > I tried the patch, and I am getting failed tests: > > 1 unexpected results: >FAILED test-ob-shell/session-async-evaluation ((should (string= ": 1\n: > 2\n" (buffer-substring-no-properties (point) (point-max :form (string= > ": 1\n: 2\n" ": 1\n: 2\n: org_babel_sh_prompt>\n") :value nil :explanation > (arrays-of-different-length 8 31 ": 1\n: 2\n" ": 1\n: 2\n: > org_babel_sh_prompt>\n" first-mismatch-at 8)) Sorry for missing that. The issue is that when I replaced `org-babel-sh-prompt' with `comint-prompt-regexp', the regexp no longer matches the output and grabs the next prompt. It looks like the reason is `comint-prompt-regexp' is set to "^org_babel_sh_prompt> *" (with two spaces between the '>' and '*'). Attached is a revised patch which removes one of the spaces by changing how `org-babel-sh-initiate-session' sets the `comint-prompt-regexp'. Another place this could be done is in the defvar for `org-babel-sh-prompt' itself (which ends with a space). However, I think it's customary to leave a trailing space for prompts? > > + (let ((uuid (org-id-uuid))) > > Do you need to use `org-id-uuid' here? ob-python directly uses `md5'. > If you still prefer org-id-uuid, we probably need to move it to > org-macs.el I just need a random string. `md5' would work for that. However, might it be better to update ob-R and ob-python to use `org-id-uuid'? Both of those manually declare the randomness. It's conceivable that someone may delete or mistype the number (1), resulting in a lower entropy. An md5 is also not a uuid, strictly speaking. Of course, the chance of collision is still basically zero and the cost of collision about the same. Using `org-id-uuid' would only provide a consistent way to do things. 0003-ob-shell-Add-async-evaluation.patch Description: Binary data
Re: [PATCH] Async evaluation in ob-shell
On Sun, 05 Mar 2023 07:14:21 -0500 Ihor Radchenko wrote --- > > Matt m...@excalamus.com> writes: > > > > Sorry for missing that. The issue is that when I replaced > > `org-babel-sh-prompt' with `comint-prompt-regexp', the regexp no > > longer matches the output and grabs the next prompt. It looks like > > the reason is `comint-prompt-regexp' is set to "^org_babel_sh_prompt> > > *" (with two spaces between the '>' and '*'). Attached is a revised > > patch which removes one of the spaces by changing how > > `org-babel-sh-initiate-session' sets the `comint-prompt-regexp'. > > Another place this could be done is in the defvar for > > `org-babel-sh-prompt' itself (which ends with a space). However, I > > think it's customary to leave a trailing space for prompts? > > The actual prompt is "org_babel_sh_prompt> ". Agreed. > And we want to skip leading spaces in addition. What do you mean by this? > Adding " *" does not make the prompt match 2 spaces, but 1+. Agreed. It's not clear to me what pattern you're looking to match. > > > > + (let ((uuid (org-id-uuid))) > > > > > > Do you need to use `org-id-uuid' here? ob-python directly uses `md5'. > > > If you still prefer org-id-uuid, we probably need to move it to > > > org-macs.el > > > > I just need a random string. `md5' would work for that. However, > > might it be better to update ob-R and ob-python to use `org-id-uuid'? > > Both of those manually declare the randomness. It's conceivable that > > someone may delete or mistype the number (1), resulting in a > > lower entropy. An md5 is also not a uuid, strictly speaking. Of > > course, the chance of collision is still basically zero and the cost > > of collision about the same. Using `org-id-uuid' would only provide a > > consistent way to do things. > > `md5' will be slightly faster compared to `org-id-uuid'. But it should > not matter. > > If we want use `org-id-uuid', lets move it to org-macs.el. Requiring the > whole org-id.el must not be done. It has side effects of defining id: > links. In the next revision (once we figure out the regex), I can create a separate commit moving `org-id-uuid' to org-macs.el and updating ob-R and ob-python from `md5' to `org-id-uuid' (assuming that's not an issue for the maintainers of those). If you think speed is a concern, however, I can switch ob-shell.el to use plain `md5'. > > > (concat "^" (regexp-quote org-babel-sh-prompt) > > -" *")) > > +"*")) > > This is wrong. It unconditionally makes the last char in > `org-babel-sh-prompt' 0+. (Imagine it is changed to non-space in > future). When you say "imagine it is changed to non-space...", do you refer to `org-babel-sh-prompt'? Honestly, it's not clear to me what pattern(s) we need to match.
Re: [PATCH] Async evaluation in ob-shell
Hi Jack and Jeremie! I'm curious your thoughts about what Ihor and I are discussing at the end of this message about `md5'. On Tue, 07 Mar 2023 07:45:02 -0500 Ihor Radchenko wrote --- > Matt m...@excalamus.com> writes: > > > > The actual prompt is "org_babel_sh_prompt> ". > > > > Agreed. > > > > > And we want to skip leading spaces in addition. > > > > What do you mean by this? > > I was following the pattern described in the docstring of > `comint-prompt-regexp', where it is suggested that prompts should follow > the pattern "^ *". > > In the case of ob-shell.el, `org-babel-sh-prompt' is a string to be used > as a prompt and the corresponding regexp patter will be > "^ *". Hence > > (concat "^" (regexp-quote org-babel-sh-prompt) " *") > > > > Adding " *" does not make the prompt match 2 spaces, but 1+. > > > > Agreed. > > > > It's not clear to me what pattern you're looking to match. > > I hope the above clarified things. I'm confused because when I run the Org from HEAD, I get: (concat "^" (regexp-quote org-babel-sh-prompt) " *") -> "^org_babel_sh_prompt> *" That's *two* spaces between '>' and '*', not one. The second space comes from either 1) the definition of `org-babel-sh-prompt', which is "org_babel_sh_prompt> " (with a single space) or 2) the " *" in the (concat "^" (regexp-quote org-babel-sh-prompt) " *") expression. Currently, the two combine via the concat to give *two* spaces in the regexp. If I understand you correctly, you're trying to match the pattern given in the `comint-prompt-regexp' which is *one* space. That's what I'm trying to do, too. Way back in https://list.orgmode.org/87sfeyc7qr.fsf@localhost/, we had this exchange: On Mon, 20 Feb 2023 11:24:52 + Ihor Radchenko wrote --- > Matt writes: > > > +(defun ob-shell-async-chunk-callback (string) > > + "Filter applied to results before insertion. > > +See `org-babel-comint-async-chunk-callback'." > > + (replace-regexp-in-string (concat org-babel-sh-prompt "*") "" string)) > > Why not using `comint-prompt-regexp'? I switched out `org-babel-sh-prompt' with `comint-prompt-regexp' so that the expression looks like: +(defun ob-shell-async-chunk-callback (string) + "Filter applied to results before insertion. +See `org-babel-comint-async-chunk-callback'." + (replace-regexp-in-string comint-prompt-regexp "" string)) This causes the new test `test-ob-shell/session-async-evaluation' to fail, as you pointed out: https://list.orgmode.org/87bkl96g6e.fsf@localhost/ The test fails when we switch out the prompt in the callback because `comint-prompt-regexp' has two spaces in it. The second space causes a prompt to not be filtered (by the callback). The output becomes ": 1\n: 2\n: org_babel_sh_prompt>\n" instead of ": 1\n: 2\n" . This looks like a bug in the `comint-prompt-regexp''. It could be that `test-ob-shell/session-async-evaluation' doesn't test correctly, but it looks right to me (I could certainly be mistaken). Therefore, I see only two options to fix it: remove a space from the concat expression (which I did in my latest patch) or remove a space from `org-babel-sh-prompt'. Am I missing something? > > > `md5' will be slightly faster compared to `org-id-uuid'. But it should > > > not matter. > > > > > > If we want use `org-id-uuid', lets move it to org-macs.el. Requiring the > > > whole org-id.el must not be done. It has side effects of defining id: > > > links. > > > > In the next revision (once we figure out the regex), I can create a > > separate commit moving `org-id-uuid' to org-macs.el and updating ob-R and > > ob-python from `md5' to `org-id-uuid' (assuming that's not an issue for > > the maintainers of those). If you think speed is a concern, however, I > > can switch ob-shell.el to use plain `md5'. > > I am in favour of using `org-id-uuid'. It might also be a useful generic > function for other purposes. > > A slight concern is that some third-party code might depend on the > current pattern used for UUID string in ob-python. But we made no > promises here. > > To be a bit safer, we can also refactor `org-uuidgen-p' exposing the > regexp used to match UUID. Also, it will make sense to move > `org-uuidgen-p' to org-macs.el. I'm okay with all that. I wonder, do Jack Kamm (of ob-python fame) and Jeremie Juste (of ob-R fame) have any thoughts on the matter. I ask out of courtesy since they're the maintainers of those packages and I don't want to cross any boundaries by changing their implementations beneath them.
Re: [PATCH] Async evaluation in ob-shell
> Matt m...@excalamus.com> writes: > > I see only two options to fix it: remove a space from the concat expression > (which I did in my latest patch) or remove a space from > `org-babel-sh-prompt'. Unfortunately, I was mistaken and the second option (removing the space from `org-babel-sh-prompt') doesn't fix the issue. The TLDR is that the code in `org-babel-comint-async-filter' which grabs the region between the indicators (incorrectly) fails to include the prompt's trailing space. #+begin_longwinded_explanation I'll first explain why removing the space from `org-babel-sh-prompt' doesn't fix the issue because it well also highlight the underlying problem. If we remove the space from the `org-babel-sh-prompt', then `comint-prompt-regexp' becomes "^org_babel_sh_prompt> *" (with one space). This would work if the string passed to the `ob-shell-async-chunk-callback' stayed the same. It doesn't (this is where my reasoning and testing failed). Changing the `org-babel-sh-prompt' to "org_babel_sh_prompt>" (without a space) causes the following string to be passed to the callback: "org_babel_sh_prompt>1 org_babel_sh_prompt>2 org_babel_sh_prompt" Note that the final prompt doesn't have a ">" and therefore the `comint-prompt-regexp' (which becomes "^org_babel_sh_prompt> * (with one space)) used in the callback fails to match it. When we remove the space from the `org-babel-sh-prompt', the session buffer looks like this: "sh-5.1$ PROMPT_COMMAND=;PS1="org_babel_sh_prompt>";PS2= org_babel_sh_prompt>echo 'ob_comint_async_shell_start_39610981-1020-4baf-9dfb-f96d10af1cf8' echo 1 echo 2 echo 'ob_comint_async_shell_end_39610981-1020-4baf-9dfb-f96d10af1cf8' ob_comint_async_shell_start_39610981-1020-4baf-9dfb-f96d10af1cf8 org_babel_sh_prompt>1 org_babel_sh_prompt>2 org_babel_sh_prompt>ob_comint_async_shell_end_39610981-1020-4baf-9dfb-f96d10af1cf8 org_babel_sh_prompt>" The `org-babel-comint-async-filter' is what calls the `ob-shell-async-chunk-callback' (ob-comint.el:284). It monitors for the end indicator. When that appears, it passes the region between the beginning of the end indicator **less 1** and the character after the end of the start indicator to the callback. For a clean run of `test-ob-shell/session-async-evaluation', the beginning of the end indicator is at 361 and the character after the end of the start indicator is at 298. This is the string I gave above which is missing the ">". In order to make the second option work, we'd need to change the "less 1" part of `org-babel-comint-async-filter' from (- (match-beginning 0) 1) to (match-beginning 0). It turns out that's actually all we need to do. When `org-babel-sh-prompt' is "org_babel_sh_prompt> " (with one space), then the session buffer looks like: "sh-5.1$ PROMPT_COMMAND=;PS1="org_babel_sh_prompt> ";PS2= org_babel_sh_prompt> echo 'ob_comint_async_shell_start_3270ed43-a99b-423f-a5fa-b15fb2e4ae26' echo 1 echo 2 echo 'ob_comint_async_shell_end_3270ed43-a99b-423f-a5fa-b15fb2e4ae26' ob_comint_async_shell_start_3270ed43-a99b-423f-a5fa-b15fb2e4ae26 org_babel_sh_prompt> 1 org_babel_sh_prompt> 2 org_babel_sh_prompt> ob_comint_async_shell_end_3270ed43-a99b-423f-a5fa-b15fb2e4ae26 org_babel_sh_prompt> " The region passed to the callback is then defined as 366 to 300, or "org_babel_sh_prompt> 1 org_babel_sh_prompt> 2 org_babel_sh_prompt>" (<-- no space) This looks okay at first glance. However, **the last line is not a valid prompt**. A prompt must end in a space! When the `org-babel-sh-prompt' is set to "org_babel_sh_prompt> " (with one space), the `comint-prompt-regexp' is "^org_babel_sh_prompt> *" (with two spaces). This means that the `comint-prompt-regexp' matches on a trailing space which the **region passed to the callback doesn't have**. Therefore, the match fails. Instead, if we modify the `org-babel-comint-async-filter' like modified lisp/ob-comint.el @@ -273,7 +273,7 @@ STRING contains the output originally inserted into the comint buffer." (res-str-raw (buffer-substring ;; move point to beginning of indicator - (- (match-beginning 0) 1) + (match-beginning 0) ;; find the matching start indicator (cl-loop do (re-search-backward indicator) then the region passed to the callback will be from 367 to 300, or "org_babel_sh_prompt> 1 org_babel_sh_prompt> 2 org_babel_sh_prompt> &
Re: [SUGGESTION] ob-shell async result output should not contains shell prompt
On Wed, 22 Mar 2023 23:25:50 -0400 Christopher M. Miles wrote --- > > The ob-shell async result output contains the shell prompt. I think it > should not be captured. > > #+begin_src shell :session "test2" :async t > sleep 30 > echo "hello, world" > #+end_src > > #+RESULTS[(2023-03-23 11:19:22) 461ed5de684f6e619890709175ec73e80b67b2d6]: > : bash-5.2$ hello, world Thanks for reporting this. Try using for the babel language whatever shell the variable `shell-file-name' gives. For example, if `shell-file-name' is /bin/bash, do this: #+begin_src bash :session "test2" :async t sleep 1 echo "hello, world" #+end_src Is there a reason you're using "shell" instead of one of the shells listed in `org-babel-shell-names'?
Re: [SUGGESTION] ob-shell async result output should not contains shell prompt
On Thu, 23 Mar 2023 07:12:29 -0400 Christopher M. Miles wrote --- > #+begin_src bash :session "*ob-shell-bash*" :async t > sleep 30 > echo "hello, world" > #+end_src > > #+RESULTS[(2023-03-23 19:14:15) 23f9ad130f7a1268e21821c6baaea2b057c70d3e]: > : org_babel_sh_prompt> hello, world > > Still got a prompt. Is this intended? I think the output should be kept > clean because the result > output might be used as input for other source blocks as data. I absolutely agree. This is a bug. It should be an easy fix on my end but, in case there are details I'm overlooking, this specific example can use the following workaround: #+begin_src bash :session "*ob-shell-bash*" :async t sleep 30 && echo "hello, world" #+end_src To explain what's going on... #+begin_longwinded_explanation Shell output is filtered for prompts which should be removed before inserting the results. The issue is that the filter assumes the prompt starts at the beginning of the line. When sleep is called, it returns nothing and the next prompt appears on the same line: sh-5.1$ PROMPT_COMMAND=;PS1="org_babel_sh_prompt> ";PS2= org_babel_sh_prompt> echo 'ob_comint_async_shell_start_770d9c8f-deda-4359-aee9-a433a75a5e0d' echo "1" sleep 3 echo "2" echo 'ob_comint_async_shell_end_770d9c8f-deda-4359-aee9-a433a75a5e0d' ob_comint_async_shell_start_770d9c8f-deda-4359-aee9-a433a75a5e0d org_babel_sh_prompt> 1 org_babel_sh_prompt> org_babel_sh_prompt> 2 org_babel_sh_prompt> ob_comint_async_shell_end_770d9c8f-deda-4359-aee9-a433a75a5e0d Changing the `ob-shell-async-chunk-callback' like this will fix it: @@ -276,7 +276,7 @@ See `org-babel-comint-async-indicator'.") (defun ob-shell-async-chunk-callback (string) "Filter applied to results before insertion. See `org-babel-comint-async-chunk-callback'." - (replace-regexp-in-string comint-prompt-regexp "" string)) + (replace-regexp-in-string (concat (regexp-quote org-babel-sh-prompt) " *") "" string)) (defun org-babel-sh-evaluate (session body &optional params stdin cmdline) "Pass BODY to the Shell process in BUFFER. #+end_longwinded_explanation
Remove "shell" as a supported Babel language within ob-shell.el (was Re: [SUGGESTION] ob-shell async result output should not contains shell prompt)
> Matt m...@excalamus.com> writes: > > > Is there a reason you're using "shell" instead of one of the shells listed > > in `org-babel-shell-names'? I'm still curious why you're using "shell". I want to know if it's something you're using for a specific reason. There's no wrong answer! I ask because I have an agenda: as far as I can tell, "shell" as a Babel language is a historical accident. #+begin_longwinded_explanation Originally, ob-shell.el was called ob-sh.el. The main function was called `org-babel-execute:sh' and only /usr/bin/env sh was supported. Over time it became clear that to support other shells, the "sh" name shouldn't be used for the package or the main function. That is, 'sh' refers to a specific binary and, if other binaries such as bash, dash, csh, etc. were to be supported, it would be misleading for the Babel language to refer to a specific shell, 'sh'. So, the terminology was changed to something more general, "shell". The package was renamed to "ob-shell.el", the "namespace" updated to "shell" (for example, `org-babel-execute:shell'), and the package load call changed from (sh . t) to (shell . t). This officially happened with Org 8.2 (ORG-NEWS noted the change in commit 1a9adcb6d34bcbdff45445c827ed99dbf0b8, Tue Jan 21 09:52:31 2014). I think this gave people the (understandable) impression that "shell" was a valid Babel language, in addition to those listed in `org-babel-shell-names'. And this is where the accident happened: "shell" as a Babel language only **happens**to work. The Babel framework looks for a function prototype like "org-babel-execute:". When ob-sh.el was changed to ob-shell.el, the function `org-babel-execute:sh' became `org-babel-execute:shell'. A call like follows is perfectly legal as far as the Babel framework is concerned: #+begin_src shell echo "hello, world" #+end_src When such a block is run, Babel looks for a function called `org-babel-execute:shell'. Running the block prior to Org 8.2 should have failed because no `org-babel-execute:shell' function existed. The name change happened to source Fri Dec 13 09:52:05 2013 in commit 7a6c0e35415c4a173d101336029262f3a09abb91. After the name change, the function existed and a block using "shell" would execute! The "shell" language specifier, as far as I can tell, was never really intentionally supported. Instead, it just happened to work. It happened to work because, as far back as the first org-babel-sh.el commit, the process buffer is created using the `shell' function. I don't know the history of `shell', but presently the documentation says, Program used comes from variable ‘explicit-shell-file-name’, or (if that is nil) from the ESHELL environment variable, or (if that is nil) from ‘shell-file-name’. That is, the `shell' command falls back to `shell-file-name'. I assume that `shell' has always had that, or a similar, fallback. The `shell-file-name' is a direct path to an executable. This means that when "shell" is used for the language, `shell-file-name' is called and **any** startup script, such as .bash_profile or .bashrc, is called. The prompt could be set to **anything** and Emacs will never know, and can never know, what the prompt is without the user explicitly informing Emacs. Aside from the code change which allowed "shell" to work, "official" support of "shell" comes from Org manual commit 9d072ad67517875069f913315d762c9bb1e9c3ec, Sun Dec 17 11:06:05 2017 (for example, https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/doc/org-manual.org?id=f7aa8c19f5170dbf09538686fb569f9b60acbd6c#n18410). This appears unconnected with the code change. The addition to the manual happened 4 years after the code name change and none of the commit messages around the time of code change suggest that "shell" was intended to work as a language. In fact, I found this email from Eric Schulte (creator of Babel and maintainer at the time of the code change) which suggests that "shell" is in fact not supported or intented as a language (https://lists.gnu.org/archive/html/emacs-orgmode/2013-12/msg00294.html): In response to the statement, "a coworker used "#+BEGIN_SRC shell" where he should have written "#+BEGIN_SRC sh" Eric says, "[The suggested work around] would protect against this particular error" #+end_longwinded_explanation Regardless of whether "shell" was intended to work as a Babel language, the fact remains that it does work and that it's been advertised in the manual (at least) for 6 years. What are
Re: [PATCH] Async evaluation in ob-shell
On Thu, 23 Mar 2023 07:48:44 -0400 Ihor Radchenko wrote --- > May you also document this new feature in ORG-NEWS and in > https://orgmode.org/worg/org-contrib/babel/languages/ob-doc-shell.html ? Done.
Re: Remove "shell" as a supported Babel language within ob-shell.el (was Re: [SUGGESTION] ob-shell async result output should not contains shell prompt)
On Fri, 24 Mar 2023 07:38:58 -0400 Ihor Radchenko wrote --- > I suggest the following: > 1. Introduce a new customization `org-babel-default-shell', defaulting >to (or (executable-find "sh") (executable-find "cmd.exe")). > 2. Use the value as default shell in "shell" code blocks. > 3. Document and announce the change. > 4. Create org-lint checker that will mark "shell" code blocks as not >desired. > > The above steps will ensure minimal breakage for existing uses of > "shell" blocks. Only users who wrote shell blocks for non-standard shell > will have to adapt. These are good suggestions. Thank you!
Re: [PATCH] Async evaluation in ob-shell
On Fri, 24 Mar 2023 05:13:34 -0400 Ihor Radchenko wrote ---
> A small note on the WORG page: it may be more natural to use :async yes
> rather than :async t. Both are viable - in fact, anything other than
> :async no and :async none will be treated as "t".
Ah, okay. I'll make that more clear.
Somewhat related, I had this left over from when I was working out the async
code. It prevents the user from running :async without the (required) :session
header.The :async header runs the default (blocking) process when :session
is missing.
diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el
index 86c2bf7a7..384bfcda8 100644
--- a/lisp/ob-comint.el
+++ b/lisp/ob-comint.el
@@ -206,11 +206,12 @@ comint outputs due to buffering.")
PARAMS are the header arguments as passed to
`org-babel-execute:lang'."
(let ((async (assq :async params))
-(session (assq :session params)))
+(sessionp (not (member (cdr (assq :session params)) '("no" "none")
+(if (and async (not sessionp))
+ (error (org-babel-eval-error-notify 1 "ERROR: must use 'async' with
'session'")))
(and async
-(not org-babel-exp-reference-buffer)
- (not (equal (cdr async) "no"))
- (not (equal (cdr session) "none")
+ sessionp
+ (not org-babel-exp-reference-buffer
(defun org-babel-comint-async-filter (string)
"Captures Babel async output from comint buffer back to Org mode buffers.
It's really just a nicety. The user can cancel the accidental blocking process
with C-g. However, the block is run in a different shell than expected and
it's jarring to have Emacs freeze when you expect async.
Thoughts on including it or something similar?
error-on-async-session-mismatch.diff
Description: Binary data
Re: Font lock for org-babel shell scripts?
On Tue, 28 Mar 2023 13:24:39 -0400 Derek Chen-Becker wrote --- > I'm trying to figure out whether there's a bug or just a misconfiguration on > my end with font lock for org-babel shell source blocks. If I run emacs 28.2 > (with -q) and open the following org file, I can evaluate both source blocks > but only the "bash" block has syntax highlighting. I've confirmed that > opening a zsh script (e.g. with a zsh shebang) highlights correctly. If this > list isn't the right place to ask about this issue, does someone know where > I could look for help? This is the right place to ask. And thanks for asking! I'm able to reproduce it. I'm not familiar with how Org handles the font lock. However, I notice that calling C-c ' (`org-edit-special') results in the error: No such language mode: zsh-mode I suspect that without a language mode, Org has no way to know how to color the block. As a quick work around, you can define zsh-mode as shell-script-mode to get coloring: (defalias 'zsh-mode 'shell-script-mode)
[BUG] No font lock in src blocks for shells in org-babel-shell-names (was Re: Font lock for org-babel shell scripts?)
Of the shells given in `org-babel-shell-names' (that is, "sh" "bash" "zsh" "fish" "csh" "ash" "dash" "ksh" "mksh" "posh"), only "sh" and "bash" have font locking in source blocks. For example, #+begin_src sh if [ -z $TEST ]; then echo Fontified fi #+end_src #+begin_src bash if [ -z $TEST ]; then echo Fontified fi #+end_src #+begin_src zsh if [ -z $TEST ]; then echo No fontification fi #+end_src #+begin_src fish if [ -z $TEST ]; then echo No fontification fi #+end_src #+begin_src csh if [ -z $TEST ]; then echo No fontification fi #+end_src #+begin_src ash if [ -z $TEST ]; then echo No fontification fi #+end_src #+begin_src dash if [ -z $TEST ]; then echo No fontification fi #+end_src #+begin_src ksh if [ -z $TEST ]; then echo No fontification fi #+end_src #+begin_src mksh if [ -z $TEST ]; then echo No fontification fi #+end_src #+begin_src posh if [ -z $TEST ]; then echo No fontification fi #+end_src Does anyone know which function is responsible for re-fontifing source blocks?
Re: [BUG] No font lock in src blocks for shells in org-babel-shell-names (was Re: Font lock for org-babel shell scripts?)
On Wed, 29 Mar 2023 13:04:31 -0400 Derek Chen-Becker wrote --- > Cool, I would be happy to submit a patch! Sure, if that's something you'd enjoy. I'm happy to assist, if needed. > On Wed, Mar 29, 2023 at 3:33 AM Ihor Radchenko yanta...@posteo.net> wrote: > > Org provides some defaults for bash in `org-src-lang-modes' but not for > other shell flavours. I guess we can add more defaults to > `org-src-lang-modes', if we know for sure that e.g. zsh can be fontified > sh-mode. I think this approach will work fine. I tried examples for each shell type and keywords like if/then/else and function names are highlighted.
Re: [BUG] No font lock in src blocks for shells in org-babel-shell-names (was Re: Font lock for org-babel shell scripts?)
On Thu, 30 Mar 2023 04:55:32 -0400 Ihor Radchenko wrote --- > Matt m...@excalamus.com> writes: > > > I think this approach will work fine. I tried examples for each shell > > type and keywords like if/then/else and function names are highlighted. > > Even for posh (powershell)? Yes. It's not great since sh-mode looks for Korn-based keywords. It does string highlighting and common keywords like 'if', 'exit', and 'param'.
Re: [BUG] No font lock in src blocks for shells in org-babel-shell-names (was Re: Font lock for org-babel shell scripts?)
On Tue, 04 Apr 2023 08:30:34 -0400 Ihor Radchenko wrote --- > See the attached tentative patch. After applying the patch, I get the following error when trying to load Emacs: Warning (comp): /home/ahab/Projects/org-mode/lisp/org.el: Error: Symbol's value as variable is void sh-ancestor-alist I wasn't able to resolve it. I suspect the issue is on my end, such as a mixed install or the need to re-byte-compile `sh-script.el'. To run Org from source I do one of the following: ;; When using my init (use-package org :straight (:local-repo "/home/ahab/Projects/org-mode")) ;; When running emacs -q (add-to-list 'load-path "/home/ahab/Projects/org-mode/lisp") (require 'org-loaddefs) If I need to recompile Emacs byte code, I'm not sure how I'd do that since I'm running Guix and those files live in the write protected /gnu/store.
Re: [BUG] No font lock in src blocks for shells in org-babel-shell-names (was Re: Font lock for org-babel shell scripts?)
On Fri, 07 Apr 2023 11:29:59 -0400 Matt wrote --- > > On Tue, 04 Apr 2023 08:30:34 -0400 Ihor Radchenko wrote --- > > > See the attached tentative patch. > > After applying the patch, I get the following error when trying to load > Emacs: > > Warning (comp): /home/ahab/Projects/org-mode/lisp/org.el: Error: Symbol's > value as variable is void sh-ancestor-alist > > I wasn't able to resolve it. I suspect the issue is on my end, such as a > mixed install or the need to re-byte-compile `sh-script.el'. I was able to resolve it by simply requiring `sh-script' in `org-src'. Committed and pushed.
Re: [PATCH] Async evaluation in ob-shell
On Fri, 24 Mar 2023 05:11:38 -0400 Ihor Radchenko wrote --- > Matt m...@excalamus.com> writes: > > > On Thu, 23 Mar 2023 07:48:44 -0400 Ihor Radchenko wrote --- > > > May you also document this new feature in ORG-NEWS and in > > > https://orgmode.org/worg/org-contrib/babel/languages/ob-doc-shell.html ? > > > > Done. > > A small note on the WORG page: it may be more natural to use :async yes > rather than :async t. Both are viable - in fact, anything other than > :async no and :async none will be treated as "t". Updated WORG with commits 9d153ea4, 754c72cb, and 18333299. Updated ORG-NEWS with 6c9104f59.
Re: [PATCH] Async evaluation in ob-shell
On Mon, 17 Apr 2023 14:53:18 -0400 Ihor Radchenko wrote --- > Matt m...@excalamus.com> writes: > > > > A small note on the WORG page: it may be more natural to use :async yes > > > rather than :async t. Both are viable - in fact, anything other than > > > :async no and :async none will be treated as "t". > > > > Updated WORG with commits 9d153ea4, 754c72cb, and 18333299. Updated > > ORG-NEWS with 6c9104f59. > > Thanks, but I am not seeing 6c9104f59. I'm sorry, I was unclear about which repo these commits were in. ORG-NEWS was updated here: https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=6c9104f59ca8085abe477a81857548461bf88f23
Re: Interest in an Org video meetup?
> > On Oct 6, 2022, at 9:20 AM, Russell Adams rlad...@adamsinfoserv.com> wrote: > > Would there be any interest in a monthly 1-2 hour long ad-hoc screen > > sharing and video discussion for Org-mode? > > > > I'm offering to schedule and moderate the first few events. I'd > > propose a Saturday meeting in the afternoon European time to cover EU > > and NA. > > > > I'm considering using Jitsi, or maybe GNU Jami. > > > > Topics could include Q&A, demonstrations of features, interactive > > troubleshooting, etc. I hadn't considered presentations, but > > that could be an option too. > > > > Comments? I'm interested. I've thought of proposing something similar, yet hesitated to because topics and decisions made in a video chat wouldn't be captured on list. To me, that record is important. FSF associate members have access to a FSF hosted Jitsi instance. I'm an FSF associate member and could host it using that if I'm able to make the time.
Re: Interest in an Org video meetup?
On Wed, 26 Apr 2023 17:13:47 -0400 Corwin Brust wrote --- > On Wed, Apr 26, 2023 at 9:31 AM Matt m...@excalamus.com> wrote: > > > > FSF associate members have access to a FSF hosted Jitsi instance. I'm an > > FSF associate member and could host it using that if I'm able to make the > > time. > > > > We can make EmacsConf instance of BigBlueButton (BBB) available, as > well. FWIW I had more success with BBB than Jitsi for larger > meetings. That's generally been my experience, too. Glad to hear that's an available option. Hopefully we can bring this to fruition.
Re: ob-shell sessions will send final echo '...' as input key sequence when previous command reads input interactively (was: [Need Help] Error to evaluate "mpv" command in inline src block)
On Tue, 02 May 2023 00:17:51 -0400 Christopher M. Miles wrote --- > Indeed, this option solved the problem real neat. > I'm surprised by your knowledge and digging problem skill. > I have to say a big TANKS to you. THANKS, THANKS, THANKS. Hahaha My thanks as well to everyone who helped. I'm tied up with personal matters currently and am only seeing this thread now. On Mon, 01 May 2023 07:36:23 -0400 Ihor Radchenko wrote --- > The only way I know how to work around this is a giant one-liner like > echo ''; ; echo '...' > > However, (1) ";" may not work in some shells; (2) may > contain multiple lines, leading to the same issue. > > Matt, maybe you have some ideas about this edge case? I have no other ideas within the current ob-shell implementation. As for modifications, I have the following thoughts. First, how might we state the problem for this edge case? To me, it looks like, "How can the ob-shell :async option manage interactive input?" Do you agree with this formulation? If not, how do you see it differently? One thought is to update :async to work with the :stdin option so that the block is run as a script. Currently, :stdin runs synchronously in a separate shell. We might be able to grab the script's output and put it into the session buffer. See how the following runs in a temporary shell, regardless of the :session/:async options. #+name: answers Matt yes #+begin_src sh :stdin answers :results output :session *test* :async echo -n "What's your name?" read -r name echo echo "Hello, $name!" echo -n "Would you like to continue?" read -r continue echo if [ $continue == 'yes' ]; then echo "Continuing..." sleep 3 echo "Process complete" else echo "Aborted" fi #+end_src #+RESULTS: : What's your name? : Hello, Matt! : Would you like to continue? : Continuing... : Process complete
Re: Interest in an Org video meetup?
Thoughts on dates and times?
Reintroduction and Questions
Hello again! I'm slowly settling in after moving to Germany and hoping to get back into maintenance. As the FSF disclaimer is still being worked out with my employer, I'm hesitant to begin non-trivial code contributions. Until then, I'm hoping to do things that don't require attribution, such as organizing tasks, issues, and updating Worg. 1. What do the FR, FP, RFC, and RFE Woof! prefixes stand for? These are listed in https://tracker.orgmode.org/howto for the "request" type under the "Adding a report" section. I assume FR is "Feature Request" and RFC is "Request for Comment". 2. Is anyone in the Hamburg, Deutschland area interested in meeting in person? They say it's hard to make friends here and I'm taking applications :) I think it'd be cool to talk Emacs and work on contributions together. Maybe we could coordinate this with an online Meetup (https://lists.gnu.org/archive/html/emacs-orgmode/2023-06/msg00196.html).
Re: Reintroduction and Questions
On Sun, 25 Jun 2023 18:07:34 +0200 Ihor Radchenko wrote --- > Be warned though that Woof! is not yet stable (see > https://git.sr.ht/~bzg/woof and https://lists.sr.ht/~bzg/woof). Thanks for the warning. To verify my understanding, the mailing list is the "source of truth" for issues and Woof! "scrapes" that? > You may also try to ask on Emacs Berlin mailing list. > https://emacs-berlin.org/ Good suggestion!
Export Documentation Confusions
In the Emacs Berlin meeting today, we noticed several items that might be improved in the manual. - Section 13.18. 1: Bare HTML may be better placed under Section 13.9 HTML Export. The information in this section appears unrelated to Exporting in non-Org buffers. In fact, the settings given for a minimal HTML export do not apply when converting a region. For example, when converting a region containing an Org table to HTML, a preamble is not inserted even when the variable `org-html-preamble` is t. https://orgmode.org/manual/Bare-HTML.html Proposed solution: Move Section 13.18.1 nearer the sections on HTML export. - Section 13.18: Export in Foreign Buffers The section uses the term "foreign buffer" without providing a definition. It appears to mean a buffer without "org-mode" enabled. The section describes functions which allow users to write Org mode syntax in non-Org buffers and use various functions to convert the Org syntax to other syntaxes, like HTML, Markdown, etc. https://orgmode.org/manual/Export-in-Foreign-Buffers.html Proposed solution: Change wording from "foreign buffers" to "Convert Region" and "non-Org" buffer. The functionality is really about conversion rather than anything to do with non-Org buffers. If fact, Org respects users' freedom to run the program as they wish–it does not prevent them from converting Org formatted text within an Org buffer to a non-Org format, no matter how absurd it may appear :) - Section 13.18: Export in Foreign Buffers There was confusion caused by the function names. Specifically, the "namespace" (such as "org-ascii-" or "org-texinfo-") is confusing to users who are not familiar with Org function naming conventions. For example, one participant thought `org-ascii-convert-region-to-utf8` might convert non-utf8 ascii to utf8 and was confused why `org-texinfo-convert-region-to-texinfo` would convert texinfo to texinfo. Of course, this is not a correct understanding of the function behavior. https://orgmode.org/manual/Export-in-Foreign-Buffers.html Proposed solution: Clarify the overall purpose of the functions. Another solution could be to make an aside and clarify what is meant by "Org export back end". For example, change "The export back-ends in Org often include..." to "The export back-ends in Org (for example, Org ASCII or Org HTML) often include..." However, my hope is that the changes I made with regard to "foreign buffers" also clarifies the purpose and use of functions listed in this section. These changes are given in the attached patches. If there are no suggested improvements, I could merge them in. Please let me know your thoughts. move-bare-html-section-to-html-export-section.patch Description: Binary data change-foreign-to-non-org.patch Description: Binary data
Re: Export Documentation Confusions
On Thu, 29 Jun 2023 12:24:12 +0200 Ihor Radchenko wrote --- > > - Section 13.18: Export in Foreign Buffers > > > > The section uses the term "foreign buffer" without providing a definition. > > > > Proposed solution: Change wording from "foreign buffers" to "Convert > > Region" and "non-Org" buffer. > > "Convert region" may be still confusing. Maybe something like > "Convert Org region to other markup syntax"? How about renaming the section to "Export Region"? I was getting ahead of myself. The confusion was about "foreign buffers" and not the meaning of "export". > > - Section 13.18: Export in Foreign Buffers > > > > There was confusion caused by the function names. > > We can also create more intuitive function aliases. How about `org-export-region-to-...`? This would correspond nicely with options given by `org-export-dispatch`.
Re: Export Documentation Confusions
Changes have been pushed
Re: org babel output
On Sat, 01 Jul 2023 05:52:42 +0200 Robert Synofzik wrote ---
> Hello,
> the exact same error occurs for me:
> https://emacs.stackexchange.com/questions/72243/macos-org-babel-python-with-session-output-give-python-el-eval
>
> do you possibly have any idea on how to fix this?
Just to check, because you didn't explicitly say it in your message, does the
solution given in the StackExchange not work for you? That is, do you get a
different result when you name the session? What happens when you run the
following?
#+begin_src python :session my-named-session :results output
print("yes")
#+end_src
Re: Maintenance status of individual Org libraries
On Mon, 30 Jan 2023 14:41:18 +0100 Ihor Radchenko wrote --- > The following libraries have no maintainer or author active on the list: > ob-comint, ob-core, ... I'm interested (and technically still maintainer of ob-shell). ob-comint and ob-core are both used by ob-shell, so I've looked at good bits of both. I'd be willing to work on those. Unfortunately, my new job has yet to sign the FSF disclaimer, so I'm unable to contribute at the moment (I'm doing what I can to facilitate communication between parties). Feel free to reach out off-list with advice for how I might help that process. Until the disclaimer language gets resolved, I may be able to help in other ways. I have notes explaining the ob- API (https://lists.gnu.org/archive/html/emacs-orgmode/2023-02/msg00531.html). Maybe if that were completed, it would make being a maintainer more approachable for people? I'd also be happy to mentor new maintainers. What other ways might I help?
Re: bash source code block: problem after ssh commands
On Fri, 27 Oct 2023 13:47:24 +0200 alain.coch...@unistra.fr wrote --- > Russell Adams writes on Thu 26 Oct 2023 16:44: > > [...] are you using an SSH key to connect, or entering a password? > > If you use an SSH key for passwordless access, try adding -n (ie: > > > "ssh -n derp@host mycommand"). > > Yes, I use an SSH key, and yes, '-n' saves the day. Thanks. Hi Alain, I'm the supposed maintainer of ob-shell who's been missing in action for this whole discussion. I want to confirm, are you able to accomplish your task by using '-n'? Matt
Re: bash source code block: problem after ssh commands
On Mon, 06 Nov 2023 14:32:16 +0100 Ihor Radchenko wrote --- > I am wondering about the possible downsides of using script approach > instead of stdin redirection. I'm curious to hear more about what you're thinking.
Re: bash source code block: problem after ssh commands
On Tue, 07 Nov 2023 09:53:46 +0100 Ihor Radchenko wrote --- > Matt m...@excalamus.com> writes: > > > On Mon, 06 Nov 2023 14:32:16 +0100 Ihor Radchenko wrote --- > > > I am wondering about the possible downsides of using script approach > > > instead of stdin redirection. > > > > I'm curious to hear more about what you're thinking. > > I am thinking to change the > (t (org-babel-eval shell-file-name (org-trim body))) > clause in `org-babel-sh-evaluate' to something that uses a script file. > > It will clearly solve the discussed problem, possibly at the cost of > small overhead to write the script file to disk. Thanks for clarifying. I've been away from the codebase for a bit and, now that the FSF paperwork is signed (still need to get Craig a copy), I'm reviewing =ob-shell= with an eye for what could be cleaned or improved. I feel like =org-babel-sh-evaluate= could use some attention. I'm open to your suggestion. The good is that using a script is how :shebang and :cmdline are processed currently (like you pointed out) so there's precedence and experience with it. Also, it would make all non-session execution use the same model (script versus comint). I like how that would create a clean separation. For bad, nothing jumps out to me as obviously a problem. Let me "think out loud" for a moment. We'd need to write to disk. Like you say, this incurs overhead opening, writing, and closing the file. It's not like we'd forget to close it, though. Nor is running out of space or inodes our problem. Writing requires permission. That's not an issue with /tmp. Then, it needs to execute. Aside from permission, any code we insert needs to be correct. For example, a shebang would need to point to the correct application and any arguments would need to correspond to the implementation being called. I doubt we'd need anything beyond /bin/. FWIW, it looks like there's been at least one instance where :shebang's formatting was questioned (https://yhetil.org/orgmode/ca+a2izz1vmmkiuf4fem1au7ca1m9gqap+bkvrosz+0bxrt6...@mail.gmail.com/). We'd also need to control for what environment the script runs in. That was another issue I saw raised in the list (https://yhetil.org/orgmode/87609ug5ae@luisa.c0t0d0s0.de/). Of course, we'd need to read the stdout and stderr. This is handled by =process-file=. Any step I missed or some kind of failure I didn't consider?
Re: bash source code block: problem after ssh commands
On Thu, 09 Nov 2023 13:13:36 +0100 Ihor Radchenko wrote --- > A bigger problem is that errors *Org Babel Error > Output* are not clickable: Interesting. > So, it looks like we may need an alternative `org-babel-eval' function > that works with script files rather than input and arranges > stdout/stderr separation. Yes, it seems like a separate handler may be in order. > We should reuse `process-file' in this scenario as well, > but utilizing its ARGS argument + arrange to generate the temporary > script file in the appropriate context - see `make-nearby-temp-file'. I was unaware of `make-nearby-temp-file'. Thank you. It's helpful for this situation. I'm not in a position to address these points right now. My main focus is re-familiarizing myself with the =ob-shell= and =ob-comint= source. When I was implementing async shell evaluatation earlier this year, I remember having the impression that execution paths, especially `org-babel-comint-with-output', could probably be simplified. It seems to me that everything we're talking about in this thread aligns with that objective. I've made a note of your points and have every intention to return to them when I get to writing code. However, don't let me stop you if this is something you were interested in doing. I'll do what I can to assist.
Replace org link with footnote
Here's a quick hack to replace an Org link with a footnote. Maybe a fun first commit for someone would be to update `org-footnote-new` to accept optional LABEL and DEFINITION arguments that allow a footnote to be created programmatically? (defun my-replace-link-with-footnote () "Replace an org link with a footnote. Place the link target in the footnote and the description in the location of the original Org link." (interactive) (if (org-in-regexp org-link-bracket-re 1) (save-excursion (let ((remove (list (match-beginning 0) (match-end 0))) (link (org-match-string-no-properties 1)) (description ; link is description (if (match-end 2) (org-match-string-no-properties 2) (org-match-string-no-properties 1 (apply 'delete-region remove) ;; replaces link with footnote and jumps to footnote (call-interactively 'org-footnote-new nil) (insert " " link) (beginning-of-line) ;; follow link back to original point (org-open-at-point) (insert description)
Re: Replace org link with footnote
On Sat, 11 Nov 2023 04:54:15 +0100 Max Nikulin wrote --- > https://list.orgmode.org/ugucve$12l4$1...@ciao.gmane.io Thanks for the reference. It's always good to relate associated information :)
Re: bash source code block: problem after ssh commands
I was poking around, learning how sessions are started. Basically, `shell` creates the comint buffer using `make-comint-in-buffer`. What I find interesting is that `make-comint-in-buffer` can also create a comint buffer from a network stream: (let ((buff "*localhost-process-buffer*")) (switch-to-buffer (make-comint-in-buffer "localhost-process" buff "ssh" nil (format "%s@localhost" (getenv "USER") So, rather than start a comint and then ssh from there, it's possible to let Emacs start the subprocess, manage the ssh connection, and just read that. Emacs makes a buffer from a network stream by calling start-process` using `shell-file-name`. `org-babel-shell-initialize` closes around `shell-file-name` with whatever shell language is used. Therefore, we could provide header arguments to pass parameters and the destination to ssh. I haven't made a judgment yet about whether any of this is good or bad. I thought it was interesting and figured I'd share.
Re: bash source code block: problem after ssh commands
On Wed, 15 Nov 2023 17:32:19 +0100 Matt wrote --- > (let ((buff "*localhost-process-buffer*")) > (switch-to-buffer >(make-comint-in-buffer > "localhost-process" > buff > "ssh" > nil > (format "%s@localhost" (getenv "USER") It looks like the way this is called actually uses `start-file-process` rather than `open-network-stream`. It's not clear to me how to call it so that `open-network-stream` is used. However, that's more a question for the Emacs mailing list and, honestly, not one that I think I care about right now. Posting this just to correct what I said before.
Re: bash source code block: problem after ssh commands
On Thu, 16 Nov 2023 10:30:59 +0100 Ihor Radchenko wrote --- > Or we can make use of TRAMP as we usually do to access remote > environment. IMHO, it is more reliable as TRAMP takes care about > arranging all the Elisp FS interaction to work on remote system. Agreed. Also, I wasn't aware TRAMP worked with shell block evaluation. That's nice. Sure enough, these both work: #+begin_src bash :results output :dir /ssh:user@localhost:/home/user :session *remote* echo foo>foo_file echo "bar" #+end_src #+begin_src bash :results output :dir /ssh:user@localhost:/home/user echo foo>foo_file echo "bar" #+end_src Of course, both calls happen on the remote. I wonder, would this have helped Alain? I'm admittedly a little confused about the original report, #+name: original-report #+begin_src bash :results output ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file" echo "bar" #+end_src Clearly, the ssh line is intended to happen on the remote server. Was the echo "bar" intended to happen locally? PS: I've updated the WORG page for ob-shell with a :dir section (https://git.sr.ht/~bzg/worg/commit/0b85a4fcfe2fa5e6c1ac4edd3f664a8cc385fa54)
Re: bash source code block: problem after ssh commands
alaincochard alain.coch...@unistra.fr writes: > Yes. (So I think that explains the 'no' above.) Thank you for clarifying. > Anyway, this gives me the opportunity to come back to the question of > whether or not there is a problem with emacs itself (like some people > here thought), and if some message should be sent to some emacs list. > Again my argument was that the 2 commands copy/yank'ed in an emacs > terminal don't work as expected, while they do if the same is done in > an X terminal. Okay, I follow you now. Yes, I agree with what others have said, it's related to Emacs (probably comint-mode). It happens with M-x shell and *not* with M-x eshell. It's hard to reproduce with M-x term because term-char-mode doesn't allow copy-paste. M-x with term-line-mode doesn' t print "bar". > I guess it is not important for Org, because workarounds have be > proposed, but I think if there is a problem with emacs it should be > reported, right? I think it's worth reporting. I'm headed to bed and can submit it tomorrow for you if you're not comfortable with doing it or don't have the time. Here are the steps I followed to reproduce with M-x shell: 1. Copy the following two lines: ssh $USER@localhost "echo foo>foo_file" echo "bar" 2. Paste them into a shell (tested with xterm or xfce4-terminal) and press return 3. After entering the correct password, "bar" is echoed 4. Paste them into the buffer associated with M-x shell 5. After entering the correct password, "bar" is NOT echoed Here are the steps I followed to reproduce with M-x term: 1. M-x term 2. C-c C-j to switch to term-line-mode 3. Copy the following two lines: ssh $USER@localhost "echo foo>foo_file" echo "bar" 4. Paste them into the buffer associated with M-x term 5. After entering the correct password, "bar" is NOT echoed
Re: bash source code block: problem after ssh commands
On Fri, 17 Nov 2023 10:20:28 +0100 Ihor Radchenko wrote ---
> This has nothing to do with Emacs comint and this is also not a bug in
> Emacs
Ihor, there were two claims made in the original report. I was referring to
Claim 2. That deals with M-x shell and therefore comint-mode.
Regarding Claim 1:
- Can anyone verify Claim 1?
- Is anyone else unable to verify Claim 1 (like me)?
- What versions are people using?
+ M-x org-version
+ M-x emacs-version
I'm running Org mode version 9.7-pre (release_9.6.10-903-g9183e3.dirty @
/home/ahab/.emacs.d/straight/build/org/) on GNU Emacs 29.1 (build 1,
x86_64-pc-linux-gnu, GTK+ Version 3.24.37, cairo version 1.16.0).
* The original report has two claims:
** Claim 1.
The following block is expected to write a remote file called "foo_file" with
contents "foo" as well as give "bar" as the result.
#+begin_src bash :results output
ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file"
echo "bar"
#+end_src
The reported behavior is that "foo_file" is created with "foo" (with "foo" is
not stated, but implied) and "bar" is *not* given as the result.
** Claim 2.
Copying and pasting the two lines from the first claim into a terminal like
xfce4-terminal executes the ssh line as expected and outputs the result of the
second line. It was noted that this does not happen with M-x shell.
* Comments about the claims:
** Comment 1.
tl;dr I can't reproduce the claim that "bar" is *not* the result. The result
is "bar" for me.
The exact "expected behavior" for a shell block is a little fuzzy. According
to my analysis (given below), what Alain reports (remote file and no "bar") is
the "expected" behavior. What I see (no remote file and "bar") is actually
"unexpected".
I used the following to test the claim:
#+begin_src bash :results output
ssh localhost "echo foo>foo_file"
echo "bar"
#+end_src
I am unable to reproduce the reported behavior (of "bar" not returning).
Instead, I get an ssh-askpass permission denied error, foo_file is not created,
and "bar" is given as the result. I do not see anywhere in the thread that the
original claim was reproduced.
The thread preceded something like follows.
Leo Butler suggested two work arounds:
- add the -f to the ssh command
- add a semi-colon and line continuation to the first line.
Russell Adams suggested another work around:
- add -n to the ssh command
Ihor identified that a non-session call does something like the following
command:
bash -c bash /tmp/temp-file-with-source-block-code.sh
where
- /tmp/temp-file-with-source-block-code -
ssh localhost "echo foo>foo_file"
echo "bar" | tee /tmp/bar.txt
-
The second line (significantly different from the original report) pipes the
echo result to stdout and to a file, bar.txt. Writing to a file allows us to
confirm if that line was executed.
Ihor suggested that
bash -c bash /tmp/temp-file-with-source-block-code.sh
does not run the second line because an interactive password prompt is
displayed by ssh. The reasoning is that the prompt hangs the process while
waiting for input and the second line never runs. Indeed, running the command
does not produce /tmp/bar.txt.
Ihor is correct about prompts messing with shell blocks (this is not the first
time he's seen this). However, the way it's stated does not demonstrate it.
This is because Emacs does *not* make a call like
bash -c bash /tmp/temp-file-with-source-block-code.sh
Alain responded by pointing out that
bash -c bash /tmp/temp-file-with-source-block-code.sh
does not execute the first line. This is true. Consider calling
bash -c bash /tmp/two-lines.sh
where
-- /tmp/two-lines.sh --
echo "first" > /tmp/first.txt
echo "second" > /tmm/second.txt
---
Neither first.txt or second.txt are created.
Max Nikulin interjected with a helpful reminder that Bash scripting is a
snakepit of footguns. (What Max said is more than that and interesting. I
skip it here because it depends on the form of the call.)
Before trying to untangle what a given Bash command does, we need to be sure
what command is actually called. Unfortunately, there's not a clear Bash
command corresponding to how Emacs makes the call.
What happens goes something like this:
1. The Lisp function process-file is called with PROGRAM "bash", INFILE a path
to a temp file containing the source block code, and ARGS ("-c" "bash")
2. This information is passed to DEFUN ("call-process"), a Lisp object
implemented in C
3. DEFUN ("call-process") forwards this information to the C function
call_process
4. call_process calls emacs_spawn
5. emacs_spawn creates a subprocess
A lot of cleaning and setup happens which is dependent on the system (GNU,
Window, Darwin, etc.). There's a call to openp which looks for the executable.
An emacs_pipe is set up (
Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)
On Sat, 18 Nov 2023 04:11:03 +0100 Max Nikulin wrote --- > > bash -c bash /tmp/two-lines.sh > > From my point of view it was a plain mistake in attempts to simulate > the issue outside of Emacs. There is no point to concentrate on this > command. I tried to explain that it is incorrect usage of "-c" shell > option and what is the actual effect of this call, but I seems I failed. As an ob-shell user, my expectation is that execution within Org produces the same behavior as outside of Emacs. This is why I've focused on the command. It acts as a guide for what is "correct." Maybe this is misguided for a reason I don't yet see? I concluded that the command is not "what Emacs does" and therefore isn't a valid source of "truth". I agree it's not worth concentrating on it further.
Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)
On Sat, 18 Nov 2023 09:29:56 +0100 Bruno Barbier
> IIUC, what Max is saying is that you should not concentrate on
> *that specific command* because that command doesn't do what you think
> it does.
Cool, it sounds like we're agreed (albeit for different reasons).
> To reproduce, I'm personally still using:
>
> cat /tmp/test.sh | bash
>
> which is, IIUC, what:
>
> (process-file "bash" "/tmp/test.sh")
Yes, agreed. I think that's more like what's happening.
What about the ("-c" "bash") passed into process-file?
The whole call looks like this:
(process-file
"bash"
"/tmp/two-lines.sh"
'(t "/tmp/babel-mS0Yyg/ob-error-AoxNqH")
nil
"-c" "bash")
Re: bash source code block: problem after ssh commands
To clarify a previous typo I made. Everything I've written was also done using: 1. emacs -q 2. C-x b "*scratch*" 3. M-x org-mode 4. Execute #+begin_src emacs-lisp :results none (org-babel-do-load-languages 'org-babel-load-languages '((shell . t))) #+end_src This corresponds to Org mode version 9.6.6 (release_9.6.6 @ /gnu/store/xjrhyapm3zwgpmq5baz6m9kavz287jjj-emacs-29.1/share/emacs/29.1/lisp/org/) I had previously, and incorrectly, given my version used with my full init (which uses a different Org version). On Sat, 18 Nov 2023 09:19:13 +0100 Bruno Barbier > It seems your SSH failed to connect. In that case, I cannot swallow the > second command; thus the command "echo bar" is executed. > > I can reproduce what you see on my side if I force the connection to fail: > > #+begin_src bash :results output > ssh WRONG_REMOTE "echo foo>foo_file" > echo "bar" > #+end_src > > #+RESULTS: > : bar Thank you! That makes sense. I forgot that without set -e Bash keeps evaluating when errors happen. Yes, once it connects, I get the same behavior as reported. Here is how I connected without a password prompt: 1. ssh-keygen with id_babel, , 2. cat ~/.ssh/id_babel.pub >> ~/.ssh/authorized_keys 3. chmod og-wx ~/.ssh/authorized_keys 4. ssh-add .ssh/id_babel
Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)
On Sat, 18 Nov 2023 09:54:46 +0100 Bruno Barbier > But, you're right. To be safe, from now on, I'll use: > > cat /tmp/test.sh | bash -c bash It's still not clear to me if this is "what Emacs does". However, that's the best I could come up with. Evaluating the following #+name: /tmp/test.sh #+begin_src bash :results output ssh localhost "echo foo>foo_file" echo "bar" | tee /tmp/bar.txt #+end_src does exactly what cat /tmp/test.sh | bash -c bash does. Both create a file "foo_file" containing "foo" on the remote machine and neither execute the second line. So, I would say that what happens in Org is the "expected" behavior.
Re: bash source code block: problem after ssh commands
On Fri, 17 Nov 2023 23:07:57 +0100 Matt wrote --- > The second claim has nothing to do with Org Babel. I was able to confirm it > and provide the steps to reproduce. I think it would make sense to report > it upstream and let them decide if it's expected behavior. I'm still happy > to do that, but I need to step away from the keyboard :) Submitted. https://lists.gnu.org/archive/html/bug-gnu-emacs/2023-11/msg00976.html
Re: [BUG] ob-shell: :shebang changes interpretation of :cmdline
On Sat, 18 Nov 2023 16:54:39 +0100 Max Nikulin wrote --- > I have faced an inconsistency with :cmdline treatment in ob-shell.el. I > expect same results in the following cases: > > #+begin_src bash :cmdline 1 2 3 >printf "%s\n" "$1" > #+end_src > > #+RESULTS: > : 1 > > #+begin_src bash :cmdline 1 2 3 :shebang #!/bin/bash >printf "%s\n" "$1" > #+end_src > > #+RESULTS: > : 1 2 3 Thank you! This makes a good test case. Ihor or Bastion, Craig and my employer reached agreement on the disclaimer language. I've sent a signed copy to Craig and haven't heard back yet with the go-ahead to continue contributing. I sent the paperwork only a week or so ago, so I'm sure he'll get back to me in time. Because of this, I'm including patches rather than pushing (which assumes I still have access (I haven't checked)). test-cmdline-alone-and-with-shebang-have-same-result.patch Description: Binary data
[BUG] ob-shell: :cmdline fails with single argument (was Re: [BUG] ob-shell: :shebang changes interpretation of :cmdline)
On Sat, 18 Nov 2023 16:54:39 +0100 Max Nikulin wrote --- > I have faced an inconsistency with :cmdline treatment in ob-shell.el. These are sadly easy to find. If you run: #+begin_src bash :cmdline 1 echo "$1" #+end_src Then it fails with list: Wrong type argument: sequencep, 1 However, running this works: #+begin_src bash :cmdline "1" echo "$1" #+end_src I didn't dig too much into it, but it looks like :cmdline expects a sequence. When multiple arguments are passed, such as :cmdline 1 2 3, then "1 2 3" is passed into process-file. (sequencep "1 2 3") is t. (sequencep 1) is nil. So, to work around this a single :cmdline argument must be surrounded by quotes to make it a sequence. (sequencep "1") is t. Obviously, this should be fixed. Attached is a patch to test for this whenever we're ready to tackle making execution mutually consistent. I'm still reviewing the library and am not quite ready for that yet. test-cmdline-with-single-argument-shouldnt-require-quotes.patch Description: Binary data
