Hi Joel
Okay. Here are my two cents:
*"I'd like to have evaluatable, and collapsable, Python fields in LyX kind
of like code blocks in Mathematica notebooks."*
Are you using Mathematica? If so, I am not sure my plugin would be of any
help. However, if you want a latex document that has render persistent
python variables, I strongly suggest you to try the knitr module combined
with reticulate package in RStudio(they could be used in LyX, too). With
these two, you can write code cells and render the variables in latex using
\sexpr command. For example, here is a psuedo code(actually grammar is
slightly different but of the same flavor) example
--------------------------------------------------------
\begin{knitr-python-code-cell}
a = 1
\end{knitr-python-code-cell}
The variable a has value \sexpr{a}.
--------------------------------------------------------
The \sexpr{a} will be 1 in the final compiled pdf doc.
The draw back of this approach is that in order to see what "a" is, we have
to compile the whole latex document, which could be a 20-30 second effort
if the document is big. So it is not very good for code intensive docs.
*"Bottom line: I'd prefer not to have two separate documents to manage but
rather have the two integrated into one."*
This is indeed something cool. But it would be quite an effort to implement
properly. Jupyter is basically a REPL(read-eval-print-loop) with "read"
being implemented in clients of different forms, and eval-print implemented
in the jupyter kernel. The communication of the two is done by ZeroMQ, a
asynchronous messaging queue. This means that if we want to 100% port
jupyter's capabilities in LyX main work buffer, to start with we need to
bundle LyX with the ZMQ's cpp binding and probably a python
interpreter(since Jupyter seems to not have a cpp binding) as well. Then we
need to create a collapsible inset that behaves like a code cell in jupyter
notebook. I am not sure the current LyX main work area supports this
feature. I tried to read its implementation, but could not figure out what
it is actually doing. Even if we can do it, we still need to connect it to
Jupyter kernel in the backend using zmq's c++ bindings.
It could be done, for sure. But it will probably take more than one
person's effort.
*Now, I would like to express my own vision on this*. I have two problems
that I would like to solve. Let's call the first one LyX Major/Jupyter
Minor Mode. In order to understand the problem better, consider the
following scenario: I would like to write a 600-1000 page machine learning
or algorithm book that contains a lot of python/R code and figures. Also,
this books is math heavy, too. Traditionally, to handle these code pieces,
I would need to either edit them in editors such as vscode/emacs and run
them and either 1. save them as local files and insert them to the LyX
documents or 2. copy and paste them into LyX ERT boxes. Yes, it could be
done for sure. But to manage these code pieces well takes some serious
effort.
But if I choose to use the webengine bases Jupyter notebook inside LyX. I
can start a notebook for each chapter. And when editing my main LyX book
document, I can use the Jupyter Notebook pane to develop my code. Note that
I want this pane also have access to the lyx main buffer, so that when I
want to insert a certain code cell, I can do so by right click on that
cell, and select "Insert to LyX as XX", where XX could be the
representation in LyX of our choosing for the code in this code cell. It
could be a proglisting, or minted, or knitr+reticulate chuck. I can do
something similar to a plot in the jupyter as well.
The second problem, without any surprise, is Jupyter Major/LyX Minor Mode:
the math editing experience for Jupyter Notebook is not very good. So we
can edit some of the math-heavy cells. An example of this is in the screen
shot here:
https://github.com/jasonsun0310/JupyLyx/blob/master/Screen%20Shot%202019-07-08%20at%2012.07.17%20AM.jpg.
The conversion function from LyX doc to Code cell in browser could be
implemented using either c++ or python. I don't think it will be too hard a
task.
On Sun, Jul 7, 2019 at 11:23 PM Joel Kulesza <jkule...@gmail.com> wrote:
> Jason,
>
> No, I've not tried those. However, I feel like we have solutions looking
> for a problem. What I'm asking is: what problem(s) are you trying to solve?
>
> I'm unsure what I'm looking for in terms of an ideal mixture of LyX and
> Jupyter, because I'm generally happy with the way things work now.
>
> If anything, I'd like to have evaluatable, and collapsable, Python fields
> in LyX kind of like code blocks in Mathematica notebooks. However, I'm
> still unsure how I would incorporate something like that into an actual LyX
> document other than to have generated matplotlib figures. I also suspect
> other developers would have concerns about arbitrary code execution
> following this approach. Bottom line: I'd prefer not to have two separate
> documents to manage but rather have the two integrated into one.
>
> Thanks,
> Joel
>
> P.S. I switched to top-posting to try to maintain consistency with you so
> the thread doesn't become too chaotic.
>
> On Sun, Jul 7, 2019 at 9:13 PM Jason Sun <ds...@cornell.edu> wrote:
>
>> Have you tried knitr and reticulate? I can build an isomorphism between
>> the embedded browser and LyX. Maybe we could do this in another way. Could
>> you tell me what do you expect to see in an ideal mixture of Jupyter and
>> LyX?
>>
>> Sent from my iPhone
>>
>> On Jul 7, 2019, at 10:46 PM, Joel Kulesza <jkule...@gmail.com> wrote:
>>
>> On Sat, Jul 6, 2019 at 2:03 PM Jason Sun <ds...@cornell.edu> wrote:
>>
>>> The proper way to establish communication is to use ZeroMQ to interact
>>> with IPython directly from LyX. I read the jupyter docs, it is something
>>> we could try to do. I can communicate with the LyX main buffer now by
>>> letting the main Browser to as ccess LyX.h, and thus have access to call
>>> dispatch, and do the injection of variables. It does seem a bit naive
>>> compared to the zeroMQ approach. But I would argue that it might be
>>> sufficient for light uses. OTOH, if webengine is used, we should utilize
>>> its full potential and design the communication framework properly.
>>>
>>> To answer some of other question: yes, this widget is dockable in either
>>> 4 positions. It also could be dragged out as a single page view.
>>>
>>> I need to try see how hard it is to embed the zeroMQ communication
>>> layer. It can be done for sure, the question is if it is worth the effort
>>> or do we have some better alternatives.
>>>
>>> Sent from my iPhone
>>>
>>> On Jul 6, 2019, at 9:12 AM, Joel Kulesza <jkule...@gmail.com> wrote:
>>>
>>> On Tue, Jul 2, 2019 at 11:03 AM Jason Sun <ds...@cornell.edu> wrote:
>>>
>>>> I have built an example prototype widget that tries to combine LyX and
>>>> Jupyter Notebook together.
>>>>
>>>> Is it implemented via QtWebEngine. Basically, by embedding a mini
>>>> webpage browser inside LyX.
>>>>
>>>> A screenshot is here:
>>>> https://github.com/jupyter/notebook/issues/1604#issuecomment-507746234
>>>>
>>>> A bidirectional communication between LyX and Jupyter Notebook could be
>>>> implemented via this browser. With the help of jupyter lab, we can get
>>>> cross platform terminal for free.
>>>>
>>>> Would this be something worth further pursuing down the road?
>>>>
>>>> Using Web, a lot could be implemented. Especially, Qt 5.13 introduces
>>>> Web Assembly. With the help of that, LyX could have it OverLeaf counter
>>>> part with stronger abilities!
>>>>
>>>
>>> Jason,
>>>
>>> Thanks for your thoughts and work on this. Sorry I've been quiet to
>>> this point, but I hoped to ask a few questions as I think about how I'd use
>>> your work.
>>>
>>> I'm a rather avid LyX user, and routinely use Jupyter notebooks.
>>> However, I'm still not entirely clear how these two will be integrated. I
>>> see the screenshot you provided on GitHub; however, it still seems like the
>>> LyX document and notebook are quite separate. It's not clear to me what is
>>> gained by putting them both within one window. Can you please help me
>>> understand how you foresee a user interacting with these two?
>>>
>>> Some more pointed questions based on the interface you provided in the
>>> screenshot:
>>>
>>> 1. I assume the browser component can be moved to be bottom half
>>> rather than right half. Is that correct? I usually run LyX on half my
>>> monitor, so I'd be reluctant to end up only running it in 1/4 (and
>>> Jupyter
>>> in the other 1/4).
>>> 2. Do you have a way to integrate Jupyter I/O with the LyX
>>> document? For example, as a Python variable gets updated in an
>>> evaluation
>>> it will then get captured the next time the LyX-produced PDF is
>>> compiled/viewed.
>>> 3. Will your .ipynb be included in the File -> Export -> LyX
>>> Archive?
>>>
>>> Thanks again,
>>> Joel
>>>
>>>
>> Jason,
>>
>> Thanks for your thoughts. However, my one main question still seems
>> unanswered: how you foresee a user interacting with these two applications
>> (LyX and Jupyter)?
>>
>> What is the workflow that you're imagining versus how it currently would
>> be: with LyX and a web browser open separately? What advantage is gained
>> through the coupling of the two applications? For example, are you working
>> toward a LyX document with Jupyter-driven evaluated Python cells? Or,
>> perhaps a Jupyter notebook that will generate a LyX document? Perhaps
>> something else?
>>
>> Thanks,
>> Joel
>>
>>
--
*Daqian Sun*
*Tel:607-379-5149*
*Department of Mathematics *
*Department of Economics*
*Cornell University*
