Pavel gave some helpful direction regarding where concurrency might be especially helpful: graphics conversion.
Attached is a work-in-progress patch, as well as an example file. When testing with the attached document, remember to clear the cache in your user directory before opening LyX. For example, I do the following: rm -rf ~/.lyx/cache/* && lyx The modified code is just about LyX's display, not about converters used for PDF export. While testing, it works surprisingly smoothly and the improvement is quite noticeable. I'm surprised that there are no known *observable* problems I've encountered, although I still consider the patch hackish at its current state. The concurrency design of the patch is not good. It was just the most minimal changes I could make to the current code to get a simple prototype. I have not worked on thread safety. To do that I need to study more about the signals mechanism that's being used. I also need to test problems like what happens if a process exits with error, what if it hangs, etc. The reason I send this preliminary patch in is to (1) see if there is at least preliminary support for this kind of code change; (2) see if anyone has intuition for whether I am on the right track (i.e., is this the right place in the code to introduce this functionality?); (3) any thoughts on how I can stress-test this patch, and future patches?; (4) what should the default be for number of concurrent processes forked and what interface could I provide to change the default? e.g., default to current master behavior and allow for a hidden preference? Regarding (3), how can I force a bunch of conversions that take a long time? Maybe a svg that is very small (or scalled down?) but is a large file size? Currently these conversions are all quick, so quick that the bars on my 'htop' don't even dance. Thanks, Scott
>From 64278aecb9647a04e8591bba1d01df5f18bd2afd Mon Sep 17 00:00:00 2001
From: Scott Kostyshak <skost...@lyx.org>
Date: Fri, 23 Sep 2022 00:28:52 -0400
Subject: [PATCH] Working on concurrent forks
---
src/support/ForkedCalls.cpp | 45 ++++++++++++++++++++++++-------------
1 file changed, 30 insertions(+), 15 deletions(-)
diff --git a/src/support/ForkedCalls.cpp b/src/support/ForkedCalls.cpp
index b9462153e0..ec23db026b 100644
--- a/src/support/ForkedCalls.cpp
+++ b/src/support/ForkedCalls.cpp
@@ -20,6 +20,7 @@
#include "support/lyxlib.h"
#include "support/os.h"
#include "support/Timeout.h"
+#include "support/lassert.h"
#include "support/bind.h"
@@ -443,6 +444,13 @@ static queue<Process> callQueue_;
/// flag whether queue is running
static bool running_ = false;
+// change nforks_max = 1 to current master behavior
+static const int nforks_max = 20;
+
+// number of current running threads.
+static int nforks_current = 0;
+
+
///
void startCaller();
///
@@ -450,6 +458,7 @@ void stopCaller();
///
void callback(pid_t, int);
+// FIXME: adapt this description
/** Add a process to the queue. Processes are forked sequentially
* only one is running at a time.
* Connect to the returned signal and you'll be informed when
@@ -468,26 +477,32 @@ ForkedCall::sigPtr add(string const & process)
void callNext()
{
- if (callQueue_.empty())
- return;
- Process pro = callQueue_.front();
- callQueue_.pop();
- // Bind our chain caller
- pro.second->connect(callback);
- ForkedCall call;
- //If we fail to fork the process, then emit the signal
- //to tell the outside world that it failed.
- if (call.startScript(pro.first, pro.second) > 0)
- pro.second->operator()(0,1);
+ while(!callQueue_.empty() && nforks_current <= nforks_max) {
+ lyxerr << "callQueue size: " << callQueue_.size() << std::endl;
+ Process pro = callQueue_.front();
+ callQueue_.pop();
+ // Bind our chain caller
+ pro.second->connect(callback);
+ ForkedCall call;
+ //If we fail to fork the process, then emit the signal
+ //to tell the outside world that it failed.
+ ++nforks_current;
+ if (call.startScript(pro.first, pro.second) > 0)
+ pro.second->operator()(0,1);
+ }
+ stopCaller();
}
void callback(pid_t, int)
{
- if (callQueue_.empty())
- stopCaller();
- else
- callNext();
+ // FIXME: this code is not threadsafe. Can this function be called
+ // concurrently? I would imagine yes.
+ // Does this have to do with the signals and nod thing?
+ // it seems there is a concurrency policy there (?).
+ --nforks_current;
+ LASSERT(nforks_current >= 0, return );
+ callNext();
}
--
2.34.1
graphics-conversions.lyx
Description: application/lyx
-- lyx-devel mailing list lyx-devel@lists.lyx.org http://lists.lyx.org/mailman/listinfo/lyx-devel
