Re: Threadpools, difference between DMD and LDC

2014-08-05 Thread Russel Winder via Digitalmars-d-learn
On Mon, 2014-08-04 at 18:34 +, Dicebot via Digitalmars-d-learn wrote: […] > Well it is a territory not completely alien to me either ;) I am > less aware of academia research on topic though, just happen to > work in industry where it matters. I have been out of academia now for 14 years, bu

Re: Threadpools, difference between DMD and LDC

2014-08-04 Thread Philippe Sigaud via Digitalmars-d-learn
> https://github.com/D-Programming-Language/phobos/pull/1910 Very interesting discussion, thanks. I'm impressed by the amount of work you guys do on github.

Re: Threadpools, difference between DMD and LDC

2014-08-04 Thread Sean Kelly via Digitalmars-d-learn
On Monday, 4 August 2014 at 21:19:14 UTC, Philippe Sigaud via Digitalmars-d-learn wrote: Has anyone used (the fiber/taks of) vibe.d for something other than powering websites? https://github.com/D-Programming-Language/phobos/pull/1910

Re: Threadpools, difference between DMD and LDC

2014-08-04 Thread Dicebot via Digitalmars-d-learn
On Monday, 4 August 2014 at 21:19:14 UTC, Philippe Sigaud via Digitalmars-d-learn wrote: Has anyone used (the fiber/taks of) vibe.d for something other than powering websites? Atila has implemented MQRR broker with it : https://github.com/atilaneves/mqtt It it still networking application tho

Re: Threadpools, difference between DMD and LDC

2014-08-04 Thread Philippe Sigaud via Digitalmars-d-learn
On Mon, Aug 4, 2014 at 6:38 PM, Russel Winder via Digitalmars-d-learn wrote: > Are these std.concurrent threads or std.parallelism tasks? > > A std.parallelism task is not a thread. Like Erlang or Java Fork/Join > framework, the program specifies units of work and then there is a > thread pool un

Re: Threadpools, difference between DMD and LDC

2014-08-04 Thread Philippe Sigaud via Digitalmars-d-learn
On Mon, Aug 4, 2014 at 6:21 PM, Dicebot via Digitalmars-d-learn wrote: > vibe.d additions may help here: > > http://vibed.org/api/vibe.core.core/runTask > http://vibed.org/api/vibe.core.core/runWorkerTask > http://vibed.org/api/vibe.core.core/workerThreadCount > > "task" abstraction allows exactl

Re: Threadpools, difference between DMD and LDC

2014-08-04 Thread Dicebot via Digitalmars-d-learn
On Monday, 4 August 2014 at 18:22:47 UTC, Russel Winder via Digitalmars-d-learn wrote: Actually with CSP / actor model one can simply consider long-running CPU computation as form of I/O an apply same asynchronous design techniques. For example, have separate dedicated thread running the comput

Re: Threadpools, difference between DMD and LDC

2014-08-04 Thread Russel Winder via Digitalmars-d-learn
On Mon, 2014-08-04 at 16:57 +, Dicebot via Digitalmars-d-learn wrote: […] > This is why I had "or close" remark :) Exact number almost always > depends on exact deployment layout - i.e. what other processes > are running in the system, how hardware interrupts are handled > and so on. It is s

Re: Threadpools, difference between DMD and LDC

2014-08-04 Thread Dicebot via Digitalmars-d-learn
On Monday, 4 August 2014 at 16:38:24 UTC, Russel Winder via Digitalmars-d-learn wrote: Modern default approach is to have amount of "worker" threads identical or close to amount of CPU cores and handle internal scheduling manually via fibers or some similar solution. I have no current data, bu

Re: Threadpools, difference between DMD and LDC

2014-08-04 Thread Russel Winder via Digitalmars-d-learn
Sorry, I missed this thread (!) till now. On Mon, 2014-08-04 at 13:36 +, Dicebot via Digitalmars-d-learn wrote: > On Monday, 4 August 2014 at 05:14:22 UTC, Philippe Sigaud via > Digitalmars-d-learn wrote: > > I have another question: it seems I can spawn hundreds of > > threads > > (Heck, ev

Re: Threadpools, difference between DMD and LDC

2014-08-04 Thread Dicebot via Digitalmars-d-learn
On Monday, 4 August 2014 at 14:56:36 UTC, Philippe Sigaud via Digitalmars-d-learn wrote: On Mon, Aug 4, 2014 at 3:36 PM, Dicebot via Digitalmars-d-learn wrote: Most likely those threads either do nothing or are short living so you don't get actually 10 000 threads running simultaneously. In ge

Re: Threadpools, difference between DMD and LDC

2014-08-04 Thread via Digitalmars-d-learn
On Monday, 4 August 2014 at 14:56:36 UTC, Philippe Sigaud via Digitalmars-d-learn wrote: On Mon, Aug 4, 2014 at 3:36 PM, Dicebot via Digitalmars-d-learn wrote: Modern default approach is to have amount of "worker" threads identical or close to amount of CPU cores and handle internal scheduling

Re: Threadpools, difference between DMD and LDC

2014-08-04 Thread Philippe Sigaud via Digitalmars-d-learn
On Mon, Aug 4, 2014 at 3:36 PM, Dicebot via Digitalmars-d-learn wrote: > Most likely those threads either do nothing or are short living so you don't > get actually 10 000 threads running simultaneously. In general you should > expect your operating system to start stalling at few thousands of >

Re: Threadpools, difference between DMD and LDC

2014-08-04 Thread Dicebot via Digitalmars-d-learn
On Monday, 4 August 2014 at 12:05:31 UTC, Philippe Sigaud via Digitalmars-d-learn wrote: IIRC, there are fibers somewhere in core, I'll have a look. I also heard the vibe.d has them. http://dlang.org/phobos/core_thread.html#.Fiber vibe.d adds some own abstraction on top, for example "Task" c

Re: Threadpools, difference between DMD and LDC

2014-08-04 Thread Dicebot via Digitalmars-d-learn
On Monday, 4 August 2014 at 05:14:22 UTC, Philippe Sigaud via Digitalmars-d-learn wrote: I have another question: it seems I can spawn hundreds of threads (Heck, even 10_000 is accepted), even when I have 4-8 cores. Is there: is there a limit to the number of threads? I tried a threadpool becau

Re: Threadpools, difference between DMD and LDC

2014-08-04 Thread Philippe Sigaud via Digitalmars-d-learn
On Mon, Aug 4, 2014 at 2:13 PM, Chris Cain via Digitalmars-d-learn wrote: >> OK, I get it. Just to be sure, there is no ThreadPool in Phobos or in >> core, right? > There is. It's called taskPool, though: > > http://dlang.org/phobos/std_parallelism.html#.taskPool Ah, std.parallelism. I stoopidl

Re: Threadpools, difference between DMD and LDC

2014-08-04 Thread Chris Cain via Digitalmars-d-learn
On Monday, 4 August 2014 at 12:05:31 UTC, Philippe Sigaud via Digitalmars-d-learn wrote: OK, I get it. Just to be sure, there is no ThreadPool in Phobos or in core, right? IIRC, there are fibers somewhere in core, I'll have a look. I also heard the vibe.d has them. There is. It's called task

Re: Threadpools, difference between DMD and LDC

2014-08-04 Thread Philippe Sigaud via Digitalmars-d-learn
> Without going into much detail: Threads are heavy, and creating a thread is > an expensive operation (which is partially why virtually every standard > library includes a ThreadPool). > I haven't looked into detail your code, but consider using the TaskPool if > you just want to schedule some ta

Re: Threadpools, difference between DMD and LDC

2014-08-04 Thread David Nadlinger via Digitalmars-d-learn
On Monday, 4 August 2014 at 05:14:22 UTC, Philippe Sigaud via Digitalmars-d-learn wrote: This is correct – the LLVM optimizer indeed gets rid of the loop completely. OK,that's clever. But I get this even when put a writeln("some msg") inside the task. I thought a write couldn't be optimized aw

Re: Threadpools, difference between DMD and LDC

2014-08-03 Thread Kapps via Digitalmars-d-learn
On Monday, 4 August 2014 at 05:14:22 UTC, Philippe Sigaud via Digitalmars-d-learn wrote: I have another question: it seems I can spawn hundreds of threads (Heck, even 10_000 is accepted), even when I have 4-8 cores. Is there: is there a limit to the number of threads? I tried a threadpool bec

Re: Threadpools, difference between DMD and LDC

2014-08-03 Thread Philippe Sigaud via Digitalmars-d-learn
> This is correct – the LLVM optimizer indeed gets rid of the loop completely. OK,that's clever. But I get this even when put a writeln("some msg") inside the task. I thought a write couldn't be optimized away that way and that it's a slow operation? Anyway, I discovered Thread.wait() in core in

Re: Threadpools, difference between DMD and LDC

2014-08-03 Thread David Nadlinger via Digitalmars-d-learn
On Sunday, 3 August 2014 at 22:24:22 UTC, safety0ff wrote: On Sunday, 3 August 2014 at 19:52:42 UTC, Philippe Sigaud wrote: Can someone confirm the results and tell me what I'm doing wrong? LDC is likely optimizing the summation: int sum = 0; foreach(i; 0..task.goal) sum +=

Re: Threadpools, difference between DMD and LDC

2014-08-03 Thread safety0ff via Digitalmars-d-learn
On Sunday, 3 August 2014 at 19:52:42 UTC, Philippe Sigaud wrote: Can someone confirm the results and tell me what I'm doing wrong? LDC is likely optimizing the summation: int sum = 0; foreach(i; 0..task.goal) sum += i; To something like: int sum = cast(int)(cast(ulong)(

Threadpools, difference between DMD and LDC

2014-08-03 Thread Philippe Sigaud via Digitalmars-d-learn
I'm trying to grok message passing. That's my very first foray into this, so I'm probably making every mistake in the book :-) I wrote a small threadpool test, it's there: http://dpaste.dzfl.pl/3d3a65a00425 I'm playing with the number of threads and the number of tasks, and getting a feel abo

Threadpools, difference between DMD and LDC

2014-08-03 Thread Philippe Sigaud via Digitalmars-d-learn
I'm trying to grok message passing. That's my very first foray into this, so I'm probably making every mistake in the book :-) I wrote a small threadpool test, it's there: http://dpaste.dzfl.pl/3d3a65a00425 I'm playing with the number of threads and the number of tasks, and getting a feel about