Will do!
Thanks for all your help!
-Thomas
From: Gregg Powell
Sent: Thursday, December 12, 2024 12:30 PM
To: Deramus, Thomas Patrick
Cc: r-help@r-project.org
Subject: Re: [R] Cores hang when calling mcapply
Hi Thomas,
Glad to hear the suggestion helped
or.
>
> - So a new call may be needed to keep this approach from failing if we
> update our version of `data.table` in the future
>
>
> - The initial approach was replacing all the `KeyID` variables with what
> was basically row numbers and that would h
at would have made merging back to the main key
document an issue so I changed the rbind funciton to keep this from happening.
Thank you for all your help on this!
-Thomas DeRamus
____________
From: Gregg Powell
Sent: Wednesday, December 11, 2024 2:11 PM
To: Deramus, Thomas Patrick
Cc:
pecs of our server:
> Thread(s) per core: 2
> Core(s) per socket: 20
> Socket(s): 4
> Stepping: 4
> CPU(s) scaling MHz: 50%
> CPU max MHz: 3700.
> CPU min MHz: 1000.
>
>
>
>
>
pecs of our server:
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 4
Stepping:4
CPU(s) scaling MHz: 50%
CPU max MHz: 3700.
CPU min MHz: 1000.
____________
From: Gregg Powell
Sent: Wednesday,
imit reached?)
> Error: cons memory exhausted (limit reached?)
> Error: cons memory exhausted (limit reached?)
>
>
>
> From: Gregg Powell
> Sent: Tuesday, December 10, 2024 7:52 PM
> To: Deramus, Thomas Patrick
> Cc: r-help@r-project.org
> Subject: Re: [R]
it reached?)
Error: cons memory exhausted (limit reached?)
Error: cons memory exhausted (limit reached?)
From: Gregg Powell
Sent: Tuesday, December 10, 2024 7:52 PM
To: Deramus, Thomas Patrick
Cc: r-help@r-project.org
Subject: Re: [R] Cores hang when calling mcap
Hello Thomas,
Consider that the primary bottleneck may be tied to memory usage and the
complexity of pivoting extremely large datasets into wide formats with tens of
thousands of unique values per column. Extremely large expansions of columns
inherently stress both memory and CPU, and splitting
Maybe ask on the HPC list? [1]
A general tip... you may be running out of memory. If at all possible you need
to make sure you extract the data subsets in the parent process, and limit the
amount of environment data passed into the child processes. That is, instead of
using an integer counter t
Hi R users.
Apologies for the lack of concrete examples because the dataset is large, and
it being so I believe is the issue.
I multiple, very large datasets for which I need to generate 0/1
absence/presence columns
Some include over 200M rows, with two columns that need presence/absence
colum
10 matches
Mail list logo
