https://bugs.llvm.org/show_bug.cgi?id=44399
Bug ID: 44399
Summary: Memory Consumption Reduction for Large Array
Initialization?
Product: clang
Version: trunk
Hardware: PC
OS: Linux
Status: NEW
Severity: enhancement
Priority: P
Component: Frontend
Assignee: unassignedclangb...@nondot.org
Reporter: phdoftheho...@gmail.com
CC: llvm-bugs@lists.llvm.org, neeil...@live.com,
richard-l...@metafoo.co.uk
I have a large number of files that I want to turn into data initializers for
(static/global/const) arrays and then compile into my program. I generally rely
on these arrays to be optimizer-transparent. Occasionally, they go up to 50~60
Megabytes.
The files are often dumped by xxd. A C++ header file representing 40 MB using
XXD goes to about a 250 MB or so file.
When I want to do work (constexpr processing in some cases) on some of these
arrays of (unsigned) char, I spend a lot of time just compiling and memory
usage shoots up. When I spent a lot of time tracking down the parts of what I
end up paying for, for just using a single entry out of an e.g. 20 MB array I
would use ~2.1 GB of memory. For example, for a 20 MB file named "image.bin", I
would use
xxd -i image.bin image.bin.h" and then attempt to return a single item from
that array through main() to just test the cost of parsing and holding onto
that data in the compiler:
#include"<image_bin.h"
int main () {
return image_bin[2];
}
This program took ~2.1 GB of memory and around 39~40 seconds of compilation
time, using 3 CPUs to nearly max. When I actually do real C++ with it, it gets
much worse.
I am not interested in using the linker to create an object file and directly
link it in, because that inhibits my ability to do constexpr/template
metaprogramming and create more compact, interesting data structures from
industry-standard file formats.
I would like to know if there are places in the Clang codebase I could
contribute to, that would allow me to improve the throughput of such commonly
deployed constructs. While I personally am the author of proposals heading to
the C and C++ groups to make working with files less painful, maybe there is
something I or others can do without new proposed functionality to help speed
this kind of code up and reduce the memory footprint?
--
You are receiving this mail because:
You are on the CC list for the bug._______________________________________________
llvm-bugs mailing list
llvm-bugs@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-bugs
