Forum: CFEngine Help
Subject: Append the content of file to another one without checking for
duplicate lines
Author: jbdenis
Link to topic: https://cfengine.com/forum/read.php?3,26695,26695#msg-26695
Hello,
I tried to achieve something trivial, but I don't get anywhere.
Imagine you have a big file (1M for example), and you simply want to append its
content to the end of another file without checking for duplicate. If I'm using
insert_lines with a "file" insert_type, an "insert_lines" is generated for each
line in the big file hence a big execution time (insert_lines needs to search
for duplicate).
Yes I know it will result in an ever growing file, but at each run I remove
then insert lines (with the help of location).
It takes around 60 seconds to append a 2MB file to another one. ncharles on IRC
reports similar execution time. Each insert_lines statement looks for a
previously same line in the growing destination file : the executation time is
exponential I guess.
Here is a sample test, just generate a multiline "big" file in /tmp/bigdata.
This test will delete and create a file called /tmp/bigfile and then append the
content of /tmp/bigdata.
body common control
{
bundlesequence => { "biginsert" };
inputs => { "cfengine_stdlib.cf" };
}
body agent control
{
editfilesize => "1M";
}
bundle agent biginsert
{
files:
# I remove the file for testing purpose
# to prevent every growing file
"/tmp/bigfile"
delete => tidy,
classes => if_ok("file_deleted");
file_deleted::
"/tmp/bigfile"
edit_line => big_edit("/tmp/bigdata"),
create => "true";
}
bundle edit_line big_edit(content_from_file)
{
insert_lines:
"$(content_from_file)"
insert_type => "file";
}
To make a long story short, how would you simply append the content of a file
to another one without checking for duplicate ?
_______________________________________________
Help-cfengine mailing list
Help-cfengine@cfengine.org
https://cfengine.org/mailman/listinfo/help-cfengine
