Lager Question: how to get the pid:module:function format
Hello,
I have been using Lager for several months now to much success. One
thing that I have not figured out though is how to get our output to
match that of what the limited documentation claims. According to
the blog post:
http://basho.com/blog/technical/2011/07/20/Introducing-Lager-A-New-Logging-Framework-for-ErlangOTP/
It states:
Now, those were
messages coming out of error_logger, which is fine for legacy or
library code, but Lager also has its own logging API that you can
use. It's actually implemented via a parse_transform so that Lager
can capture the current module, function, line number and pid for
inclusion in the log message. All this is done automatically, and
the logging call in the code looks like this:
lager:error("oh no!")
lager:warning("~s, ~s and ~s, oh my!", [lions, tigers, bears])
Which will be
displayed like:
2011-07-19 18:02:02 [error] <0.31.0>@test2:start:8 oh no!
2011-07-19 18:02:02 [warning] <0.31.0>@test2:start:9 lions, tigers and bears, oh my!
Note that you can
easily see where the error occurred just by glancing at the line.
Also notice that you don't need to stick a newline on the end of
the log message. Lager automatically (and happily) does that for
you.
Unfortunately, I have never been able to get the
@module:function:line format. Here is my output:
00:04:45.504 [info] Riak socket link started <0.88.0>
00:04:45.513 [info] Starting <0.90.0>
00:04:45.514 [info] Riak socket link started <0.91.0>
00:04:45.520 [info] Starting <0.93.0>
00:04:45.520 [info] Riak socket link started <0.94.0>
00:04:45.527 [info] Pid1 = {ok,<0.63.0>}
00:04:45.530 [debug] next_worker: <0.64.0>
00:04:45.532 [debug] {new, file,
<<"data/2012-03-31/test.data">>, }
What am I doing wrong?
Thanks,
Bryan
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak Duplicates, storage_backend
The leveldb back end is not as fast as the default (bitcask), but leveldb does not keep all keys in memory. So for a very large or unbounded set of keys leveldb is superior. Leveldb stores values sorted by key, which also lets riak speed up certain operations such as listing keys in a bucket. Kresten On 30/03/2012, at 18.17, "idmartin" wrote: > Is there anyway to analyze Riak objectes to make sure there isnt duplicates > or near duplicates? > > also, > > I want to switch to riak_kv_eleveldb_backend for secondary indexes, was > wondering if there is any disadvantages to this backend? > > Thx > > > -- > View this message in context: > http://riak-users.197444.n3.nabble.com/Riak-Duplicates-storage-backend-tp3871293p3871293.html > Sent from the Riak Users mailing list archive at Nabble.com. > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Lager Question: how to get the pid:module:function format
Bryan,
You'll need to enable Lager's parse transform. The easiest way to do that is to
add the {parse_transform, lager_transform} to your compile options in
rebar.config.
Sean Cribbs
On Mar 31, 2012, at 3:13 AM, Bryan Hughes wrote:
> Hello,
>
> I have been using Lager for several months now to much success. One thing
> that I have not figured out though is how to get our output to match that of
> what the limited documentation claims. According to the blog post:
>
> http://basho.com/blog/technical/2011/07/20/Introducing-Lager-A-New-Logging-Framework-for-ErlangOTP/
>
> It states:
>
> Now, those were messages coming out of error_logger, which is fine for legacy
> or library code, but Lager also has its own logging API that you can
> use. It's actually implemented via a parse_transform so that Lager can
> capture the current module, function, line number and pid for inclusion in
> the log message. All this is done automatically, and the logging call in the
> code looks like this:
>
> lager:error("oh no!")
> lager:warning("~s, ~s and ~s, oh my!"
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Lager Question: how to get the pid:module:function format
Actually, by default, the console doesn't show the 'verbose'
information. You should be able to see the extra information by looking
at the logfile, or you can enable it by changing the console config to
look like:
{lager_console_backend, [info, true]}
Hope that helps,
Andrew
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak Recap for March 28 - 30
Happy Saturday. New code, slides, and more for today's Recap. Enjoy. Mark twitter.com/pharkmillups --- Riak Recap for March 28 - 31 = 1) Joseph Blomstedt's Erlang Factory Slides are online. His talk was called, "Test First Construction of Distributed Systems" and is a great look at some of the approaches he and others at Basho take to writing and testing the code that make up Riak's internals. * Slides here ---> http://speakerdeck.com/u/jtuple/p/test-first-construction-of-distributed-systems 2) More from Erlang Factory: Jesse Newland gave an exceptional talk about how he and a few others are rewriting GitHub Pages using Riak and Webmachine. * Slides here ---> http://speakerdeck.com/u/jnewland/p/github-pages-on-riak-and-webmachine 3) Some new code: Alice Kaerast just released a proof of concept of a Riak plugin for Moodle 2. * Code here on GitHub ---> https://github.com/kaerast/repository_riak 4) New Stack Overflow Content: * Issues building a three node Riak Cluster from Homewbrew install ---> http://stackoverflow.com/questions/9906386/running-a-three-node-riak-cluster-using-a-homebrew-installation * Is there any way to use JS libraries in Riak? ---> http://stackoverflow.com/questions/9937416/is-there-anyway-to-use-js-libraries-in-riak 5) Colin Austin has picked up work on txriak, a library that the AuricSystems crew released last year. * Code here ---> https://bitbucket.org/calston/txriak # Issues 1) New * preflist_exhausted message when running out of VMs ---> https://github.com/basho/riak_pipe/issues/44 ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
slow mapred_search key lookups for single terms
I'm seeing very slow performance from Riak search even when querying
single terms, and I'd appreciate any advice on how to get insight into
where the time is going.
Right now, I'm using this function to time queries with the Erlang pb client:
TS =
fun (Pid, Bucket, Query) ->
T0 = now(),
{ok, Results} = riakc_pb_socket:search(Pid, Bucket, Query),
MuSec = timer:now_diff(now(), T0),
io:format(user, "~b results, ~f sec~n", [length(Results), MuSec/100])
end.
The bucket I'm querying currently has ~300k keys total (each 16
bytes). (The whole cluster has maybe 1.5M entries in about a dozen
buckets. It's running 1.0.2, 4 nodes on 4 8-core c1.xlarge EC2
instances.)
For single-term queries that return 10k+ keys, I'm routinely seeing
times above 6 seconds to run the above function. Intermittently,
however, I'll see the same queries take only 2 seconds:
> TS(Pid,Bucket,<<"full_text:flower">>).
12574 results, 6.094149 sec
ok
> TS(Pid,Bucket,<<"full_text:flower">>).
12574 results, 1.938894 sec
ok
> TS(Pid,Bucket,<<"full_text:flower">>).
12574 results, 1.981492 sec
ok
> TS(Pid,Bucket,<<"full_text:flower">>).
12574 results, 6.120589 sec
ok
> TS(Pid,Bucket,<<"full_text:red">>).
13461 results, 6.572473 sec
ok
> TS(Pid,Bucket,<<"full_text:red">>).
13461 results, 6.626049 sec
ok
> TS(Pid,Bucket,<<"full_text:red">>).
13461 results, 2.155847 sec
ok
Queries with fewer results are still slow, but not quite as slow as 6 seconds:
> TS(Pid,Bucket,<<"full_text:ring">>).
6446 results, 2.831806 sec
ok
> TS(Pid,Bucket,<<"full_text:ring">>).
6446 results, 3.037162 sec
ok
> TS(Pid,Bucket,<<"full_text:ring">>).
6447 results, 0.780944 sec
ok
And queries with no matches only take a few milliseconds:
> TS(Pid,Bucket,<<"full_text:blorf">>).
0 results, 0.003269 sec
ok
During the slow queries, none of the 4 machines seems to be fully
taxing even one cpu, or doing almost any disk i/o.
As far as I can tell from looking at the riak_kv/riak_search source,
my query should only be hitting the index and streaming back the keys,
not trying to read every document from disk or sort by score. Is that
correct?
Assuming that's the case, I can't imagine why it takes so long to look
up 10k keys from the index for a single term, or why the times seem to
be bimodal (which seems like a big clue). Any pointers welcome!
Thanks,
Mike
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
