Re: Yokozuna indexes too slow
Hi Ilyas,
So, from those stats it's looking like it's taking around 3 seconds on average
to write a message into Solr (search_index_latency_mean, in microseconds),
though the max value is almost 27 seconds. So there is definitely something
wrong with the way Solr is behaving, or possibly with the way you are indexing
you data, though from your description everything looks fine -- 200 bytes of
JSON, which I assume contains fields like username, device id, ip address etc,
looking at your schema. You should definitely be able to get better
performance out of Solr, though those stats are being taken in Yokozuna, so I
wouldn't rule out what might be going on in Riak, at least not without knowing
more.
One thing you might start looking at are the JVM stats via JConsole, for
example, to see if there is anything suspicious going on with the JVM. You
should at least be able to get things like stats from the garbage collector,
heap size, etc.
I would also recommend using tools on your debian machine like top, vmstat,
iotsat, to get a picture of how much time is being spent in Solr vs Riak. It
would be interesting to see what the CPU and I/O behavior of Riak and Java are,
in this case.
While you don't necessarily need collectd to tract stats, I have some
collectd/python scripts for gathering data about Riak and the JVM. Please feel
free to pilfer/use at your discretion (The proc man page on most lines is
helpful). All they do is scrape the /proc file system to get stats about CPU
time, io, etc. There is also some collectd config for collecting stats via JMX
from the JVM, in case you are interested in that.
https://github.com/fadushin/riak_puppet_stuff/tree/master/modules/riak_node/files/collectd
Yokozuna uses ibrowse to connect (via HTTP) to the Solr, and there is a way to
set the browse connection pool to something larger than 10. You might be able
to get better throughput that way, but I would first try to sort out why the
latency is so bad.
To change the size of the ibrowse connection pool, attach to Riak, and set the
ibrowse default_max_sessions environment variable to something greater than 10,
e.g., 100.
prompt$ riak attach
Remote Shell: Use "Ctrl-C a" to quit. q() or init:stop() will terminate the
riak node.
Erlang R16B02_basho8 (erts-5.10.3) [source] [64-bit] [async-threads:10]
[kernel-poll:false] [frame-pointer]
Eshell V5.10.3 (abort with ^G)
(riak@192.168.1.202)1> rpc:multicall(application, get_env, [ibrowse,
default_max_sessions]).
{[undefined,undefined,undefined,undefined,undefined],[]}
(riak@192.168.1.202)2> rpc:multicall(application, set_env, [ibrowse,
default_max_sessions, 100]).
{[ok,ok,ok,ok,ok],[]}
(riak@192.168.1.202)3> rpc:multicall(application, get_env, [ibrowse,
default_max_sessions]).
{[{ok,100},{ok,100},{ok,100},{ok,100},{ok,100}],[]}
One other thought -- does any of your JSON contain internationalized data, and
if so, how is it encoded, e.g., UTF-8 or UTF-16, ISO-8859, etc? Your etop
listing didn't suggest anything out of sorts with extractors, but we might want
to get a handle on what is going on there, as well.
Since your quoted issue 320, are the Java errors in your logs associated with
broken pipes on the Solr server? That would suggest that we might be getting
timeouts on the client (yokozuna) side, and connections are getting closed
before the server can write a response, but I think the default timeout is 60
seconds, so you shouldn't be hitting that (looking at your stats), though those
stats are taken from a relatively small time window.
I hope that helps you diagnose where the bottleneck is. Keep us posted.
-Fred
> On Oct 2, 2015, at 2:23 AM, ilyas wrote:
>
>
> It looks like this issues
>
> https://github.com/basho/yokozuna/issues/320
>
>
> I try to set
> maxThreads to 150
> Acceptors to 10
> lowResourcesMaxIdleTime to 5
> in /usr/lib/riak/lib/yokozuna/priv/solr/etc/jetty.xml as recommended in
> https://github.com/basho/yokozuna/issues/330
>
>
> but it has no effect
>
> On 10/01/2015 11:53 PM, Fred Dushin wrote:
>> Is there any more information in these logs that you can share? For
>> example, is this the only entry with this exception? Or are there more?
>> Are there any associated stack traces? An EOF exception can come from many
>> different scenarios.
>>
>> Is there anything in the Riak console.log that looks suspicious?
>>
>> Finally, you might want to take a look at what is going on inside of riak
>> when you get into this state (slow writes to Solr), by looking at Riak stats.
>>
>> You get get to Riak stats via curl, e.g.,
>>
>> curl http://localhost:8098/stats | python
>> -m json.tool
> ok, output is attached
>
>> Stats you might want to pay special attention to:
>>
>> riak_kv_vnodeq (min, max, median, etc) -- the aggregate length of the vnode
>> qu
Re: Yokozuna indexes too slow
Hi Ilya,
Along with all of Fred's excellent recommendations, make sure that
Riak and the JVM aren't fighting for RAM. The default is for LevelDB
to take up to 70% of system RAM, leaving the remaining 30% for the
rest of the erlang VM as well as the JVM and operating system.
Monitoring system RAM usage by process should show if this is happening.
Also, if you would like to run "riak-debug" on a node and share the
generated tar file (via dropbox, for instance), it may uncover system
tuning that can be done.
--
Luke Bakken
Engineer
lbak...@basho.com
On Wed, Sep 30, 2015 at 11:18 PM, ilyas wrote:
>
> # create bucket type for leleldb backend
> sudo /usr/sbin/riak-admin bucket-type create leveldb
> '{"props":{"backend":"leveldb_mult"}}'
> sudo /usr/sbin/riak-admin bucket-type activate leveldb
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Performance with riak can't reach expected
Hi Yang Zhang, To Ruan's point above, there are a number of variables to keep in mind when measuring performance. If you have the option to use a recommended testing framework, please read about basho_bench [1]. If you have to test using whatever you currently use for consistency across systems, please outline what you're using to test and what requirements you would like to achieve. For more on tuning Riak to balance consistency and availability, this article ons system performance tuning is a great place to look [2]. [1] http://docs.basho.com/riak/latest/ops/building/benchmarking/ [2] http://docs.basho.com/riak/latest/ops/tuning/linux/ Matt Brender | Developer Advocacy Lead Basho Technologies t: @mjbrender On Wed, Sep 30, 2015 at 12:46 PM, Ruan Jonker wrote: > > Hi, > > 1) which client are you using? > 2) avg obj/key size & net latency between nodes. > 3) are put operations done sequentially or in parallel? > 4) what is the round-trip time of one put > 5) over which unit of time are you measuring your 1 put operations. > 6) bucket/type properties - nval pw w etc. > > R > > On 30 Sep 2015 2:29 pm, "张扬" wrote: >> >> I benchmark riak's performance using basho_bench in the last days. But the >> performance is unsatisfying. >> >> I have a 2 node cluster, each node running on the same vm(8 CPU, 32 GB RAM). >> Riak's storage_backend is memory. The total ops could reach 1 with the >> put command. Can I do something to make the performance better ? >> >> -- >> Best Regards, >> >> Yang Zhang >> >> >> ___ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Using Riak on systemd based Ubuntu
Hey Matthew, It seems you ran into some documentation gaps that we should address. Please open up one or more of them on the docs repo when you can [1] Could you restate what specifically someone can help you do? [1] https://github.com/basho/basho_docs/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+riak+cs Thanks! Matt Brender | Developer Advocacy Lead Basho Technologies t: @mjbrender On Wed, Sep 30, 2015 at 7:23 PM, Matthew Hall wrote: > Hello List, > > Today I tried to get the following items to work in Ubuntu 15.04 Vivid with > systemd based SysV compatibility to run the init scripts: > > Riak 2.1.1 > Stanchion 2.0.0 > Riak CS 2.0.1 > Riak CS Control 1.0.2 > > My app.config files are here for reference: > > https://gist.github.com/megahall/676ca0410af4aacc4f74 > > When I attempted this, both Stanchion and Riak CS fail to stay running for the > 15 second expected time in the init scripts. > > So I read through the "stanchion console" and "riak-cs console" output. > > There were some backtraces there complaining about EACCES coming via mochiweb. > > Upon modifying /usr/sbin/stanchion and /usr/sbin/riak-cs to add an strace -f > to track the system calls, I determined that there is an EACCES error raised > by attempting to bind some sub-process socket onto Port 80. It is hard to see > what sub-thread or sub-process exactly was the cause. > > This is very confusing because none of my configuration mentions Port 80 at > all. > > The configuration works OK in Ubuntu 14.10 without systemd. So now I am > wondering if anybody has figured out how to get it to work right inside of > systemd with some .service file examples and so on. It's really important for > the project I am trying to get done. > > Thanks, > Matthew. > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak CS - Unable to create/view bucket details using dragon disk
Hey Johan,
I have executed s3cmd with --debug option.
Please find the output.
Test access with supplied credentials? [Y/n] Y
Please wait, attempting to list all buckets...
DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Fri, 02 Oct 2015 16:32:44
+\n/'
DEBUG: CreateRequest: resource[uri]=/
DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Fri, 02 Oct 2015 16:32:44
+\n/'
DEBUG: Processing request, please wait...
DEBUG: get_hostname(None): 172.31.44.38
DEBUG: format_uri(): http://172.31.44.38/?delimiter=/
DEBUG: Sending request method_string='GET',
uri='http://172.31.44.38/?delimiter=/', headers={'content-length': '0',
'Authorization': 'AWS access-key:OeIcaJSH58eWgIlLiyQBgzN16Pc=',
'x-amz-date': 'Fri, 02 Oct 2015 16:32:44 +'}, body=(0 bytes)
DEBUG: Response: {'status': 403, 'headers': {'date': 'Fri, 02 Oct 2015
16:32:44 GMT', 'content-length': '154', 'content-type': 'application/xml',
'server': 'Riak CS'}, 'reason': 'Forbidden', 'data': 'AccessDeniedAccess
Denied'}
DEBUG: S3Error: 403 (Forbidden)
DEBUG: HttpHeader: date: Fri, 02 Oct 2015 16:32:44 GMT
DEBUG: HttpHeader: content-length: 154
DEBUG: HttpHeader: content-type: application/xml
DEBUG: HttpHeader: server: Riak CS
DEBUG: ErrorXML: Code: 'AccessDenied'
DEBUG: ErrorXML: Message: 'Access Denied'
DEBUG: ErrorXML: Resource: None
DEBUG: ErrorXML: RequestId: None
ERROR: Test failed: 403 (AccessDenied): Access Denied
Do you have any idea about this error?
--
View this message in context:
http://riak-users.197444.n3.nabble.com/Riak-CS-Unable-to-create-view-bucket-details-using-dragon-disk-tp4033494p4033524.html
Sent from the Riak Users mailing list archive at Nabble.com.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak Recap - October 2, 2015
Riak Recap time. The user lists have been quieter than usual, so we will use this round to share some other news as well. Send me feedback if you ever have articles to share or recommendations for topics! ## Latest Releases * The .NET client upgraded to 2.2.0 [0]. Thanks to Luke and community contributors! * Basho Data Platform 1.0.0 is out there [1] and we love to see your code using it - share with us [2] * We also continue to update documentation for our open source Spark Connector, that leverages an updated 2i query for parallel extract from BDP [3] ## RICON! * RICON speakers were announced and we have some incredible ones. As a community member, you can take the $100 off your ticket with the Basho100 code. This discount code will expire on October 31. Click here to use it: http://bit.ly/ricon15u ## Community Update * Interested in helping people learn how to build applications on Riak or Riak Core? You can find requests for Help Wanted in the Basho Community Repo [4] * The discussion on repo naming convention and branching strategy will close soon [5] ## Exciting Pull Requests * TBK15 opened a PR on riak_core that has it compiling on Erlang 18! [6] * Muntasir is due to open a PR that includes his adaptive middleware that tunes consistency on Riak KV. Read the thread for more details [7] ## Recently Answered * Ciprian provides Fast Track guidance for Riak S2 (aka Riak CS) [8] and opens a ticket to address it in docs [9] * Outback Dingo has a few threads on setting up Riak S2 (aka Riak CS) on freeBSD 10 [10] * Kota gives a detailed explanation of why sharing memory across Riak S2 and Riak KV is not recommended, but is possible [11] * Magnus correlates a Solr shutdown error with the need for more memory dedicated to it in riak.conf [12] ## Jobs for Riak Users * Riot Games tweeted about an open software engineering position that our community may be interested in [13] * We, at Basho, have opened up a few software engineering roles as well [14] ## Weekend Reads * Scott Lystig Fritchie continues to update the Machi documentation on GitHub. If you're curious about Chain Replication and the future of large object storage in Riak, it's well worth a read [15] Have a great weekend, Matt Brender Developer Advocate @ Basho @mjbrender [0] https://github.com/basho/riak-dotnet-client/releases/tag/v2.2.0 [1] http://docs.basho.com/dataplatform/latest/ [2] https://github.com/basho-labs/the-basho-community/issues/62 [3] https://github.com/basho/spark-riak-connector [4] https://github.com/basho-labs/the-basho-community/labels/status%2Fhelp-wanted [5] https://github.com/basho-labs/the-basho-community/issues/74 [6] https://github.com/basho/riak_core/pull/785 [7] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2015-September/017596.html [8] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2015-September/017595.html [9] https://github.com/basho/basho_docs/issues/1830 [10] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2015-September/thread.html [11] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2015-September/017583.html [12] http://lists.basho.com/pipermail/riak-users_lists.basho.com/2015-September/017591.html [13] http://riotgames.com/careers/10838 [14] http://bashojobs.theresumator.com/ [15] https://github.com/basho/machi/tree/master/doc Matt Brender | Developer Advocacy Lead Basho Technologies t: @mjbrender c: +1 617.817.3195 ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: adding auth user not valid
On Tue, Sep 29, 2015 at 11:01 AM, Shunichi Shinohara
wrote:
> Hi Outback,
>
> Sorry for very late response.
> It seems that riak-cs doesn't/can't communicate with stanchion node.
> There are a couple of possible reasons:
>
> - admin.key and admin.secret should be set both in riak-cs and stanchion
> - stanchion_host should be set properly as stanchion node's listen
> configuration
> - servers that riak-cs node and stanchion nodes are running should be
> communicate by TCP.
>
> I think packet capture (by tcpdump or wireshark or...) is useful to
> investigate the communication between riak-cs and stanchion.
>
> Thanks,
> Shino
>
more like ENOTENOUGHNODES. pretty perplexing to chase down when you
think somethings running and its not
anyway i got it sorted out thanks,.
>
> 2015-09-15 13:21 GMT+09:00 Outback Dingo :
> > seems the docs with auth creds is invalid... or im still broken...
> >
> >
> >
> > curl -XPOST http://localhost:8080/riak-cs/user \
> > -H 'Content-Type: application/json' \
> > -d '{"email":"ad...@admin.com", "name":"admin"}'
> >
> >
> > returns
> >
> > curl -H 'Content-Type: application/json' -X POST
> > http://localhost:8080/riak-cs/user --data '{"email":"ad...@admin.com",
> > "name":"admin"}'
> >
> > 500 Internal Server
> > ErrorInternal Server ErrorThe server
> > encountered an error while processing this
> > requestroot@vmbsd:/usr/local/etc/riak-cs #
> >
> > and the log shows
> > 2015-09-15 14:20:33.678 [error] <0.9846.4> Webmachine error at path
> > "/riak-cs/user" :
> >
> {error,{error,{badmatch,{error,malformed_xml}},[{riak_cs_s3_response,xml_error_code,1,[{file,"src/riak_cs_s3_response.erl"},{line,335}]},{riak_cs_s3_r
> >
> esponse,error_response,1,[{file,"src/riak_cs_s3_response.erl"},{line,249}]},{riak_cs_wm_user,accept_json,2,[{file,"src/riak_cs_wm_user.erl"},{line,119}]},{webmachine_resource,resource_call,3,[{file,"src/webmachine_resource.erl"},{line,1
> >
> 86}]},{webmachine_resource,do,3,[{file,"src/webmachine_resource.erl"},{line,142}]},{webmachine_decision_core,resource_call,...},...]}}
> > in riak_cs_s3_response:xml_error_code/1 line 335
> >
> >
> > ___
> > riak-users mailing list
> > riak-users@lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
