Riak.filterNotFound and Riak.reduceSort in reduce phase dramaticaly decrease performance
query1.sh:
=
curl -w %{time_connect}:%{time_starttransfer}:%{time_total} -X POST \
-H "content-type: application/json" \
-d @- \
http://localhost:8098/mapred \
< 0.005:1.468:1.468
query2.sh:
curl -w %{time_connect}:%{time_starttransfer}:%{time_total} -X POST \
-H "content-type: application/json" \
-d @- \
http://localhost:8098/mapred \
<0.005:1.439:1.439
query3.sh:
curl -w %{time_connect}:%{time_starttransfer}:%{time_total} -X POST \
-H "content-type: application/json" \
-d @- \
http://localhost:8098/mapred \
< 0.005:0.218:0.218
As for me, this is not so fast too. because currently it is very simple and
if I add some logic or users data is grow, all things will be much
slower.
Total number of keys in bucket: 3703
Keys matched with index: 299
Environment: cluster of 3 ec2 c3.xlarge instances(debian).
storage_backend : riak_kv_eleveldb_backend
erlydtl_version : <<"0.7.0">>
riak_control_version : <<"1.4.4-0-g9a74e57">>
cluster_info_version : <<"1.2.4">>
riak_search_version : <<"1.4.8-0-gbe6e4ed">>
merge_index_version : <<"1.3.2-0-gcb38ee7">>
riak_kv_version : <<"1.4.8-0-g7545390">>
sidejob_version : <<"0.2.0">>
riak_api_version : <<"1.4.4-0-g395e6fd">>
riak_pipe_version : <<"1.4.4-0-g7f390f3">>
riak_core_version : <<"1.4.4">>
bitcask_version : <<"1.6.6-0-g230b6d6">>
basho_stats_version : <<"1.0.3">>
webmachine_version : <<"1.10.4-0-gfcff795">>
mochiweb_version : <<"1.5.1p6">>
inets_version : <<"5.9">>
erlang_js_version : <<"1.2.2">>
runtime_tools_version : <<"1.8.8">>
os_mon_version : <<"2.2.9">>
riak_sysmon_version : <<"1.1.3">>
ssl_version : <<"5.0.1">>
public_key_version : <<"0.15">>
crypto_version : <<"2.1">>
sasl_version : <<"2.2.1">>
lager_version : <<"2.0.1">>
goldrush_version : <<"0.1.5">>
compiler_version : <<"4.8.1">>
syntax_tools_version : <<"1.6.8">>
stdlib_version : <<"1.18.1">>
kernel_version : <<"2.15.1">>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak 2.0 search changes
1. Does it support wildcards in a middle or start? *abc, a*bc, 2. Does presort supports any field instead of key or score? I'm not found this in 2.0 docs. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
[no subject]
from documentation: >>Currently, the wildcard must come at the end of the term in both cases, and must be preceded by a minimum of two characters. Does it any way to able to search with one char? I use RiakSearch to search contacts, and should show something for users when they type one char only Maybe some combination with proximity search or custom tokenizers? Also what about riak 2.0? Does it work there? I have just one idea, create custom index with field first_letter and put first char there and when only single char comes to input - I will search exact q=first_letter:a ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
riak search presort
does riak support DESC order with presort=key ?
simple presort=key desc gives me error
{error,
{error,function_clause,
[{riak_search_client,search_doc,
[{riak_search_schema,<<"items">>,"1.1",3,<<"value">>,
<<"id">>,
[{riak_search_field,<<"^.*_num$">>,[],integer,10,48,false,
false,true,
{erlang,text_analyzers,integer_analyzer_factory},
["10"],
false},
{riak_search_field,<<"^.*_int$">>,[],integer,10,48,false,
false,true,
{erlang,text_analyzers,integer_analyzer_factory},
["10"],
false},
{riak_search_field,<<"^.*_dt$">>,[],date,10,48,false,
false,true,
{erlang,text_analyzers,noop_analyzer_factory},
undefined,false},
{riak_search_field,<<"^.*_date$">>,[],date,10,48,false,
false,true,
{erlang,text_analyzers,noop_analyzer_factory},
undefined,false},
{riak_search_field,<<"^.*_txt$">>,[],string,0,32,false,
false,true,
{erlang,text_analyzers,standard_analyzer_factory},
undefined,false},
{riak_search_field,<<"^.*_text$">>,[],string,0,32,false,
false,true,
{erlang,text_analyzers,standard_analyzer_factory},
undefined,false},
{riak_search_field,<<"^.*$">>,[],string,0,32,false,false,
true,
{erlang,text_analyzers,whitespace_analyzer_factory},
undefined,false}],
'or',
{erlang,text_analyzers,whitespace_analyzer_factory}},
{scope,#Ref<0.0.5.19421>,"items","value",
{scope,#Ref<0.0.5.19420>,undefined,"__version",
{term,#Ref<0.0.5.19424>,<<"1">>,
[{'riak@127.0.0.1',162},
{'riak@127.0.0.1',195},
{'riak@127.0.0.1',162}],
173.0,1.0}}},
[],0,10,'key desc',6,
{riak_search_client,{riak_client,'riak@127.0.0.1',undefined}}],
[{file,"src/riak_search_client.erl"},{line,141}]},
{riak_search_utils,run_query,7,
[{file,"src/riak_search_utils.erl"},{line,283}]},
{riak_solr_searcher_wm,to_xml,2,
[{file,"src/riak_solr_searcher_wm.erl"},{line,131}]},
{webmachine_resource,resource_call,3,
[{file,"src/webmachine_resource.erl"},{line,186}]},
{webmachine_resource,do,3,
[{file,"src/webmachine_resource.erl"},{line,142}]},
{webmachine_decision_core,resource_call,1,
[{file,"src/webmachine_decision_core.erl"},{line,48}]},
{webmachine_decision_core,decision,1,
[{file,"src/webmachine_decision_core.erl"},{line,558}]},
{webmachine_decision_core,handle_request,2,
[{file,"src/webmachine_decision_core.erl"},{line,33}]}]}}
--
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak 1.4.8, Riak Search numFound inconsistency
solr/some_index/select?q=some_query&wt=json&rows=50
{
- responseHeader:
{
- status: 0,
- QTime: 47,
- params:
{
- q: "some_query",
- q.op: "or",
- filter: "",
- wt: "json"
}
},
- response:
{
- numFound: 34,
- start: 0,
- maxScore: "0.353553",
- docs: [ ]
}
}
I was manage my index only using the Solr HTTP Interface
1. Looks i got some broken records. How can i found and cleanup it?
2. How to prevent this one in future
I was try:
1. Reboot entire cluster( to eliminate in memory cache )
2. search-cmd delete my_index
no success
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak 1.4.8, Riak Search numFound inconsistency
I can see records ids if I add &fl=id in query:
solr/some_index/select?q=some_query&wt=json&rows=50&fl=id
{
-
- response:
{
- numFound: 33,
- start: 0,
- maxScore: "0.0",
- docs:
[
-
{
- id:
"1396297239219-7a5886c8d03549798f23b8c2e46739e6-660761ea9e3d472487be4e01f077f706"
},
-
{
- id:
"8600017235068-7a5886c8d03549798f23b8c2e46739e6-e3d0b4904161437ca47368afed842737"
},
-
{
- id:
"8600019503366-7a5886c8d03549798f23b8c2e46739e6-9e3990c226b44b1fb747baf6dfd868a2"
},
But when I try remove this IDs by POST to http://db:8098/solr/some_index
/update
1396297239219-7a5886c8d03549798f23b8c2e46739e6-660761ea9e3d472487be4e01f077f706
Nothing happens. entry still returnig
On Tue, May 13, 2014 at 5:11 PM, Alexander Popov wrote:
> solr/some_index/select?q=some_query&wt=json&rows=50
>
> {
>
>- responseHeader:
>{
> - status: 0,
> - QTime: 47,
> - params:
> {
> - q: "some_query",
> - q.op: "or",
> - filter: "",
> - wt: "json"
> }
> },
>- response:
>{
> - numFound: 34,
> - start: 0,
> - maxScore: "0.353553",
> - docs: [ ]
> }
>
> }
> I was manage my index only using the Solr HTTP Interface
>
> 1. Looks i got some broken records. How can i found and cleanup it?
> 2. How to prevent this one in future
>
> I was try:
> 1. Reboot entire cluster( to eliminate in memory cache )
> 2. search-cmd delete my_index
>
> no success
>
>
>
>
>
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Indexing using the Solr Interface, add after delete will not applied on cluster
I've delete index using http://docs.basho.com/riak/latest/dev/references/search-indexing/#Deleting-using-the-Solr-Interface and than immediately add it by http://docs.basho.com/riak/latest/dev/references/search-indexing/#Indexing-using-the-Solr-Interface Item will not appears in search results in cluster setup. On single node not seen this issue. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Yokozuna search
Will queries support masks at beging and 1 char mask like *lala and a* ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Forbidden error after upgrade on riak 2.0
When query Yokozuna, sometimes fails with error: 403 ForbiddenForbiddenForbiddenmochiweb+webmachine web server url logged with error: /search/query/full-proof-info_index?q=owner%3A6d87f18a3dca4a60b0fc385b1f46c165%20AND%20archived%3Afalse%20AND%20NOT%20draft%3Atrue&start=0&rows=20&wt=json&sort=updated_t imestamp%20desc if repeat query to this url in browser - all works fine. No errors in logs. No clue whats wrong ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
should solr facet.pivot works with yokozuna?
I provide it to query, see in response in params section: id,owner,type but facet output section have no pivot section 71 14 1 1 does anybody can confirm this functionality works? probably it is disabled in SOLR? Or need some schema setup ? according to http://docs.basho.com/riak/latest/dev/advanced/search-schema/#Field-Properties-By-Use-Case I just need indexed=true for all fields ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Proper way do delete datatypes ( map )
I have map bucket with some data( with neseted maps )
aprox structure( don't know does it matter or not ):
{
"update": {
"some_register": "value",
"some_flag": "disable",
"nested_map": {
"update": {
"nested1_map": {
"update": {
"some_register": "value",
"some_flag": "disable",
}
},
"nested1_map": {
"update": {
"some_register": "value",
"some_flag": "disable",
}
}
}
},
"some_counter": 13,
}
}
Updates works fine. Even simultaneous
But Sometimes I need recreate entire value, so I Delete it using
curl -XDELETE http://host:8098/types/maps/buckets/mybucket/keys/some
After that sometimes siblings appears.
curl -H "Accept: multipart/mixed"
http://host:8098/types/maps/buckets/mybucket/keys/some
show that confict with delete
--XZ98hy0TJbr4sVETS44XBEJf7Yt
Last-Modified: Thu, 02 Oct 2014 11:29:15 GMT
E
��A�6- some binary
--XZ98hy0TJbr4sVETS44XBEJf7Yt
Content-Type: application/octet-stream
Link: ; rel="up"
Etag: 1MqocFt6qWeQxIw8bE1B8e
Last-Modified: Thu, 02 Oct 2014 11:29:03 GMT
X-Riak-Deleted: true
--XZ98hy0TJbr4sVETS44XBEJf7Yt--
Further updates to datatype using
http://host:8098/types/maps/buckets/mybucket/datatypes/some NOT create new
sibling. It is replace previous one.
Problems:
1. I should remove datatype with different method? Or how to resolve such
conflicts? probably
Data I receive is binary, and query like
curl -H "Accept: multipart/mixed"
http://host:8098/types/maps/buckets/mybucket/datatypes/some
to get json data not works.Should I post it as binary back?
2. I have also search index on this bucket.
each further updates to this datatype before resolution creates new
records in solr because
_yz_id includes sibling id:
1*maps*mybucket*some*34*46hGXxyhuW3yn3L8bRHIml
good news that when I delete record again all entries in solr deleted too.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Proper way do delete datatypes ( map )
Since I use http interface I decide for now to not delete records at all.
I just empty it by
{ remove :[allmykeys], update:{timestamp}}
command.
Probably later I will implement kind of task to kill lost empty items, but
now this way work fine for me. I will never get siblings headache and
yokozuna index is fine all time
P.s I'm realy appreciate new RIAK. It is incredible improvement. Great
work!
02.10.2014 20:10 пользователь "Russell Brown"
написал:
> Hi Alexander,
>
> I think you are deleting data-types the proper way.
>
> What is your `delete_mode` setting, please?
>
> I would guess that sibling you are seeing is a tombstone, which suggests
> you have some concurrent update with the delete.
>
> You will only ever have a single CRDT sibling, and 1 (or possibly more)
> tombstone siblings.
>
> If you write back just the binary CRDT sibling as the value in the normal
> K/V interface, any concurrent data type updates will merge ok to a single
> value.
>
> I don’t know about your Yokozuna question, but I think it is just that
> Yokozuna indexes each sibling as an independent document.
>
> Let me know if that works out OK for you, please?
>
> Cheers
>
> Russell
>
> On 2 Oct 2014, at 12:54, Alexander Popov wrote:
>
> I have map bucket with some data( with neseted maps )
> aprox structure( don't know does it matter or not ):
> {
> "update": {
> "some_register": "value",
> "some_flag": "disable",
> "nested_map": {
> "update": {
> "nested1_map": {
> "update": {
> "some_register": "value",
> "some_flag": "disable",
> }
> },
> "nested1_map": {
> "update": {
> "some_register": "value",
> "some_flag": "disable",
> }
> }
> }
> },
> "some_counter": 13,
> }
> }
>
>
> Updates works fine. Even simultaneous
> But Sometimes I need recreate entire value, so I Delete it using
> curl -XDELETE http://host:8098/types/maps/buckets/mybucket/keys/some
>
> After that sometimes siblings appears.
>
>
> curl -H "Accept: multipart/mixed"
> http://host:8098/types/maps/buckets/mybucket/keys/some
>
> show that confict with delete
>
> --XZ98hy0TJbr4sVETS44XBEJf7Yt
> Last-Modified: Thu, 02 Oct 2014 11:29:15 GMT
>
> E
> ��A�6- some binary
> --XZ98hy0TJbr4sVETS44XBEJf7Yt
> Content-Type: application/octet-stream
> Link: ; rel="up"
> Etag: 1MqocFt6qWeQxIw8bE1B8e
> Last-Modified: Thu, 02 Oct 2014 11:29:03 GMT
> X-Riak-Deleted: true
>
>
> --XZ98hy0TJbr4sVETS44XBEJf7Yt--
>
>
> Further updates to datatype using
> http://host:8098/types/maps/buckets/mybucket/datatypes/some NOT create
> new sibling. It is replace previous one.
>
> Problems:
>
> 1. I should remove datatype with different method? Or how to resolve such
> conflicts? probably
> Data I receive is binary, and query like
> curl -H "Accept: multipart/mixed"
> http://host:8098/types/maps/buckets/mybucket/datatypes/some
> to get json data not works.Should I post it as binary back?
>
>
> 2. I have also search index on this bucket.
> each further updates to this datatype before resolution creates new
> records in solr because
> _yz_id includes sibling id:
> 1*maps*mybucket*some*34*46hGXxyhuW3yn3L8bRHIml
>
> good news that when I delete record again all entries in solr deleted too.
>
>
>
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
transfer partition restarts and failed many times.
and seems he cannot finish transfer
part of log:
2014-10-09 07:07:27.360 [error]
<0.988.2>@riak_core_handoff_sender:start_fold:272 ownership_transfer
transfer of riak_kv_vnode from 'riak@10.0.0.218'
1370157784997721485815954530671515330927436759040 to 'ri
ak@10.0.0.152' 1370157784997721485815954530671515330927436759040 failed
because of enotconn
2014-10-09 07:07:27.360 [error]
<0.177.0>@riak_core_handoff_manager:handle_info:285 An outbound handoff of
partition riak_kv_vnode 1370157784997721485815954530671515330927436759040
was terminated for reason
: {shutdown,{error,enotconn}}
2014-10-09 07:08:30.686 [error]
<0.3648.2>@riak_core_handoff_sender:start_fold:269 ownership_transfer
transfer of riak_kv_vnode from 'riak@10.0.0.218'
913438523331814323877303020447676887284957839360 to 'ri
ak@10.0.0.152' 913438523331814323877303020447676887284957839360 failed
because of TCP recv timeout
2014-10-09 07:08:30.686 [error]
<0.177.0>@riak_core_handoff_manager:handle_info:285 An outbound handoff of
partition riak_kv_vnode 913438523331814323877303020447676887284957839360
was terminated for reason:
{shutdown,timeout}
2014-10-09 07:08:37.386 [error]
<0.4068.2>@riak_core_handoff_sender:start_fold:272 ownership_transfer
transfer of riak_kv_vnode from 'riak@10.0.0.218'
1370157784997721485815954530671515330927436759040 to 'r
iak@10.0.0.152' 1370157784997721485815954530671515330927436759040 failed
because of enotconn
2014-10-09 07:08:37.386 [error]
<0.177.0>@riak_core_handoff_manager:handle_info:285 An outbound handoff of
partition riak_kv_vnode 1370157784997721485815954530671515330927436759040
was terminated for reason
: {shutdown,{error,enotconn}}
2014-10-09 07:09:40.646 [error]
<0.6721.2>@riak_core_handoff_sender:start_fold:269 ownership_transfer
transfer of riak_kv_vnode from 'riak@10.0.0.218'
913438523331814323877303020447676887284957839360 to 'ri
ak@10.0.0.152' 913438523331814323877303020447676887284957839360 failed
because of TCP recv timeout
2014-10-09 07:09:40.646 [error]
<0.177.0>@riak_core_handoff_manager:handle_info:285 An outbound handoff of
partition riak_kv_vnode 913438523331814323877303020447676887284957839360
was terminated for reason:
{shutdown,timeout}
2014-10-09 07:09:47.345 [error]
<0.7132.2>@riak_core_handoff_sender:start_fold:272 ownership_transfer
transfer of riak_kv_vnode from 'riak@10.0.0.218'
1370157784997721485815954530671515330927436759040 to 'r
iak@10.0.0.152' 1370157784997721485815954530671515330927436759040 failed
because of enotconn
2014-10-09 07:09:47.345 [error]
<0.177.0>@riak_core_handoff_manager:handle_info:285 An outbound handoff of
partition riak_kv_vnode 1370157784997721485815954530671515330927436759040
was terminated for reason
: {shutdown,{error,enotconn}}
2014-10-09 07:10:50.590 [error]
<0.9901.2>@riak_core_handoff_sender:start_fold:269 ownership_transfer
transfer of riak_kv_vnode from 'riak@10.0.0.218'
913438523331814323877303020447676887284957839360 to 'ri
ak@10.0.0.152' 913438523331814323877303020447676887284957839360 failed
because of TCP recv timeout
2014-10-09 07:10:50.590 [error]
<0.177.0>@riak_core_handoff_manager:handle_info:285 An outbound handoff of
partition riak_kv_vnode 913438523331814323877303020447676887284957839360
was terminated for reason:
{shutdown,timeout}
2014-10-09 07:10:57.412 [error]
<0.10318.2>@riak_core_handoff_sender:start_fold:272 ownership_transfer
transfer of riak_kv_vnode from 'riak@10.0.0.218'
1370157784997721485815954530671515330927436759040 to '
riak@10.0.0.152' 1370157784997721485815954530671515330927436759040 failed
because of enotconn
2014-10-09 07:10:57.413 [error]
<0.177.0>@riak_core_handoff_manager:handle_info:285 An outbound handoff of
partition riak_kv_vnode 1370157784997721485815954530671515330927436759040
was terminated for reason
: {shutdown,{error,enotconn}}
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
[SOLR] different number of result on same query
RIAK 2.0.1, 5 nodes on different hosts
query:
comments_index?q=owner:6d87f18a3dca4a60b0fc385b1f46c165%20AND%20target:35975db44af44b2494751abddfcfe466&fl=id&wt=json&rows=15
RESULT1:
{
responseHeader: {
status: 0,
QTime: 3,
params: {
10.0.0.150:8093: "_yz_pn:56 OR _yz_pn:41 OR _yz_pn:26 OR _yz_pn:11",
shards: "
10.0.0.150:8093/internal_solr/comments_index,10.0.0.151:8093/internal_solr/comments_index,10.0.0.152:8093/internal_solr/comments_index,10.0.0.153:8093/internal_solr/comments_index,10.0.0.218:8093/internal_solr/comments_index
",
fl: "id",
10.0.0.152:8093: "_yz_pn:63 OR _yz_pn:53 OR _yz_pn:38 OR _yz_pn:23 OR
_yz_pn:8",
q: "owner:6d87f18a3dca4a60b0fc385b1f46c165 AND
target:35975db44af44b2494751abddfcfe466",
10.0.0.218:8093: "(_yz_pn:60 AND (_yz_fpn:60)) OR _yz_pn:50 OR _yz_pn:35 OR
_yz_pn:20 OR _yz_pn:5",
wt: "json",
10.0.0.153:8093: "_yz_pn:59 OR _yz_pn:44 OR _yz_pn:29 OR _yz_pn:14",
10.0.0.151:8093: "_yz_pn:47 OR _yz_pn:32 OR _yz_pn:17 OR _yz_pn:2",
rows: "15"
}
},
response: {
numFound: 12,
start: 0,
maxScore: 6.72534,
docs: [
.
RESULT2:
{
responseHeader: {
status: 0,
QTime: 3,
params: {
10.0.0.150:8093: "_yz_pn:61 OR _yz_pn:46 OR _yz_pn:31 OR _yz_pn:16 OR
_yz_pn:1",
shards: "
10.0.0.150:8093/internal_solr/comments_index,10.0.0.151:8093/internal_solr/comments_index,10.0.0.152:8093/internal_solr/comments_index,10.0.0.153:8093/internal_solr/comments_index,10.0.0.218:8093/internal_solr/comments_index
",
fl: "id",
10.0.0.152:8093: "_yz_pn:58 OR _yz_pn:43 OR _yz_pn:28 OR _yz_pn:13",
q: "owner:6d87f18a3dca4a60b0fc385b1f46c165 AND
target:35975db44af44b2494751abddfcfe466",
10.0.0.218:8093: "_yz_pn:55 OR _yz_pn:40 OR _yz_pn:25 OR _yz_pn:10",
wt: "json",
10.0.0.153:8093: "_yz_pn:49 OR _yz_pn:34 OR _yz_pn:19 OR _yz_pn:4",
10.0.0.151:8093: "(_yz_pn:62 AND (_yz_fpn:62)) OR _yz_pn:52 OR _yz_pn:37 OR
_yz_pn:22 OR _yz_pn:7",
rows: "15"
}
},
response: {
numFound: 11,
start: 0,
maxScore: 6.72534,
docs: [
Does it not guaranteed that all records will be found each time?
also, seems not random record missed, each time specific one
if I run query for this specific record with q=id:
50473c1239934ef29f24b87f5a6d1ca2
it is random return or not return this record
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: [SOLR] different number of result on same query
Also if run query with specific id, on internal_solr directly on each host,
for this problematic id it founds only on 2 hosts, but other id's found on
3 of them.
On Wed, Oct 22, 2014 at 10:18 PM, Alexander Popov
wrote:
> RIAK 2.0.1, 5 nodes on different hosts
>
>
> query:
> comments_index?q=owner:6d87f18a3dca4a60b0fc385b1f46c165%20AND%20target:35975db44af44b2494751abddfcfe466&fl=id&wt=json&rows=15
>
> RESULT1:
> {
> responseHeader: {
> status: 0,
> QTime: 3,
> params: {
> 10.0.0.150:8093: "_yz_pn:56 OR _yz_pn:41 OR _yz_pn:26 OR _yz_pn:11",
> shards: "
> 10.0.0.150:8093/internal_solr/comments_index,10.0.0.151:8093/internal_solr/comments_index,10.0.0.152:8093/internal_solr/comments_index,10.0.0.153:8093/internal_solr/comments_index,10.0.0.218:8093/internal_solr/comments_index
> ",
> fl: "id",
> 10.0.0.152:8093: "_yz_pn:63 OR _yz_pn:53 OR _yz_pn:38 OR _yz_pn:23 OR
> _yz_pn:8",
> q: "owner:6d87f18a3dca4a60b0fc385b1f46c165 AND
> target:35975db44af44b2494751abddfcfe466",
> 10.0.0.218:8093: "(_yz_pn:60 AND (_yz_fpn:60)) OR _yz_pn:50 OR _yz_pn:35
> OR _yz_pn:20 OR _yz_pn:5",
> wt: "json",
> 10.0.0.153:8093: "_yz_pn:59 OR _yz_pn:44 OR _yz_pn:29 OR _yz_pn:14",
> 10.0.0.151:8093: "_yz_pn:47 OR _yz_pn:32 OR _yz_pn:17 OR _yz_pn:2",
> rows: "15"
> }
> },
> response: {
> numFound: 12,
> start: 0,
> maxScore: 6.72534,
> docs: [
> .
>
>
> RESULT2:
> {
> responseHeader: {
> status: 0,
> QTime: 3,
> params: {
> 10.0.0.150:8093: "_yz_pn:61 OR _yz_pn:46 OR _yz_pn:31 OR _yz_pn:16 OR
> _yz_pn:1",
> shards: "
> 10.0.0.150:8093/internal_solr/comments_index,10.0.0.151:8093/internal_solr/comments_index,10.0.0.152:8093/internal_solr/comments_index,10.0.0.153:8093/internal_solr/comments_index,10.0.0.218:8093/internal_solr/comments_index
> ",
> fl: "id",
> 10.0.0.152:8093: "_yz_pn:58 OR _yz_pn:43 OR _yz_pn:28 OR _yz_pn:13",
> q: "owner:6d87f18a3dca4a60b0fc385b1f46c165 AND
> target:35975db44af44b2494751abddfcfe466",
> 10.0.0.218:8093: "_yz_pn:55 OR _yz_pn:40 OR _yz_pn:25 OR _yz_pn:10",
> wt: "json",
> 10.0.0.153:8093: "_yz_pn:49 OR _yz_pn:34 OR _yz_pn:19 OR _yz_pn:4",
> 10.0.0.151:8093: "(_yz_pn:62 AND (_yz_fpn:62)) OR _yz_pn:52 OR _yz_pn:37
> OR _yz_pn:22 OR _yz_pn:7",
> rows: "15"
> }
> },
> response: {
> numFound: 11,
> start: 0,
> maxScore: 6.72534,
> docs: [
>
>
>
> Does it not guaranteed that all records will be found each time?
> also, seems not random record missed, each time specific one
> if I run query for this specific record with q=id:
> 50473c1239934ef29f24b87f5a6d1ca2
> it is random return or not return this record
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: [SOLR] different number of result on same query
Initially it was 1 node, upgraded from 1.4.8.
Than I add second node, and than added another 3 nodes before transfer to
second is finished.
There was several hangs while transferring between nodes, and I was
restart they several times.
Should re-save problematic records helps with this issue?
On Wed, Oct 22, 2014 at 10:42 PM, Eric Redmond wrote:
> Have you added/removed nodes from the cluster, or did you start populating
> data before the cluster was done being built? You may have ran into a known
> handoff bug that we're currently working on.
>
> Eric
>
>
> On Oct 22, 2014, at 11:32 AM, Alexander Popov wrote:
>
> Also if run query with specific id, on internal_solr directly on each
> host, for this problematic id it founds only on 2 hosts, but other id's
> found on 3 of them.
>
> On Wed, Oct 22, 2014 at 10:18 PM, Alexander Popov
> wrote:
>
>> RIAK 2.0.1, 5 nodes on different hosts
>>
>>
>> query:
>> comments_index?q=owner:6d87f18a3dca4a60b0fc385b1f46c165%20AND%20target:35975db44af44b2494751abddfcfe466&fl=id&wt=json&rows=15
>>
>> RESULT1:
>> {
>> responseHeader: {
>> status: 0,
>> QTime: 3,
>> params: {
>> 10.0.0.150:8093: "_yz_pn:56 OR _yz_pn:41 OR _yz_pn:26 OR _yz_pn:11",
>> shards: "
>> 10.0.0.150:8093/internal_solr/comments_index,10.0.0.151:8093/internal_solr/comments_index,10.0.0.152:8093/internal_solr/comments_index,10.0.0.153:8093/internal_solr/comments_index,10.0.0.218:8093/internal_solr/comments_index
>> ",
>> fl: "id",
>> 10.0.0.152:8093: "_yz_pn:63 OR _yz_pn:53 OR _yz_pn:38 OR _yz_pn:23 OR
>> _yz_pn:8",
>> q: "owner:6d87f18a3dca4a60b0fc385b1f46c165 AND
>> target:35975db44af44b2494751abddfcfe466",
>> 10.0.0.218:8093: "(_yz_pn:60 AND (_yz_fpn:60)) OR _yz_pn:50 OR _yz_pn:35
>> OR _yz_pn:20 OR _yz_pn:5",
>> wt: "json",
>> 10.0.0.153:8093: "_yz_pn:59 OR _yz_pn:44 OR _yz_pn:29 OR _yz_pn:14",
>> 10.0.0.151:8093: "_yz_pn:47 OR _yz_pn:32 OR _yz_pn:17 OR _yz_pn:2",
>> rows: "15"
>> }
>> },
>> response: {
>> numFound: 12,
>> start: 0,
>> maxScore: 6.72534,
>> docs: [
>> .
>>
>>
>> RESULT2:
>> {
>> responseHeader: {
>> status: 0,
>> QTime: 3,
>> params: {
>> 10.0.0.150:8093: "_yz_pn:61 OR _yz_pn:46 OR _yz_pn:31 OR _yz_pn:16 OR
>> _yz_pn:1",
>> shards: "
>> 10.0.0.150:8093/internal_solr/comments_index,10.0.0.151:8093/internal_solr/comments_index,10.0.0.152:8093/internal_solr/comments_index,10.0.0.153:8093/internal_solr/comments_index,10.0.0.218:8093/internal_solr/comments_index
>> ",
>> fl: "id",
>> 10.0.0.152:8093: "_yz_pn:58 OR _yz_pn:43 OR _yz_pn:28 OR _yz_pn:13",
>> q: "owner:6d87f18a3dca4a60b0fc385b1f46c165 AND
>> target:35975db44af44b2494751abddfcfe466",
>> 10.0.0.218:8093: "_yz_pn:55 OR _yz_pn:40 OR _yz_pn:25 OR _yz_pn:10",
>> wt: "json",
>> 10.0.0.153:8093: "_yz_pn:49 OR _yz_pn:34 OR _yz_pn:19 OR _yz_pn:4",
>> 10.0.0.151:8093: "(_yz_pn:62 AND (_yz_fpn:62)) OR _yz_pn:52 OR _yz_pn:37
>> OR _yz_pn:22 OR _yz_pn:7",
>> rows: "15"
>> }
>> },
>> response: {
>> numFound: 11,
>> start: 0,
>> maxScore: 6.72534,
>> docs: [
>>
>>
>>
>> Does it not guaranteed that all records will be found each time?
>> also, seems not random record missed, each time specific one
>> if I run query for this specific record with q=id:
>> 50473c1239934ef29f24b87f5a6d1ca2
>> it is random return or not return this record
>>
>>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
update schema without reindex all data
does it possible?
now I use flow:
curl -XPUT "http://localhost:8098/buckets/some_bucket/props";
-H'content-type: application/json'
-d'{"props":{"search_index":"_dont_index_", "search":false}}'
curl -XDELETE "http://localhost:8098/search/index/some_index";
curl -XPUT "http://localhost:8098/search/schema/some_schema";
-H'content-type:application/xml' --data-binary @ some_schema.xml
curl -XPUT "http://localhost:8098/search/index/some_index";
-H'content-type:application/json' -d"{\"schema\":\"some_schema\"}"
curl -XPUT "http://localhosy:8098/buckets/some_bucket/props";
-H'content-type: application/json'
-d"{\"props\":{\"search_index\":\"some_index\"}}"
after that All indexed data removed, and I need resave all data.
It works when there is mid size bucket. but how I should update schema
in production when it have millions records.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
[SOLR]Does indexing is async operatoin?
I put data to some bucket with indexing schema, Immediately after that I'm trying to query this data using search interface very often just inserted data is missed. Does it any way to know when it will be done? or any other ways how to achieve this? ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
How you are dealing with spikes?
I have Riak 2.0.1 cluster with 5 nodes ( ec2 m3-large ) with elnm in front sometimes I got spikes up to 10 seconds I can't say that I have huge load at this time, max 200 requests per second for all 5 nodes. Most expensive queries is * list by secondary index ( usually returns from 0 to 100 records ) * and solr queries( max 10 records ) save operations is slowdown sometimes but not so much ( up to 1 sec ) It's slowdown not for specific requests, same one work pretty fast later. Does it any possibilities to profile|log somehow to determine reason why this happen? ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: How you are dealing with spikes?
Stats when recent spike happens for 15 minutes around it get (826) save (341) listByIndex (1161) mapReduce (621) //Input is IDs list SOLR (4294) 6 Solr requests was longer than 9sec ( all returns 0 rows ) 4 Solr requests was longer within 4-5s ( both returns 0 rows ) 11 listByIndex requests was longer than within 4-5s ( both returns 0 rows ) all another requests was less than 300ms Sometimes more load do not make such spikes Some graphs from maintanance tasks: 1. http://i.imgur.com/xAE6B06.png 3 simple tasks, first 2 of them reads all keys, decide to do nothing and continue so just read happens, third task resave all data in bucket. since rate is pretty good, some peaks happens 2. More complex task http://i.imgur.com/7nwHb3Q.png, it have more serious computing, and updating typed bucked( map ), but no peaks to 9s sysctl -a | fgrep vm.dirty_: vm.dirty_background_bytes = 0 vm.dirty_background_ratio = 10 vm.dirty_bytes = 0 vm.dirty_expire_centisecs = 3000 vm.dirty_ratio = 20 vm.dirty_writeback_centisecs = 500 On Tue, Dec 9, 2014 at 5:46 PM, Luke Bakken wrote: > Hi Alexander, > > Can you comment on the read vs. write load of this cluster/ > > Could you please run the following command and reply with the output? > > sysctl -a | fgrep vm.dirty_ > > We've seen cases where dirty pages get written in a synchronous manner > all at once, causing latency spikes due to I/O blocking. > -- > Luke Bakken > Engineer / CSE > lbak...@basho.com > > > On Tue, Dec 9, 2014 at 4:58 AM, Alexander Popov wrote: >> I have Riak 2.0.1 cluster with 5 nodes ( ec2 m3-large ) with elnm in front >> sometimes I got spikes up to 10 seconds >> >> I can't say that I have huge load at this time, max 200 requests per >> second for all 5 nodes. >> >> Most expensive queries is >> * list by secondary index ( usually returns from 0 to 100 records ) >> * and solr queries( max 10 records ) >> >> save operations is slowdown sometimes but not so much ( up to 1 sec ) >> >> It's slowdown not for specific requests, same one work pretty fast later. >> >> Does it any possibilities to profile|log somehow to determine reason >> why this happen? >> >> ___ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: How you are dealing with spikes?
Left graph show counts, Right graph show times, graphs are synchronized by time What about SOLR requests instead 2i? should it be faster? Or what you recommend to use for populating lists of users data? for example now we have files, that have 2i like owner, so when user request his files, we populating buckets/files/owner_bin/user_id, If we change this query to SOLR analog -> can we gave some boost? Also, does key length matter for 2i performance? Does number of 2i indexes per object matter for 2i? On Tue, Dec 9, 2014 at 7:54 PM, Alexander Popov wrote: > Stats when recent spike happens for 15 minutes around it > get (826) > save (341) > listByIndex (1161) > mapReduce (621) //Input is IDs list > SOLR (4294) > > 6 Solr requests was longer than 9sec ( all returns 0 rows ) > 4 Solr requests was longer within 4-5s ( both returns 0 rows ) > 11 listByIndex requests was longer than within 4-5s ( both returns 0 rows ) > all another requests was less than 300ms > > > Sometimes more load do not make such spikes > Some graphs from maintanance tasks: > 1. http://i.imgur.com/xAE6B06.png > 3 simple tasks, first 2 of them reads all keys, decide to do > nothing and continue so just read happens, third task resave all data > in bucket. > since rate is pretty good, some peaks happens > > 2. More complex task > http://i.imgur.com/7nwHb3Q.png, it have more serious computing, and > updating typed bucked( map ), but no peaks to 9s > > > > sysctl -a | fgrep vm.dirty_: > > vm.dirty_background_bytes = 0 > vm.dirty_background_ratio = 10 > vm.dirty_bytes = 0 > vm.dirty_expire_centisecs = 3000 > vm.dirty_ratio = 20 > vm.dirty_writeback_centisecs = 500 > > On Tue, Dec 9, 2014 at 5:46 PM, Luke Bakken wrote: > > Hi Alexander, > > > > Can you comment on the read vs. write load of this cluster/ > > > > Could you please run the following command and reply with the output? > > > > sysctl -a | fgrep vm.dirty_ > > > > We've seen cases where dirty pages get written in a synchronous manner > > all at once, causing latency spikes due to I/O blocking. > > -- > > Luke Bakken > > Engineer / CSE > > lbak...@basho.com > > > > > > On Tue, Dec 9, 2014 at 4:58 AM, Alexander Popov > wrote: > >> I have Riak 2.0.1 cluster with 5 nodes ( ec2 m3-large ) with elnm in > front > >> sometimes I got spikes up to 10 seconds > >> > >> I can't say that I have huge load at this time, max 200 requests per > >> second for all 5 nodes. > >> > >> Most expensive queries is > >> * list by secondary index ( usually returns from 0 to 100 records ) > >> * and solr queries( max 10 records ) > >> > >> save operations is slowdown sometimes but not so much ( up to 1 sec ) > >> > >> It's slowdown not for specific requests, same one work pretty fast > later. > >> > >> Does it any possibilities to profile|log somehow to determine reason > >> why this happen? > >> > >> ___ > >> riak-users mailing list > >> riak-users@lists.basho.com > >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: How you are dealing with spikes?
Was replaced most 2i and MapReduce calls to SOLR requests, seems not helps to much. Now Solr request has peaks sometimes looking on /stats : what is difference between search_index_latency search_query_latency - search_query_throughput_count: 364711, - search_query_throughput_one: 0, - search_query_fail_count: 6, - search_query_fail_one: 0, - search_query_latency_min: 0, - search_query_latency_max: 0, - search_query_latency_median: 0, - search_query_latency_95: 0, - search_query_latency_99: 0, - search_query_latency_999: 0, - search_index_throughput_count: 300612, - search_index_throughtput_one: 2585, - search_index_fail_count: 367, - search_index_fail_one: 12, - search_index_latency_min: 765, - search_index_latency_max: 49859, - search_index_latency_median: 1097, - search_index_latency_95: 2801, - search_index_latency_99: 18763, - search_index_latency_999: 37138, On Mon, Dec 22, 2014 at 1:39 AM, Alexander Popov wrote: > Left graph show counts, Right graph show times, graphs are synchronized by > time > What about SOLR requests instead 2i? should it be faster? > Or what you recommend to use for populating lists of users data? for > example now we have files, > that have 2i like owner, so when user request his files, we populating > buckets/files/owner_bin/user_id, If we change this query to SOLR analog -> > can we gave some boost? > > Also, does key length matter for 2i performance? Does number of 2i > indexes per object matter for 2i? > > On Tue, Dec 9, 2014 at 7:54 PM, Alexander Popov > wrote: > >> Stats when recent spike happens for 15 minutes around it >> get (826) >> save (341) >> listByIndex (1161) >> mapReduce (621) //Input is IDs list >> SOLR (4294) >> >> 6 Solr requests was longer than 9sec ( all returns 0 rows ) >> 4 Solr requests was longer within 4-5s ( both returns 0 rows ) >> 11 listByIndex requests was longer than within 4-5s ( both returns 0 rows >> ) >> all another requests was less than 300ms >> >> >> Sometimes more load do not make such spikes >> Some graphs from maintanance tasks: >> 1. http://i.imgur.com/xAE6B06.png >> 3 simple tasks, first 2 of them reads all keys, decide to do >> nothing and continue so just read happens, third task resave all data >> in bucket. >> since rate is pretty good, some peaks happens >> >> 2. More complex task >> http://i.imgur.com/7nwHb3Q.png, it have more serious computing, and >> updating typed bucked( map ), but no peaks to 9s >> >> >> >> sysctl -a | fgrep vm.dirty_: >> >> vm.dirty_background_bytes = 0 >> vm.dirty_background_ratio = 10 >> vm.dirty_bytes = 0 >> vm.dirty_expire_centisecs = 3000 >> vm.dirty_ratio = 20 >> vm.dirty_writeback_centisecs = 500 >> >> On Tue, Dec 9, 2014 at 5:46 PM, Luke Bakken wrote: >> > Hi Alexander, >> > >> > Can you comment on the read vs. write load of this cluster/ >> > >> > Could you please run the following command and reply with the output? >> > >> > sysctl -a | fgrep vm.dirty_ >> > >> > We've seen cases where dirty pages get written in a synchronous manner >> > all at once, causing latency spikes due to I/O blocking. >> > -- >> > Luke Bakken >> > Engineer / CSE >> > lbak...@basho.com >> > >> > >> > On Tue, Dec 9, 2014 at 4:58 AM, Alexander Popov >> wrote: >> >> I have Riak 2.0.1 cluster with 5 nodes ( ec2 m3-large ) with elnm in >> front >> >> sometimes I got spikes up to 10 seconds >> >> >> >> I can't say that I have huge load at this time, max 200 requests per >> >> second for all 5 nodes. >> >> >> >> Most expensive queries is >> >> * list by secondary index ( usually returns from 0 to 100 records ) >> >> * and solr queries( max 10 records ) >> >> >> >> save operations is slowdown sometimes but not so much ( up to 1 sec ) >> >> >> >> It's slowdown not for specific requests, same one work pretty fast >> later. >> >> >> >> Does it any possibilities to profile|log somehow to determine reason >> >> why this happen? >> >> >> >> ___ >> >> riak-users mailing list >> >> riak-users@lists.basho.com >> >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> > > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
adding nodes to cluster
Upgraded riak 2.0.2-> 2.0.4
trying to add nodes to cluster( was single node )
all new nodes located on another host, but multi-nodes installation from
source
each new node have config settings tuned like
nodename = riak1@10.0.0.28
platform_data_dir = /var/lib/riak1
listener.http.internal = 0.0.0.0:18098
listener.protobuf.internal = 0.0.0.0:18087
handoff.port = 18099
search.solr.port = 18093
search.solr.jmx_port = 18985
search=on
after adding this nodes to cluster for long time was nothing was
transferred for new nodes.
I was down this nodes and force-remove it from cluster.
After this strange things happens.
On new nodes:
Even after restart cleaning entire data dir and start again, in folder yz
indexes somehow appears again.
logs contains only
2015-01-20 13:15:04.815 [info] <0.352.0>@riak_core:wait_for_service:483
Wait complete for service riak_kv (10 seconds)
2015-01-20 13:15:06.660 [info] <0.547.0>@yz_index:local_create:189 Created
index contacts_index with schema contacts
2015-01-20 13:15:08.683 [info] <0.547.0>@yz_index:local_create:189 Created
index users_index with schema users
Main Node now have bunch of errors on start and periodically after:
015-01-20 13:29:00.382 [error] <0.871.0>@riak_core_vnode:vnode_command:348
riak_kv_vnode command failed
{{badrecord,dict},[{dict,filter_dict,2,[{file,"dict.erl"},{line,464}]},{riak_dt_map,'-filter_unique/4-fun-1-',4,[{file,"src/ria
k_dt_map.erl"},{line,466}]},{sets,fold_bucket,3,[{file,"sets.erl"},{line,313}]},{sets,fold_seg,4,[{file,"sets.erl"},{line,309}]},{sets,fold_segs,4,[{file,"sets.erl"},{line,305}]},{riak_dt_map,merge,2,[{file,"src/riak_dt_map.erl"},{l
ine,454}]},{riak_kv_crdt,'-merge_value/2-fun-0-',5,[{file,"src/riak_kv_crdt.erl"},{line,204}]},{orddict,update,4,[{file,"orddict.erl"},{line,170}]}]}
2015-01-20 13:29:00.382 [error] <0.650.0>@riak_core_vnode:vnode_command:348
riak_kv_vnode command failed
{{badrecord,dict},[{dict,filter_dict,2,[{file,"dict.erl"},{line,464}]},{riak_dt_map,'-filter_unique/4-fun-1-',4,[{file,"src/ria
k_dt_map.erl"},{line,466}]},{sets,fold_bucket,3,[{file,"sets.erl"},{line,313}]},{sets,fold_seg,4,[{file,"sets.erl"},{line,309}]},{sets,fold_segs,4,[{file,"sets.erl"},{line,305}]},{riak_dt_map,merge,2,[{file,"src/riak_dt_map.erl"},{l
ine,454}]},{riak_kv_crdt,'-merge_value/2-fun-0-',5,[{file,"src/riak_kv_crdt.erl"},{line,204}]},{orddict,update,4,[{file,"orddict.erl"},{line,170}]}]}
2015-01-20 13:29:00.383 [error] <0.4520.0> gen_fsm <0.4520.0> in state
waiting_local_vnode terminated with reason: no case clause matching
{vnode_error,{{badrecord,dict},[{dict,filter_dict,2,[{file,"dict.erl"},{line,464}]},{riak_dt_
map,'-filter_unique/4-fun-1-',4,[{file,"src/riak_dt_map.erl"},{line,466}]},{sets,fold_bucket,3,[{file,"sets.erl"},{line,313}]},{sets,fold_seg,4,[{file,"sets.erl"},{line,309}]},{sets,fold_segs,4,[{file,"sets.erl"},{line,305}]},{riak_
dt_map,merge,2,[{file,"src/riak_dt_map.erl"},{line,454}]},{riak_kv_crdt,'-merge_value/2-fun-0-',5,[{file,"src/riak_kv_crdt.erl"},{line,204}]},{orddict,update,4,[{file,"or..."},...]}]}}
in riak_kv_put_fsm:waiting_local_vnode/2 line 5
67
2015-01-20 13:29:00.384 [error] <0.4520.0> CRASH REPORT Process <0.4520.0>
with 0 neighbours exited with reason: no case clause matching
{vnode_error,{{badrecord,dict},[{dict,filter_dict,2,[{file,"dict.erl"},{line,464}]},{riak_dt_ma
p,'-filter_unique/4-fun-1-',4,[{file,"src/riak_dt_map.erl"},{line,466}]},{sets,fold_bucket,3,[{file,"sets.erl"},{line,313}]},{sets,fold_seg,4,[{file,"sets.erl"},{line,309}]},{sets,fold_segs,4,[{file,"sets.erl"},{line,305}]},{riak_dt
_map,merge,2,[{file,"src/riak_dt_map.erl"},{line,454}]},{riak_kv_crdt,'-merge_value/2-fun-0-',5,[{file,"src/riak_kv_crdt.erl"},{line,204}]},{orddict,update,4,[{file,"or..."},...]}]}}
in riak_kv_put_fsm:waiting_local_vnode/2 line 567
in gen_fsm:terminate/7 line 622
2015-01-20 13:29:00.384 [error] <0.4521.0> gen_fsm <0.4521.0> in state
waiting_local_vnode terminated with reason: no case clause matching
{vnode_error,{{badrecord,dict},[{dict,filter_dict,2,[{file,"dict.erl"},{line,464}]},{riak_dt_
map,'-filter_unique/4-fun-1-',4,[{file,"src/riak_dt_map.erl"},{line,466}]},{sets,fold_bucket,3,[{file,"sets.erl"},{line,313}]},{sets,fold_seg,4,[{file,"sets.erl"},{line,309}]},{sets,fold_segs,4,[{file,"sets.erl"},{line,305}]},{riak_
dt_map,merge,2,[{file,"src/riak_dt_map.erl"},{line,454}]},{riak_kv_crdt,'-merge_value/2-fun-0-',5,[{file,"src/riak_kv_crdt.erl"},{line,204}]},{orddict,update,4,[{file,"or..."},...]}]}}
in riak_kv_put_fsm:waiting_local_vnode/2 line 5
67
2015-01-20 13:29:00.384 [error] <0.4521.0> CRASH REPORT Process <0.4521.0>
with 0 neighbours exited with reason: no case clause matching
{vnode_error,{{badrecord,dict},[{dict,filter_dict,2,[{file,"dict.erl"},{line,464}]},{riak_dt_ma
p,'-filter_unique/4-fun-1-',4,[{file,"src/riak_dt_map.erl"},{line,466}]},{sets,fold_bucket,3,[{file,"sets.erl"},{line,313}]},{sets,fold_seg,4,[{file,"sets.erl"},{line,309}]},{sets,fold_segs,4,[{file,"sets.erl"},{line,3
Re: adding nodes to cluster
Thanks.
On Tue, Jan 20, 2015 at 5:07 PM, Russell Brown wrote:
> FYI https://github.com/basho/riak/issues/667
>
> I’ll get to work on it at once, we might be able to hit the 2.0.5 build
> date at the end of the week.
>
> On 20 Jan 2015, at 14:04, Russell Brown wrote:
>
> > I’ll open a ticket for it. We changed the internal structure of both
> Maps and Sets, and tested that they were backwards compatible, but clearly
> missed something.
> >
> > On 20 Jan 2015, at 13:54, Alexander Popov wrote:
> >
> >> Upgraded riak 2.0.2-> 2.0.4
> >> trying to add nodes to cluster( was single node )
> >> all new nodes located on another host, but multi-nodes installation
> from source
> >> each new node have config settings tuned like
> >>
> >> nodename = riak1@10.0.0.28
> >> platform_data_dir = /var/lib/riak1
> >> listener.http.internal = 0.0.0.0:18098
> >> listener.protobuf.internal = 0.0.0.0:18087
> >> handoff.port = 18099
> >> search.solr.port = 18093
> >> search.solr.jmx_port = 18985
> >> search=on
> >>
> >> after adding this nodes to cluster for long time was nothing was
> transferred for new nodes.
> >> I was down this nodes and force-remove it from cluster.
> >>
> >> After this strange things happens.
> >> On new nodes:
> >> Even after restart cleaning entire data dir and start again, in folder
> yz indexes somehow appears again.
> >>
> >> logs contains only
> >> 2015-01-20 13:15:04.815 [info] <0.352.0>@riak_core:wait_for_service:483
> Wait complete for service riak_kv (10 seconds)
> >> 2015-01-20 13:15:06.660 [info] <0.547.0>@yz_index:local_create:189
> Created index contacts_index with schema contacts
> >> 2015-01-20 13:15:08.683 [info] <0.547.0>@yz_index:local_create:189
> Created index users_index with schema users
> >>
> >>
> >> Main Node now have bunch of errors on start and periodically after:
> >>
> >> 015-01-20 13:29:00.382 [error]
> <0.871.0>@riak_core_vnode:vnode_command:348 riak_kv_vnode command failed
> {{badrecord,dict},[{dict,filter_dict,2,[{file,"dict.erl"},{line,464}]},{riak_dt_map,'-filter_unique/4-fun-1-',4,[{file,"src/ria
> >>
> k_dt_map.erl"},{line,466}]},{sets,fold_bucket,3,[{file,"sets.erl"},{line,313}]},{sets,fold_seg,4,[{file,"sets.erl"},{line,309}]},{sets,fold_segs,4,[{file,"sets.erl"},{line,305}]},{riak_dt_map,merge,2,[{file,"src/riak_dt_map.erl"},{l
> >>
> ine,454}]},{riak_kv_crdt,'-merge_value/2-fun-0-',5,[{file,"src/riak_kv_crdt.erl"},{line,204}]},{orddict,update,4,[{file,"orddict.erl"},{line,170}]}]}
> >> 2015-01-20 13:29:00.382 [error]
> <0.650.0>@riak_core_vnode:vnode_command:348 riak_kv_vnode command failed
> {{badrecord,dict},[{dict,filter_dict,2,[{file,"dict.erl"},{line,464}]},{riak_dt_map,'-filter_unique/4-fun-1-',4,[{file,"src/ria
> >>
> k_dt_map.erl"},{line,466}]},{sets,fold_bucket,3,[{file,"sets.erl"},{line,313}]},{sets,fold_seg,4,[{file,"sets.erl"},{line,309}]},{sets,fold_segs,4,[{file,"sets.erl"},{line,305}]},{riak_dt_map,merge,2,[{file,"src/riak_dt_map.erl"},{l
> >>
> ine,454}]},{riak_kv_crdt,'-merge_value/2-fun-0-',5,[{file,"src/riak_kv_crdt.erl"},{line,204}]},{orddict,update,4,[{file,"orddict.erl"},{line,170}]}]}
> >> 2015-01-20 13:29:00.383 [error] <0.4520.0> gen_fsm <0.4520.0> in state
> waiting_local_vnode terminated with reason: no case clause matching
> {vnode_error,{{badrecord,dict},[{dict,filter_dict,2,[{file,"dict.erl"},{line,464}]},{riak_dt_
> >>
> map,'-filter_unique/4-fun-1-',4,[{file,"src/riak_dt_map.erl"},{line,466}]},{sets,fold_bucket,3,[{file,"sets.erl"},{line,313}]},{sets,fold_seg,4,[{file,"sets.erl"},{line,309}]},{sets,fold_segs,4,[{file,"sets.erl"},{line,305}]},{riak_
> >>
> dt_map,merge,2,[{file,"src/riak_dt_map.erl"},{line,454}]},{riak_kv_crdt,'-merge_value/2-fun-0-',5,[{file,"src/riak_kv_crdt.erl"},{line,204}]},{orddict,update,4,[{file,"or..."},...]}]}}
> in riak_kv_put_fsm:waiting_local_vnode/2 line 5
> >> 67
> >> 2015-01-20 13:29:00.384 [error] <0.4520.0> CRASH REPORT Process
> <0.4520.0> with 0 neighbours exited with reason: no case clause matching
> {vnode_error,{{badrecord,dict},[{dict,filter_dict,2,[{file,"dict.erl"},{line,464}]},{riak_dt_ma
> >>
> p,'-filter_unique/4-fun-1-',4,[{file,&quo
Re: adding nodes to cluster
Actually, problematic node is production. but when shit happens i was
migrate all data ( export/import) to clean setup.
so i can experiment with old node.
It have not source installation, from package.
I will try compile it tomorrow.
On Wed, Jan 21, 2015 at 9:01 PM, Russell Brown wrote:
> There’s a branch of riak_dt here https://github.com/basho/riak_dt/pull/111
>
> Since you’re running against a source install (I assume in development, or
> test?) would you be so kind as to try this branch and see if it solves your
> issue?
>
> Many thanks
>
> Russell
>
> On 20 Jan 2015, at 15:12, Alexander Popov wrote:
>
> > Thanks.
> >
> > On Tue, Jan 20, 2015 at 5:07 PM, Russell Brown
> wrote:
> > FYI https://github.com/basho/riak/issues/667
> >
> > I’ll get to work on it at once, we might be able to hit the 2.0.5 build
> date at the end of the week.
> >
> > On 20 Jan 2015, at 14:04, Russell Brown wrote:
> >
> > > I’ll open a ticket for it. We changed the internal structure of both
> Maps and Sets, and tested that they were backwards compatible, but clearly
> missed something.
> > >
> > > On 20 Jan 2015, at 13:54, Alexander Popov wrote:
> > >
> > >> Upgraded riak 2.0.2-> 2.0.4
> > >> trying to add nodes to cluster( was single node )
> > >> all new nodes located on another host, but multi-nodes installation
> from source
> > >> each new node have config settings tuned like
> > >>
> > >> nodename = riak1@10.0.0.28
> > >> platform_data_dir = /var/lib/riak1
> > >> listener.http.internal = 0.0.0.0:18098
> > >> listener.protobuf.internal = 0.0.0.0:18087
> > >> handoff.port = 18099
> > >> search.solr.port = 18093
> > >> search.solr.jmx_port = 18985
> > >> search=on
> > >>
> > >> after adding this nodes to cluster for long time was nothing was
> transferred for new nodes.
> > >> I was down this nodes and force-remove it from cluster.
> > >>
> > >> After this strange things happens.
> > >> On new nodes:
> > >> Even after restart cleaning entire data dir and start again, in
> folder yz indexes somehow appears again.
> > >>
> > >> logs contains only
> > >> 2015-01-20 13:15:04.815 [info]
> <0.352.0>@riak_core:wait_for_service:483 Wait complete for service riak_kv
> (10 seconds)
> > >> 2015-01-20 13:15:06.660 [info] <0.547.0>@yz_index:local_create:189
> Created index contacts_index with schema contacts
> > >> 2015-01-20 13:15:08.683 [info] <0.547.0>@yz_index:local_create:189
> Created index users_index with schema users
> > >>
> > >>
> > >> Main Node now have bunch of errors on start and periodically after:
> > >>
> > >> 015-01-20 13:29:00.382 [error]
> <0.871.0>@riak_core_vnode:vnode_command:348 riak_kv_vnode command failed
> {{badrecord,dict},[{dict,filter_dict,2,[{file,"dict.erl"},{line,464}]},{riak_dt_map,'-filter_unique/4-fun-1-',4,[{file,"src/ria
> > >>
> k_dt_map.erl"},{line,466}]},{sets,fold_bucket,3,[{file,"sets.erl"},{line,313}]},{sets,fold_seg,4,[{file,"sets.erl"},{line,309}]},{sets,fold_segs,4,[{file,"sets.erl"},{line,305}]},{riak_dt_map,merge,2,[{file,"src/riak_dt_map.erl"},{l
> > >>
> ine,454}]},{riak_kv_crdt,'-merge_value/2-fun-0-',5,[{file,"src/riak_kv_crdt.erl"},{line,204}]},{orddict,update,4,[{file,"orddict.erl"},{line,170}]}]}
> > >> 2015-01-20 13:29:00.382 [error]
> <0.650.0>@riak_core_vnode:vnode_command:348 riak_kv_vnode command failed
> {{badrecord,dict},[{dict,filter_dict,2,[{file,"dict.erl"},{line,464}]},{riak_dt_map,'-filter_unique/4-fun-1-',4,[{file,"src/ria
> > >>
> k_dt_map.erl"},{line,466}]},{sets,fold_bucket,3,[{file,"sets.erl"},{line,313}]},{sets,fold_seg,4,[{file,"sets.erl"},{line,309}]},{sets,fold_segs,4,[{file,"sets.erl"},{line,305}]},{riak_dt_map,merge,2,[{file,"src/riak_dt_map.erl"},{l
> > >>
> ine,454}]},{riak_kv_crdt,'-merge_value/2-fun-0-',5,[{file,"src/riak_kv_crdt.erl"},{line,204}]},{orddict,update,4,[{file,"orddict.erl"},{line,170}]}]}
> > >> 2015-01-20 13:29:00.383 [error] <0.4520.0> gen_fsm <0.4520.0> in
> state waiting_local_vnode terminated with reason: no case clause matching
> {vnode_error,{{badrecord,dict},[{dict,filter_dict,2,[{file,"dict.erl"},{line,464}]},{riak_dt_
> > >>
> map,'-filt
2.0.5 compiled from source SOLR failed to start
>
> 2015-02-27 10:38:27.960 [info] <0.563.0>@yz_solr_proc:ensure_data_dir:219
> No solr config found, creating a new one
> 2015-02-27 10:38:27.961 [info] <0.563.0>@yz_solr_proc:init:96 Starting
> solr: "/usr/bin/java"
> ["-Djava.awt.headless=true","-Djetty.home=/var/riak-dist/riak1/bin/../lib/yokozuna-2.0.0-34-g122659d/priv/solr","
>
> -Djetty.port=18093","-Dsolr.solr.home=/var/lib/riak1/yz","-DhostContext=/internal_solr","-cp","/var/riak-dist/riak1/bin/../lib/yokozuna-2.0.0-34-g122659d/priv/solr/start.jar","-Dlog4j.configuration=file:///
>
> var/riak-dist/riak1/etc/solr-log4j.properties","-Dyz.lib.dir=/var/riak-dist/riak1/bin/../lib/yokozuna-2.0.0-34-g122659d/priv/java_lib","-d64","-Xms1g","-Xmx3g","-XX:+UseStringCache","-XX:+UseCompressedOops"
>
> ,"-Dcom.sun.management.jmxremote.port=18985","-Dcom.sun.management.jmxremote.authenticate=false","-Dcom.sun.management.jmxremote.ssl=false","org.eclipse.jetty.start.Main"]
> 2015-02-27 10:38:28.004 [info] <0.7.0> Application yokozuna started on
> node 'riak1@10.0.0.133'
> 2015-02-27 10:38:28.007 [info] <0.7.0> Application cluster_info started on
> node 'riak1@10.0.0.133'
> 2015-02-27 10:38:28.033 [info]
> <0.198.0>@riak_core_capability:process_capability_changes:555 New
> capability: {riak_control,member_info_version} = v1
> 2015-02-27 10:38:28.035 [info] <0.7.0> Application riak_control started on
> node 'riak1@10.0.0.133'
> 2015-02-27 10:38:28.035 [info] <0.7.0> Application erlydtl started on node
> 'riak1@10.0.0.133'
> 2015-02-27 10:38:28.043 [info] <0.7.0> Application riak_auth_mods started
> on node 'riak1@10.0.0.133'
> 2015-02-27 10:38:28.307 [info] <0.563.0>@yz_solr_proc:handle_info:135 solr
> stdout/err: java.io.FileNotFoundException: No XML configuration files
> specified in start.config or command line.
>
> 2015-02-27 10:38:28.308 [info] <0.563.0>@yz_solr_proc:handle_info:135 solr
> stdout/err: at org.eclipse.jetty.start.Main.start(Main.java:502)
> at org.eclipse.jetty.start.Main.main(Main.java:96)
>
> Usage: java -jar start.jar [options] [properties] [configs]
>java -jar start.jar --help # for more information
> 2015-02-27 10:38:28.625 [error] <0.563.0> gen_server yz_solr_proc
> terminated with reason: {"solr OS process exited",251}
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Still getting peaks on solr queries
I not have stable reproduce but peaks happens several times in day and takes 10-50 sec. some times it happens on very low traffic, so I think it not related on traffic. Looks like query is locked somehow. What can I do for troubleshoot peaks? Enable addition logs or Monitoring ? There is no errors in logs at time of the peaks ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Solr indexes becomes broken after node replacement
I was perform all nodes upgrade in x6 nodes cluster and at same time 4 of them was prooceed with node name changed by steps: http://docs.basho.com/riak/latest/ops/running/recovery/failed-node/#Node-Name-Changed after that search indexes becomes completely broken - many data missed, returns different results on every requests. Does it have any options now except re-save entire data in bucket to fix it? ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Solr indexes becomes broken after node replacement
Does it require remove yz_anti_entropy directory on all nodes at same time? Does it require to restart nodes? On Mon, Jun 8, 2015 at 4:01 PM, Zeeshan Lakhani wrote: > Hello Alexander, > > Can you send us the logs? I’d like to see the errors that occurred after > the time of the node name change. > > In terms of getting your data back into the indexes, I’d recommend letting > AAE handle the situation by removing the `yz_anti_entropy` directory and > letting the keys re-index. You can even remove the search index > directories, e.g. > https://github.com/basho/riak_test/blob/master/tests/yz_core_properties_create_unload.erl#L94, > and > let the them get recreated, which will still re-index via AAE. > > Nonetheless, we’d like to see the logs and try and reproduce the issue in > a test when the node name changes. > > Thanks. > > Zeeshan Lakhani > programmer | > software engineer at @basho > > On Jun 8, 2015, at 8:30 AM, Alexander Popov wrote: > > I was perform all nodes upgrade in x6 nodes cluster > and at same time 4 of them was prooceed with node name changed by steps: > > http://docs.basho.com/riak/latest/ops/running/recovery/failed-node/#Node-Name-Changed > > after that search indexes becomes completely broken - many data missed, > returns different results on every requests. > > Does it have any options now except re-save entire data in bucket to fix > it? > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
How you are manage your clusters on Amazon aws
does it any reasonable alternatives to chef/ops works? Use cases: easy to scale up/scale down Easy to manage - run commands, apply configuration to all nodes, restart nodes, buck up. Monitor cluster, audit all actions, collect logs ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
riak restart does not restart yokozuka well
i'm doing riak restart
and got 'ok' in answer, but node appears in shutdown state.
in proccess list i found that solr is still running, and in logs found :
2015-09-14 09:21:03.939 [info] <0.579.0>@yz_solr_proc:init:96 Starting
solr: "/usr/bin/java"
["-Djava.awt.headless=true","-Djetty.home=/usr/lib/riak/lib/yokozuna-2.1.0-0-gcb41c27/priv/solr","-Djetty.port=80
2015-09-14 09:21:04.006 [warning] <0.6.0> lager_error_logger_h dropped 6
messages in the last second that exceeded the limit of 100 messages/sec
2015-09-14 09:21:04.006 [info] <0.7.0> Application yokozuna started on node
'riak@ip-10-0-0-147.ec2.internal'
2015-09-14 09:21:04.010 [info] <0.7.0> Application cluster_info started on
node 'riak@ip-10-0-0-147.ec2.internal'
2015-09-14 09:21:04.047 [info]
<0.209.0>@riak_core_capability:process_capability_changes:555 New
capability: {riak_control,member_info_version} = v1
2015-09-14 09:21:04.051 [info] <0.7.0> Application riak_control started on
node 'riak@ip-10-0-0-147.ec2.internal'
2015-09-14 09:21:04.051 [info] <0.7.0> Application erlydtl started on node
'riak@ip-10-0-0-147.ec2.internal'
2015-09-14 09:21:04.062 [info] <0.7.0> Application riak_auth_mods started
on node 'riak@ip-10-0-0-147.ec2.internal'
2015-09-14 09:21:04.156 [info] <0.579.0>@yz_solr_proc:handle_info:135 solr
stdout/err: Error: Exception thrown by the agent.
2015-09-14 09:21:04.156 [info] <0.579.0>@yz_solr_proc:handle_info:135 solr
stdout/err: : java.rmi.server.ExportException: Port already in use: 8985;
nested exception is:.
<-->java.net.BindException: Address already in use
2015-09-14 09:21:04.476 [error] <0.579.0> gen_server yz_solr_proc
terminated with reason: {"solr OS process exited",1}
2015-09-14 09:21:04.476 [error] <0.579.0> CRASH REPORT Process yz_solr_proc
with 0 neighbours exited with reason: {"solr OS process exited",1} in
gen_server:terminate/6 line 744
2015-09-14 09:21:04.476 [error] <0.578.0> Supervisor yz_solr_sup had child
yz_solr_proc started with yz_solr_proc:start_link("/var/lib/riak/yz", 8093,
8985) at <0.579.0> exit with reason {"solr OS process e
2015-09-14 09:21:04.477 [info] <0.622.0>@yz_solr_proc:init:96 Starting
solr: "/usr/bin/java"
["-Djava.awt.headless=true","-Djetty.home=/usr/lib/riak/lib/yokozuna-2.1.0-0-gcb41c27/priv/solr","-Djetty.port=80
2015-09-14 09:21:04.710 [info] <0.622.0>@yz_solr_proc:handle_info:135 solr
stdout/err: Error: Exception thrown by the agent.
2015-09-14 09:21:04.710 [info] <0.622.0>@yz_solr_proc:handle_info:135 solr
stdout/err: : java.rmi.server.ExportException: Port already in use: 8985;
nested exception is:.
<-->java.net.BindException: Address already in use
2015-09-14 09:21:05.029 [error] <0.622.0> gen_server yz_solr_proc
terminated with reason: {"solr OS process exited",1}
2015-09-14 09:21:05.029 [error] <0.622.0> CRASH REPORT Process yz_solr_proc
with 0 neighbours exited with reason: {"solr OS process exited",1} in
gen_server:terminate/6 line 744
2015-09-14 09:21:05.030 [error] <0.578.0> Supervisor yz_solr_sup had child
yz_solr_proc started with yz_solr_proc:start_link("/var/lib/riak/yz", 8093,
8985) at <0.622.0> exit with reason {"solr OS process e
2015-09-14 09:21:05.030 [error] <0.578.0> Supervisor yz_solr_sup had child
yz_solr_proc started with yz_solr_proc:start_link("/var/lib/riak/yz", 8093,
8985) at <0.622.0> exit with reason reached_max_restart
2015-09-14 09:21:05.030 [error] <0.577.0> Supervisor yz_sup had child
yz_solr_sup started with yz_solr_sup:start_link() at <0.578.0> exit with
reason shutdown in context child_terminated
2015-09-14 09:21:05.030 [error] <0.577.0> Supervisor yz_sup had child
yz_solr_sup started with yz_solr_sup:start_link() at <0.578.0> exit with
reason reached_max_restart_intensity in context shutdown
2015-09-14 09:21:05.031 [info] <0.7.0> Application yokozuna exited with
reason: shutdown
2015-09-14 09:21:05.033 [info] <0.340.0>@riak_kv_app:prep_stop:232 Stopping
application riak_kv - marked service down.
2015-09-14 09:21:05.034 [info] <0.340.0>@riak_kv_app:prep_stop:236
Unregistered pb services
2015-09-14 09:21:05.034 [info] <0.340.0>@riak_kv_app:prep_stop:241
unregistered webmachine routes
2015-09-14 09:21:05.037 [info] <0.340.0>@riak_kv_app:prep_stop:243 all
active put FSMs completed
2015-09-14 09:21:05.038 [info] <0.415.0>@riak_kv_js_vm:terminate:237
Spidermonkey VM (pool: riak_kv_js_hook) host stopping (<0.415.0>)
2015-09-14 09:21:05.038 [info] <0.416.0>@riak_kv_js_vm:terminate:237
Spidermonkey VM (pool: riak_kv_js_hook) host stopping (<0.416.0>)
2015-09-14 09:21:05.038 [info] <0.408.0>@riak_kv_js_vm:terminate:237
Spidermonkey VM (pool: riak_kv_js_reduce) host stopping (<0.408.0>)
2015-09-14 09:21:05.038 [info] <0.410.0>@riak_kv_js_vm:terminate:237
Spidermonkey VM (pool: riak_kv_js_reduce) host stopping (<0.410.0>)
2015-09-14 09:21:05.038 [info] <0.409.0>@riak_kv_js_vm:terminate:237
Spidermonkey VM (pool: riak_kv_js_reduce) host stopping (<0.409.0>)
2015-09-14 09:21:05.038 [info] <0.412.0>@riak_kv_js_vm:terminate:237
Spidermonkey VM (p
Re: riak restart does not restart yokozuka well
Yes there is plenty of errors there like
Committed before 500 {msg=GC overhead limit
exceeded,trace=java.lang.OutOfMemoryError: GC overhead limit exceeded
null:org.eclipse.jetty.io.EofException
and so on, this is reason why I try to restart node
My concerns is:
* search on this node come to un-working state and not repaired itself
* halted node, requires manual actions
* false positive report or* riak restart *
On Mon, Sep 14, 2015 at 6:22 PM, Fred Dushin wrote:
> Check the solr logs to see why it failed to shut down. If necessary, find
> the pid bound to port 8985 and kill it.
>
> -Fred
>
> On Sep 14, 2015, at 5:28 AM, Alexander Popov wrote:
>
> i'm doing riak restart
> and got 'ok' in answer, but node appears in shutdown state.
>
> in proccess list i found that solr is still running, and in logs found :
>
> 2015-09-14 09:21:03.939 [info] <0.579.0>@yz_solr_proc:init:96 Starting
> solr: "/usr/bin/java"
> ["-Djava.awt.headless=true","-Djetty.home=/usr/lib/riak/lib/yokozuna-2.1.0-0-gcb41c27/priv/solr","-Djetty.port=80
> 2015-09-14 09:21:04.006 [warning] <0.6.0> lager_error_logger_h dropped 6
> messages in the last second that exceeded the limit of 100 messages/sec
> 2015-09-14 09:21:04.006 [info] <0.7.0> Application yokozuna started on
> node 'riak@ip-10-0-0-147.ec2.internal'
> 2015-09-14 09:21:04.010 [info] <0.7.0> Application cluster_info started on
> node 'riak@ip-10-0-0-147.ec2.internal'
> 2015-09-14 09:21:04.047 [info]
> <0.209.0>@riak_core_capability:process_capability_changes:555 New
> capability: {riak_control,member_info_version} = v1
> 2015-09-14 09:21:04.051 [info] <0.7.0> Application riak_control started on
> node 'riak@ip-10-0-0-147.ec2.internal'
> 2015-09-14 09:21:04.051 [info] <0.7.0> Application erlydtl started on node
> 'riak@ip-10-0-0-147.ec2.internal'
> 2015-09-14 09:21:04.062 [info] <0.7.0> Application riak_auth_mods started
> on node 'riak@ip-10-0-0-147.ec2.internal'
> 2015-09-14 09:21:04.156 [info] <0.579.0>@yz_solr_proc:handle_info:135 solr
> stdout/err: Error: Exception thrown by the agent.
> 2015-09-14 09:21:04.156 [info] <0.579.0>@yz_solr_proc:handle_info:135 solr
> stdout/err: : java.rmi.server.ExportException: Port already in use: 8985;
> nested exception is:.
> <-->java.net.BindException: Address already in use
>
> 2015-09-14 09:21:04.476 [error] <0.579.0> gen_server yz_solr_proc
> terminated with reason: {"solr OS process exited",1}
> 2015-09-14 09:21:04.476 [error] <0.579.0> CRASH REPORT Process
> yz_solr_proc with 0 neighbours exited with reason: {"solr OS process
> exited",1} in gen_server:terminate/6 line 744
> 2015-09-14 09:21:04.476 [error] <0.578.0> Supervisor yz_solr_sup had child
> yz_solr_proc started with yz_solr_proc:start_link("/var/lib/riak/yz", 8093,
> 8985) at <0.579.0> exit with reason {"solr OS process e
> 2015-09-14 09:21:04.477 [info] <0.622.0>@yz_solr_proc:init:96 Starting
> solr: "/usr/bin/java"
> ["-Djava.awt.headless=true","-Djetty.home=/usr/lib/riak/lib/yokozuna-2.1.0-0-gcb41c27/priv/solr","-Djetty.port=80
> 2015-09-14 09:21:04.710 [info] <0.622.0>@yz_solr_proc:handle_info:135 solr
> stdout/err: Error: Exception thrown by the agent.
> 2015-09-14 09:21:04.710 [info] <0.622.0>@yz_solr_proc:handle_info:135 solr
> stdout/err: : java.rmi.server.ExportException: Port already in use: 8985;
> nested exception is:.
> <-->java.net.BindException: Address already in use
>
> 2015-09-14 09:21:05.029 [error] <0.622.0> gen_server yz_solr_proc
> terminated with reason: {"solr OS process exited",1}
> 2015-09-14 09:21:05.029 [error] <0.622.0> CRASH REPORT Process
> yz_solr_proc with 0 neighbours exited with reason: {"solr OS process
> exited",1} in gen_server:terminate/6 line 744
> 2015-09-14 09:21:05.030 [error] <0.578.0> Supervisor yz_solr_sup had child
> yz_solr_proc started with yz_solr_proc:start_link("/var/lib/riak/yz", 8093,
> 8985) at <0.622.0> exit with reason {"solr OS process e
> 2015-09-14 09:21:05.030 [error] <0.578.0> Supervisor yz_solr_sup had child
> yz_solr_proc started with yz_solr_proc:start_link("/var/lib/riak/yz", 8093,
> 8985) at <0.622.0> exit with reason reached_max_restart
> 2015-09-14 09:21:05.030 [error] <0.577.0> Supervisor yz_sup had child
> yz_solr_sup started with yz_solr_sup:start_link() at <0.578.0> exit with
> reason shutdown in context child_terminated
> 2015-09-14 09:21:05.030 [error] <0.577.0> Supervisor yz_sup had ch
limit of bucket/solr indexes
1. does Riak have restrictions on bucket numbers? 2. same for SOLR indexes? Interesting not theoretical, but practical limits when it start works bad. Use case: we want split users bucket to bucket-per-team model, this bucket have also solr index on top. Can it work for 1000 teams? 10k? 100k? ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
AWS instance types
What to choose C4 or M4 ? M4 benefits: * twice more memory C4 benefits: * can be launched in one placement group, that potential can improve nodes communications performance ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
2i indexes and keys request inconsistencies
Noticied that sometimes 2i query and all keys requesrs returns extra records ~2% of all records. When call this items by get request after, it returns 404 and after that key stops to returns in 2i and keys requests. Does it normally or my database is corrupted? ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: 2i indexes and keys request inconsistencies
@Matthew No, db activity was very low at this time, and keys returned by this queries was deleted long time ago ( some of them at Dec 2015 ) I got this issue when proceed maintenance task which touch all keys in DB, list all keys by /keys?keys=true query, read, upgrade, save. We have some logic depends on 2i indexes results - ( count number of related keys ). But if it returns phantom keys, I cannot trust this data, need to double check by getting each object, but this is more expensive operation On Tue, Mar 8, 2016 at 10:21 PM, Matthew Von-Maszewski wrote: > Is the database being actively modified during your queries? > > Queries can lock down a "snapshot" within leveldb. The query operation > can return keys that existed at the time of the snapshot, but have been > subsequently deleted by normal operations. > > In such a case, the query is correct in giving you the key and the 404 > afterward is also correct. They represent two different versions of the > database over time. > > Not sure if this is a valid scenario for you or not. > > Matthew > > > On Mar 8, 2016, at 1:22 PM, Alexander Popov wrote: > > Noticied that sometimes 2i query and all keys requesrs returns extra > records ~2% of all records. > > When call this items by get request after, it returns 404 and after that > key stops to returns in 2i and keys requests. > > Does it normally or my database is corrupted? > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Solr http endpoint and POST
Does it possible? if yes - which encoding? form-data or multipart? if no - does it possible to enlarge GET query length? (looks like maximum is around 6500) ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Update solr schema
After release riak 2.2 does it any new possibilities to update search schema without re-index all data? ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
2.2 CPU usage
Did Upgrade production cluster to 2.2.0 looks that it eat more CPU, in idle time 10-30% always( on 2.1.4 was up to 10% with same load ) in Peaks it takes up to 100% ( on 2.1.4 was peaks was up to 75% ) 2.2: [image: Inline image 1] [image: Inline image 2] 2.1.4: [image: Inline image 4] [image: Inline image 3] ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
