I've see this behaviour as well. The only thing I've found to be reliable is to
actually try to read the objects as you describe below and discard the keys if
notfound in application logic. This obviously sucks if your trying to interact
with the key index as a light weight way to count items. I
be less common under Riak 1.3,
> although ownership handoff will still exacerbate the problem.
>
> [1] https://github.com/basho/riak_kv/issues/358
>
> -John Daily
> Technical Evangelist
> jda...@basho.com
>
>
> On Mar 6, 2013, at 3:14 PM, Jeremy Raymond wrote:
>
&g
, 2013 at 11:28 AM, Jeremy Raymond wrote:
> Yesterday I added a new node to my cluster. During the time when ownership
> handoff was happening (several hours of work) mapreduce map functions were
> receiving {error, notfound} as inputs. My Erlang mapred functions weren't
> design
Yesterday I added a new node to my cluster. During the time when ownership
handoff was happening (several hours of work) mapreduce map functions were
receiving {error, notfound} as inputs. My Erlang mapred functions weren't
designed to handle this. They hadn't encountered this before during normal
index the items I wished to be searchable after re-enabling search on the
buckets.
--
Jeremy
On Fri, Nov 2, 2012 at 1:09 PM, Ryan Zezeski wrote:
> Jeremy,
>
> On Fri, Nov 2, 2012 at 12:31 PM, Jeremy Raymond wrote:
>
>> I cycled through the compaction on another nod
gh (and then re-enabling) before re-indexing?
--
Jeremy
On Fri, Nov 2, 2012 at 11:19 AM, Ryan Zezeski wrote:
> "Active" is any segment file that has the suffix .data.
>
> [Sent from my iPhone]
>
> On Nov 2, 2012, at 11:11 AM, Jeremy Raymond wrote:
>
> When do se
When do segments become active/inactive?
--
Jeremy
On Fri, Nov 2, 2012 at 10:50 AM, Ryan Zezeski wrote:
>
>
> On Fri, Nov 2, 2012 at 9:52 AM, Jeremy Raymond wrote:
>
>> Some files changed and some didn't.
>> Not really sure how to interpret the differences.
>
on decided that no
compaction was needed does that mean that all the tombstones are gone?
There is no more disk space that could possibly be freed?
--
Jeremy
On Fri, Nov 2, 2012 at 9:52 AM, Jeremy Raymond wrote:
> Here is the new listing of the segments:
> https://gist.github.com/400
each time check for changes in the segment file sizes.
>
>
> -Z
>
>
> On Thu, Nov 1, 2012 at 11:25 AM, Jeremy Raymond wrote:
>>
>> I reindexed a bunch of items that are still in the search index but no
>> disk space was reclaimed. Is there any Riak console Erlang v
I reindexed a bunch of items that are still in the search index but no
disk space was reclaimed. Is there any Riak console Erlang voodoo I
can do to convince Riak Search that now would be a good time to
compact the merge_index?
--
Jeremy
On Tue, Oct 30, 2012 at 4:26 PM, Jeremy Raymond wrote
I've posted the list of buffer files [1] and segment files [2].
The current data set I have in Riak is static, so no new items are
being written. So this looks like the reason as to why compaction
isn't happening since there is no time based trigger on the merge
index. To get compaction to kick in
So the only way to actually free the disk space consumed by the
tombstones in the search index is to bring down the cluster and blow
away the merge index (at /var/lib/riak/merge_index)?
--
Jeremy
On Fri, Oct 26, 2012 at 9:14 AM, Jeremy Raymond wrote:
> Yes I've read about the tombston
d was
> also surprised about that fact that the data grows instead of reduce.
>
> Cheers
> Vladimir
>
> On Fri, Oct 26, 2012 at 2:26 PM, Jeremy Raymond wrote:
>>
>> Hello,
>>
>> I had Riak search enabled on a bucket containing millions of items. I
>> no l
Hello,
I had Riak search enabled on a bucket containing millions of items. I
no longer need these items to be searchable so I uninstalled search on
the bucket via search-cmd. I'm looking to free the space consumed by
the search index for this bucket. Following a previous post [1] on
this list I'm
It's also worth noting that if a field is not specified in a query the
> default field is assumed.
>
> Does this answer your question?
>
> On Fri, Jun 15, 2012 at 10:26 AM, Jeremy Raymond wrote:
>> Hello,
>>
>> I'm creating a custom search extractor and a
Hi,
I was testing my application against the latest Riak source from
github (168c2ae67c) and noticed that I'm getting a different response
when putting pre-existing objects into Riak with the if_none_match
option. The response to riakc_pb_socket:put(Pid, RiakObj,
[if_none_match]) was {error, <<"ma
Hello,
I'm creating a custom search extractor and am not entirely clear on
what is the purpose of the default field parameter. The docs[1] are
out of date and describe an extractor function with arity 2 rather
than 3. Can someone give me a quick description of the purpose and
usage for default fie
+1 for better online docs (wiki). I also don't read them anymore and
also go to the source to figure out what to do or ask on the list.
Also using the Erlang client.
--
Jeremy
On Apr 19, 2012, at 11:31 PM, Daniil Churikov wrote:
> I wonder why you all ask a book, when there is no any docs on mo
Hi,
I need to reindex a bucket with a ~4 million items. If I do a
streaming list keys using the Erlang client and then read/write the
items as they keys come in it puts too much load on the cluster and
other mapred queries that get run timeout. I already have a date based
index on the items and wa
4031556999544297163190906134303066185487351808/001060.sst:
> No such file or directory
>
> -
>
>
> see that pattern over and over again in the file.
>
>
> ~ John
>
>
> - Original Message -
> From: "David Smith"
> To: "John Loe
interface doing the same thing as the
Erlang client and just doing an index query fed into reduce_identity?
--
Jeremy
From: Ian Plosker [mailto:i...@basho.com]
Sent: Wednesday, January 25, 2012 5:55 PM
To: Jeremy Raymond
Cc: riak-users@lists.basho.com
Subject: Re: Fastest way to count
What is the fastest way to count the number of items in a secondary
index range query? I'm currently feeding an index input directly into
the riak_kv_mapreduce:reduce_count_inputs/2. Anything faster than
doing this? Anyway to query an index without loading the objects
themselves?
--
Jeremy
__
ntation on the Riak
> side, so that would probably be the preferred method.
>
>
> On Fri, Jan 20, 2012 at 11:14 AM, Jeremy Raymond
> wrote:
>>
>> I assume the preferred type to be binaries?
>>
>> --
>> Jeremy
>>
>>
>>
>> On Fri, Jan
t/issues. We will be working to
> resolve inconsistencies like these in the next dev cycle, across all our
> supported clients.
>
> On Fri, Jan 20, 2012 at 9:51 AM, Jeremy Raymond wrote:
>>
>> The Erlang PB client gives and takes secondary index information as
>>
The Erlang PB client gives and takes secondary index information as
binaries in some places and strings in others. For example
riakc_pb_socket:get_index/4 wants the Index and Key as binaries.
Retrieving the index information from the meta data of an object read
out of Riak gives the Index and Key a
On Tue, Jan 10, 2012 at 3:16 PM, Bogunov wrote:
>
> Hi, Jeremy.
>
> Had a problem with this myself (while running few nodes on one machine for
> tests, was having problems with write permissions), actually its hardcoded in
> some module so - no.
>
> On Tue, Jan 10, 201
Ah, OK thanks. I misread in the Riak Control README of the availability of
the self signed certs in Riak 1.0.2 and later as an indication Riak Control
was available in 1.0.2.
--
Jeremy
On Tue, Jan 10, 2012 at 2:56 PM, Andrew Thompson wrote:
> Riak control will be part of the 1.1 series, it is
Hi,
I was following the instructions here (
https://github.com/basho/riak_control ) to enable Riak Control. After
enabling https/ssl and changing the https port (to ) I get a 404 when
navigating to https://host:/admin. Would Riak Control be installed when
upgrading a cluster or only on fre
Is there a way to change riak's temp folder location from /tmp/riak to
something else? I didn't see an option for this in the app.config or
elsewhere.
--
Jeremy
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinf
ere anything that is different from the defaults?
>
> Best,
> Rusty
>
> On Thu, Dec 22, 2011 at 9:18 AM, Jeremy Raymond wrote:
>
>> I ran the queries against the production cluster and could not reproduce
>> the issue (*whew*). It gave me consistent correct results for
the
riak-admin backup/restore commands do more than read all the objects out of
riak, serialize them to disk, and then write them back via normal put
operations when restoring?
--
Jeremy
On Thu, Dec 22, 2011 at 8:24 AM, Jeremy Raymond wrote:
> The output from the commands all look normal (j
p us
> immensely while troubleshooting. (Feel free to email the link to me off
> list.)
>
> Best,
> Rusty
>
> On Wed, Dec 21, 2011 at 12:32 PM, Jeremy Raymond wrote:
>
>> I'm seeing an issue where running the same index query multiple times
>> against an un
011 at 21:04, Ciprian Dorin Craciun
> wrote:
> > On Thu, Dec 15, 2011 at 17:42, Jeremy Raymond
> wrote:
> >> Hello,
> >>
> >> Are calls to the client riakc_pb_socket server blocking if I'm reusing
> the
> >> same Pid across multiple threads? Th
I'm seeing an issue where running the same index query multiple times
against an unchanging set of data returns a differing numbers of items.
Riak error and crash logs are empty.
I've seen this on my dev machine (haven't noticed it in production, but it
could be happening). The production cluster
Hi John,
I'm curious if you ever figured out what was going on?
--
Jeremy
On Fri, Dec 9, 2011 at 2:53 PM, John Loehrer wrote:
> I am currently evaluating riak. I'd like to be able to do periodic
> snapshots of /var/lib/riak using LVM without stopping the node. According
> to a response on this
the exception to this rule, it splits buckets into separate
> files.)
>
> Best,
> Rusty
>
> On Thu, Dec 15, 2011 at 8:29 AM, Jeremy Raymond wrote:
>
>> Let me pose the question a different way. What is a good way to pull out
>> objects across multiple buckets based up
Hello,
Are calls to the client riakc_pb_socket server blocking if I'm reusing the
same Pid across multiple threads? The pb_socket server will only process
one request at a time?
--
Jeremy
___
riak-users mailing list
riak-users@lists.basho.com
http://lis
indexes to
pull the find them?
If the keys of my objects are unique does organizing them into buckets
really gain me anything over just putting them all in a single bucket and
using secondary indexes to find them?
--
Jeremy
On Sat, Dec 10, 2011 at 2:05 PM, Jeremy Raymond wrote:
> He
Would an integer based secondary index be faster than a binary secondary
index for exact matching lookups?
--
Jeremy
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Hello,
Is it possible to run an index query across multiple buckets at once?
Something similar to how you can feed a map query with bucket/key pairs,
I'm looking to run an index query with bucket/index pairs.
--
Jeremy
___
riak-users mailing list
riak-u
So if I do have a bucket that I regularly want a listing of the keys for (or
really that I want to access all of the objects) what is the recommend
procedure? Put the same 2i index on each object in the bucket and then do an
index query on it? Is doing a mapred operation specifying a bucket as the
ut it back up(riak is really good at fault tolerance, you simply don't
> notice it).
>
> On Fri, Nov 18, 2011 at 3:55 PM, Jeremy Raymond wrote:
> Something else I tried to give the cluster more time to settle was to wait
> until riak-admin transfers reported no pending trans
up(riak is really good at
fault tolerance, you simply don't notice it).
On Fri, Nov 18, 2011 at 3:55 PM, Jeremy Raymond wrote:
> Something else I tried to give the cluster more time to settle was to wait
> until riak-admin transfers reported no pending transfers between updating
> no
f time for
pending transfers to complete?
--
Jeremy
On Fri, Nov 18, 2011 at 6:48 AM, Jeremy Raymond wrote:
> Hello,
>
> I'll setup my deploy script to capture this information and send you the
> info off-list (probably sometime next week).
>
> --
> Jeremy
>
> On
please contact me off list.
>
> BR, Jon
>
> On Tue, Nov 15, 2011 at 6:48 AM, Jeremy Raymond wrote:
> I'm using Riak 1.0.1 and I have a script that deploys updates to each of my 3
> nodes to update the Erlang mapred modules. What I do is stop a node, deploy
> the new
No had this happening a few weeks ago. I just upped the file hand
limits on the OS and the problem went away.
--
Jeremy
On Nov 16, 2011, at 8:50 AM, David Smith wrote:
> On Tue, Nov 15, 2011 at 6:01 PM, Jeremy Raymond wrote:
>> I've seen issues when leveldb runs out of fil
I've seen issues when leveldb runs out of file handles. The leveldb
log then fills with error messages.
--
Jeremy
On Nov 15, 2011, at 6:25 PM, David Smith wrote:
> On Mon, Oct 24, 2011 at 5:29 PM, Elias Levy
> wrote:
>
>> This morning, while performing a query against one of the nodes, Riak
>
I'm using Riak 1.0.1 and I have a script that deploys updates to each of my
3 nodes to update the Erlang mapred modules. What I do is stop a node,
deploy the new mapred modues, restart the node, wait for the riak_kv
service to start, then move onto the next node. Sometimes when I do this
one of the
wrote:
Jeremy,
Yes, the same procedure should be fine with the eleveldb data
directories.
Kelly
On Oct 15, 2011 6:27 AM, "Jeremy Raymond" <mailto:jeraym...@gmail.com>> wrote:
Hello,
I recall with using the bitcask backend you could backup the
cluster data si
Hello,
I recall with using the bitcask backend you could backup the cluster data
simply by copying the data directories of each node. You could then restore by
copy the data back to the data folder should something bad happen. Does this
also apply to the leveldb backend?
Thanks,
Jeremy
__
exceeds any of the thresholds it will be included in the
> merge process. The small_file_threshold means that any inactive data file
> that is less than 10MB will be included in the merge process.
>
> Thanks,
> Dan
>
>
> Daniel Reverri
> Developer Advocate
> Basho Technolog
In /var/lib/riak/bitcask there are a bunch of data directories whose name
are a string of numbers. On once node I noticed I have a directory whose
name is 0. This seemed out of place as the directory names are typically a
large string of numbers. Is this 0 directory normal?
- Jeremy
__
ed-when-a-riak-node-is-restarted
>
> Thanks,
> Dan
>
> Daniel Reverri
> Developer Advocate
> Basho Technologies, Inc.
> d...@basho.com
>
>
>
> On Wed, Sep 14, 2011 at 8:49 AM, Jeremy Raymond wrote:
>
>> If I'm reading the docs correctly, only files
If I'm reading the docs correctly, only files smaller
than small_file_threshold will be included in a merge. So
if small_file_threshold must be bigger than max_file_size for a merge to
happen?
- Jeremy
On Wed, Sep 14, 2011 at 10:23 AM, Jeremy Raymond wrote:
> Maybe I just need to t
> I would think that the InnoDB backend would be a better backend for the use
> case you're describing.
> ---
> Jeremiah Peschka - Founder, Brent Ozar PLF, LLC
> Microsoft SQL Server MVP
>
> On Sep 14, 2011, at 8:09 AM, Jeremy Raymond wrote:
>
> > Hi,
> >
>
Hi,
I store data in Riak whose keys constantly get overwritten with new data.
I'm currently using Bitcask as the back-end and recently noticed the Bitcask
data folder grow to 24GB. After restarting the nodes, which I think
triggered Bitcask merge, the data went down to 96MB. Today the data dirs ar
2, 2011 at 1:43 PM, Daniel Reverri wrote:
> Hi Jeremy
>
> Are you doing a full bucket query or retrieving individual objects?
>
> Is your bucket configured to allow siblings (allow_mult=true)? Do the
> objects being retrieved have many siblings?
>
> Are you using multiple back
Hello,
I have a bucket with about 70 items in it the Keys look like this
<<"1294549200_UbEu1topckAFu2UYy3spr8AG5lc">> and the values are about 2.8K
of JSON. For some reason operations on data in this bucket are extremely
slow. It takes between 5 - 10 seconds to read a value of of the bucket by
it'
Sent from my rotary phone.
>
> On Aug 3, 2011 12:49 PM, "Jeremy Raymond" wrote:
>> So just manually restart the nodes? Is this what most people do?
>>
>> - Jeremy
>>
>> On 2011-08-03 12:48 PM, "Jeremy Raymond" wrote:
>>> So just man
So just manually restart the nodes? Is this what most people do?
- Jeremy
On 2011-08-03 12:48 PM, "Jeremy Raymond" wrote:
> So just manually restart the nod
>
> - Jeremy
>
> On 2011-08-03 9:29 AM, "David Smith" wrote:
>> Generally, I would adv
riak scripts was my attempt (way back
> when) to ensure that the generic Erlang scripts provided access to as
> much functionality as possible. In production usage, however, I advise
> against using it.
>
> D.
>
> On Tue, Aug 2, 2011 at 9:48 AM, Jeremy Raymond
wrote:
>> H
Hello,
I occasionally have a Riak node that goes down. Is recommended (or not) to
enable the heartbeat restart of the node in /etc/riak/vm.args? Anyone
have experience with this enabled for Riak?
- Jeremy
___
riak-users mailing list
riak-users@lists.bas
I increased the mapreduce timeout to 10 minutes and the system has been
running for about a day and a half with no flow_timeout errors and also none
of the nodes going down. The crashed nodes seem somehow related to the
mapreduce operations timing out.
I did a search on the machine and there were
echnologies
>
>
> On Dienstag, 21. Juni 2011 at 14:25, Jeremy Raymond wrote:
>
> > I increased the memory to 3GB on the VMs I'm using for Riak and also
> replaced a JavaScript reduce function I had missed converting to Erlang with
> the Erlang version. Monitoring the memo
And I'm certain it's not a memory or
> disk issue.
>
> On Sunday, June 19, 2011 at 6:01 PM, Jeremy Raymond wrote:
>
> Actually it's a bit later on where I see this:
>
> = Fri Jun 17 16:26:46 EDT 2011
>
> =ERROR REPORT 17-Jun-2011::16:26:46 =
gt; /usr/lib/riak/lib/os_mon-2.2.5/priv/bin/memsup: Erlang has
closed.
Erlang has closed
Other than that the only other thing that stands out is the info messages
about system_memory_high_watermark.
- Jeremy
On Sun, Jun 19, 2011 at 8:47 PM, Jeremy Raymond wrote:
> I see these messages
tend to only happen when you run out of RAM or disk space, or can't bind to
> a TCP port during riak_core startup. Were there other indications in the
> erlang.log.*?
>
> --
> Sean Cribbs
> Developer Advocate
> Basho Technologies, Inc.
> http://www.basho.com/
>
>
&g
Hello,
I have a 3 node Riak 0.14.2 cluster from the deb packages running on Ubuntu
10.10. I had a node go down with the following error from the
sasl-error.log. Ideas on tracking down the cause?
- Jeremy
=ERROR REPORT 17-Jun-2011::16:26:46 ===
** Generic server riak_kv_stat terminating
** L
So just backing up the files from separate nodes works? There won't be
inconsistencies in the data say if all the nodes had to be restored?
- Jeremy
On 2011-05-13 8:35 PM, "Justin Sheehy" wrote:
> Hi, Mike.
>
> Assuming that the cluster is using the default storage engine
> (bitcask) then the bac
I'm using Riak installed from riak_0.14.0-1_amd64.deb but am having a
problem with an Erlang reduce function I wrote because it used
calendar:iso_week_number/1 which isn't available in R13B04 which is bundled
with the .deb package. Is there an easy way to configure Riak to use a
different Erlang in
o.com>>
Developer Advocate
Basho Technologies, Inc.
http://basho.com/
On May 10, 2011, at 9:30 AM, Jeremy Raymond wrote:
Okay thanks. I figure something like this was going on. Besides the
timeout return value is there any way to tell what happened or at
what phase of some chained
er Advocate
> Basho Technologies, Inc.
> http://basho.com/
>
> On May 10, 2011, at 9:20 AM, Jeremy Raymond wrote:
>
> Hello,
>
> In a MapReduce job I have filtering through about 6500 items if I filter
> out enough items my m/r job completes successfully. If I loosen the
> f
Hello,
In a MapReduce job I have filtering through about 6500 items if I filter out
enough items my m/r job completes successfully. If I loosen the filtering such
that a large number of items would be included in the results the m/r fails
with error {error,javascript_reduce_timeout}. This error
I was looking for the change on bitbucket but found it on github. In general
where do changes show up first for basho projects?
On 2010-11-01 10:06 AM, "Jeremy Raymond" wrote:
Cool thanks.
>
> On 2010-11-01 1:12 AM, "Dan Reverri" wrote:
>
> Hi Jeremy,
>
at 8:07 PM, Jeremy Raymond wrote:
> >
> > I was taking a look at wriaki and noticed a potential bug. In wobj.erl
> the remove_links/3 funct...
>
> --
> Jeremy Raymond
>
> ___
> riak-users mailing list
> riak-users@li
the link_filter/2 function). Is this a bug or am I misreading
this function's intent?
--
Jeremy Raymond
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
76 matches
Mail list logo
