RE: accumulo 1.9 slow startup

How are you shutting down and starting up? For start-up, it works much better if most, if not all tservers are up before the master is started. That way, the assignments are distributed across the available tservers. If you start the master first, what will happen is that when the first few t

RE: Unassigned tables on every restart accumulo 1.10.1

There was additional discussion at https://github.com/apache/accumulo/issues/2085 incase that helps anyone. From: Ivershyn Viktor Sent: Wednesday, May 12, 2021 1:09 PM To: user@accumulo.apache.org Subject: Unassigned tables on every restart accumulo 1.10.1 HI all, have accumulo 1.10.

RE: Hadoop ConnectException

Did you verify that Hadoop is really up and healthy? Look at the Hadoop monitor pages and confirm that you can use the Hadoop cli to navigate around? You may also need to update the accumulo configuration files / env to match your configuration. You might what to look at using https://gith

RE: Re: Re: Re: Hadoop ConnectException

Use jps -m to check which processes you have running. Check the accumulo logs – are there any with *.err that have a size > 0? The .err files will be created on an unexpected exit. The other debug logs will provide a clearer picture of what is happening. Tail the master debug log / and

RE: How to stop accumulo replication

See: https://accumulo.apache.org/docs/2.x/administration/replication Replication should be controlled using the accumulo shell config command. To see the current system replication configuration: (-f provides a filter and will show only properties that contain replication in the name)

RE: hdfs rack awareness and accumulo

Are there any Accumulo system table that are offline (root, metadata)? Is there a manager (master) process available? What is the manager log saying? From: Ligade, Shailesh [USA] Sent: Thursday, August 5, 2021 7:55 AM To: user@accumulo.apache.org Subject: hdfs rack awareness and accumulo

RE: unassigned tablet "is SUSPENDED"

Are the wal files available for the table in hdfs? (there should be a metadata table entry with the file paths) From: Bulldog20630405 Sent: Wednesday, August 4, 2021 5:08 PM To: accumulo-user Cc: Bulldog20630405 Subject: unassigned tablet "is SUSPENDED" we have a few tablets that ar

RE: how to decommission tablet server

Maybe you could clarify. Decommissioning tablet servers and hdfs replication are separate and distinct issues. Accumulo will generally be unaware of hdfs replication and table assignment does not change the hdfs replication. You can set the replication factor for a tablet - but that is used on w

RE: [External] RE: how to decommission tablet server

The accumulo admin service should attempt to unload the tablets a little more gracefully than just stopping the service. I don't know why it's not stopping the service. I do know that some had issues when the services were controlled with systemd. Stopping the service without the admin command (

RE: [EXTERNAL EMAIL] - Re: [External] RE: how to decommission tablet server

If the admin stop fails to stop the service you will need to either kill or service stop the linux process. The hosts file should be able to be modified either before or after. If you do it before remember that things like admin start-all, stop-all will not know about those nodes. Likewis

RE: accumulo 1.10 replication issue

How are you inserting the data? From: Ligade, Shailesh [USA] Sent: Wednesday, September 22, 2021 10:22 PM To: user@accumulo.apache.org Subject: accumulo 1.10 replication issue Hello, I am following Apache Accumulo

RE: Triggering Table Balancer in Accumulo [SEC=UNOFFICIAL]

Just thinking about other ways that might work - have not tried any of this, so safe may be relative... Sometimes it seems easier to give Accumulo what it wants rather than fighting it - an example would be when you have a "missing" file - you can add an "empty" file to serve as a placeholder a

RE: accumulo tserver rolling restart

See https://accumulo.apache.org/1.10/accumulo_user_manual.html#_restarting_process_on_a_node - A note on rolling restarts. There is property that can be set (table.suspend.duration) that will delay the reassignment while a tserver is restarting - there is a trade-off on the data not being avai

RE: accumulo tserver rolling restart

ffect immediately? Thanks -S From: dev1 mailto:d...@etcoleman.com>> Sent: Monday, November 29, 2021 9:25 AM To: 'user@accumulo.apache.org' mailto:user@accumulo.apache.org>> Subject: [External] RE: accumulo tserver rolling restart See https://accumulo.apache.org/1.1

RE: [EXTERNAL EMAIL] - Re: accumulo tserver rolling restart

there a reason to not just stop the cluster, reset the heap and restart the cluster? That is simpler. On Mon, Nov 29, 2021 at 9:37 AM dev1 mailto:d...@etcoleman.com>> wrote: Yes – and don’t forget to reset it back when you are done. From: Ligade, Shailesh [USA] mailto:ligade_shail...@bah.

RE: [EXTERNAL EMAIL] - Re: accumulo tserver rolling restart

9, 2021 9:54 AM To: user@accumulo.apache.org<mailto:user@accumulo.apache.org> mailto:user@accumulo.apache.org>> Subject: [EXTERNAL EMAIL] - Re: accumulo tserver rolling restart Is there a reason to not just stop the cluster, reset the heap and restart the cluster? That is simpler. On Mon, No

RE: [EXTERNAL EMAIL] - Re: accumulo tserver rolling restart

my testing it didn’t make any difference, but am just curious. -S From: dev1 mailto:d...@etcoleman.com>> Sent: Tuesday, November 30, 2021 9:17 AM To: 'user@accumulo.apache.org' mailto:user@accumulo.apache.org>> Subject: RE: [EXTERNAL EMAIL] - Re: accumulo tserver rolling r

RE: [EXTERNAL EMAIL] - Re: accumulo tserver rolling restart

tablet.suspend.duration setting, once I update that setting, do I need to restart master? After updating the setting, I saw in master log had old value (0s), but if I restart master it shows correct value..in my testing it didn’t make any difference, but am just curious. -S From: dev1 mailto:d

RE: [External] Re: [EXTERNAL EMAIL] - Re: accumulo tserver rolling restart

ables and then bring down the master… We can take short downtime and my understanding is that the master is the one keeping track of tservers and offline tablets situation. So just curious… Thanks again -S From: dev1 mailto:d...@etcoleman.com>> Sent: Monday, November 29, 2021 2:56 PM T

RE: replication table offline issue

Deleting / recreating the replication table should not be necessary and in any case you very likely cannot delete / create the accumulo.replication table - the shell will error on the delete because it is in the accumulo namespace. Is the replication table hosted on a single tserver? Are there

RE: issue with OS user using shell

Did you try to pass a log4j configuration using -D option on the command line? Ed Coleman From: Ligade, Shailesh [USA] Sent: Friday, January 21, 2022 8:19 AM To: user@accumulo.apache.org Subject: issue with OS user using shell Heo, When a regular OS user use shell command, looks like that OS u

RE: uassigned tablets issue

Can you look at the master log? Do you see migrations occurring? Ed Coleman From: Ligade, Shailesh [USA] Sent: Tuesday, January 25, 2022 9:23 AM To: user@accumulo.apache.org Subject: uassigned tablets issue Hello, For some strange reason, monitor master page (version 1.10.0) shows unassigned

Re: uassigned tablets issue

umulo.apache.org' Subject: Re: uassigned tablets issue Sorry for late reply, from the log i can't tell if it is migrating tablets..we have replication on so logs are very chatty, but whatever i saw I didn't see it as migrating tablets. Thanks S From: d

RE: tablets per tablet server for accumulo 1.10.0

You can get the hdfs size using standard hdfs commands - count or ls. As long as you have not cloned the table, the size of the hdfs files and the space occupied by the table are equivalent. You can also get a sense of the referenced files examining the metadata table - the column qualifier fi

RE: tablets per tablet server for accumulo 1.10.0

again, -S ____ From: dev1 mailto:d...@etcoleman.com>> Sent: Monday, January 31, 2022 1:14 PM To: 'user@accumulo.apache.org' mailto:user@accumulo.apache.org>> Subject: [External] RE: tablets per tablet server for accumulo 1.10.0 You can get the hdfs s

RE: Zero size tablets in accumulo.metadata

What version of Accumulo? From: Hart, Andrew Sent: Thursday, February 3, 2022 2:29 AM To: user@accumulo.apache.org Subject: Zero size tablets in accumulo.metadata Hi, I have about twenty accumulo.metadata tablets and they all have zero size except one with most of the data and one tablet with

RE: tablets per tablet server for accumulo 1.10.0

one shell command..is there a better way of doing all this? I mean i want to be as gentle to my production system and yet as fast as possible.. don't want to spend days doing compact/merge 🙁 Thanks -S From: dev1 mailto:d...@etcoleman.com>> Sent: Tuesd

Re: accumulo 1.10.0 masters won't start

Does the monitor or any of the logs show errors that relate to exceeding the ZooKeeper jute buffer size? Is so, have you tried increasing the ZooKeeper jute.maxbuffer limit(https://zookeeper.apache.org/doc/r3.5.9/zookeeperAdmin.html#Unsafe+Options)? Ed Coleman

Re: accumulo 1.10.0 masters won't start

s up -S ____ From: dev1 Sent: Wednesday, February 9, 2022 12:26 PM To: user@accumulo.apache.org Subject: [External] Re: accumulo 1.10.0 masters won't start Does the monitor or any of the logs show errors that relate to exceeding the ZooKeeper jute buffer size? Is so, have you tried increasin

Re: accumulo 1.10.0 masters won't start

eeper: java.lang.IllegalArgumentException: No enum constant org.apache.accumulo.core.master.thrift.MasterGoalState I hope i didn't delete goal_state accidently ...;-( currently ls on goal_state is [], is there a way to add some value there? -S ____ From: dev1 Sent:

Re: accumulo 1.10.0 masters won't start

ood values in zookeeper…so not sure.. 🙁 -S ____ From: dev1 Sent: Wednesday, February 9, 2022 2:22 PM To: user@accumulo.apache.org Subject: [External] Re: accumulo 1.10.0 masters won't start The is a utility - SetGoalState that can be run from the command

Re: accumulo 1.10.0 masters won't start

i have not update any timeout is that going in zoo.cfg? I can see there is min/maxSessionTimeout 2/20, is that what are you refering to? -S From: dev1 Sent: Wednesday, February 9, 2022 2:51 PM To: user@accumulo.apache.org Subject: [External] Re: accumulo 1.1

Re: accumulo 1.10.0 masters won't start

start Thanks I can try that, At this point, my goal is to get accumulo up. I was just wondering if I can set different goal like SAFE_MODE will it come up by ignoring fate and other issues? If that comes up, can I switch back to NORMAL, will that work? I understand there may be some data l

Re: accumulo 1.10.0 masters won't start

re where to set that on accumulo side. I set instance.zookeeper.timeout value to 90s in accumulo-site.xml But still get those zookeeper KeeperErrorCode errors -S From: dev1 Sent: Wednesday, February 9, 2022 4:27 PM To: user@accumulo.apache.org Subject: [EXTERNAL EMAIL] - Re: accumulo 1.1

Re: accumulo 1.10.0 masters won't start

want to see if 4194304 (or larger) as a value works, From: dev1 Sent: Wednesday, February 9, 2022 5:25 PM To: user@accumulo.apache.org Subject: Re: accumulo 1.10.0 masters won't start jute.maxbuffer is a ZooKeeper property - it needs to be set on the zook

RE: Tablet assignment slow upon restart [SEC=UNOFFICIAL]

There is a setting table.suspend.duration (- https://accumulo.apache.org/1.10/accumulo_user_manual.html#_table_suspend_duration) That will pause the tablet reassignment while a tserver restarts. There was a discussion on doing rolling restarts on this list around Dec 2, 2021 (one of the emails

RE: accumulo 1.10.0 masters won't start

the file size limit to 2MB: SOLR_OPTS="$SOLR_OPTS -Djute.maxbuffer=0x20" - Anyways, a master is up and running for an hour now..so just trying to understand what was changed and revert it after it stabilize. Thanks a bunch. -S ___

Re: accumulo 1.10.0 masters won't start

, and then setup replication on the new cluster again..i was avoiding that. but i guess that may be my only option. A last question, if I let this table continue to grow tablets, what is the worst case scenerio? How it may affect system performance? Thanks -S ____ From

Re: accumulo 1.10.0 masters won't start

tem performance? Thanks -S ____ From: dev1 Sent: Tuesday, February 15, 2022 3:11 PM To: 'user@accumulo.apache.org' Subject: [External] RE: accumulo 1.10.0 masters won't start Can you compact the table? How aggressive do you want to get? I do not understand wh

RE: accumulo 1.10.0 masters won't start

. Thank you so much -S From: dev1 mailto:d...@etcoleman.com>> Sent: Wednesday, February 16, 2022 9:29 AM To: 'user@accumulo.apache.org' mailto:user@accumulo.apache.org>> Subject: [External] Re: accumulo 1.10.0 masters won't start >

RE: how to stop entire (or on a single table) replication properly

I do not understand what you are asking - it would help if you stated what you are trying to accomplish and if you clearly identified source vs. destination. Ed Coleman From: Ligade, Shailesh [USA] Sent: Thursday, February 17, 2022 7:37 AM To: user@accumulo.apache.org Subject: how to stop entir

RE: how to stop entire (or on a single table) replication properly

imary replication table? One time i tried to do deleteall on primary replication table but when accumulo master re-started, it was complaining a lot about replication data, so just wanted to figure out proper steps. thanks -S From: dev1 mailto:d...@etcoleman.com&g

RE: ENTRIES listed on accumulo 1.10 monitor

On bulk import the count is not known. The count is updated on compactions. Once a tablet / file is compacted it's count will be updated. The count is only approximate anyway - it will not be able to determine of a row has expired (past age-off) after the count has been calculated. Ed Coleman

RE: [External] Re: accumulo 1.10.0 unassigned tablets issue

To amplify Christopher's comments that 33k+ tablets is a lot of work for a tserver to manage, and may be causing some of your stability issues. Besides consolidating tablets into fewer files / tablets you probably should look into running multiple tservers per node. You will need to adjust the

RE: [External] Re: odd issue with accumulo 1.10.0 starting up

When an Accumulo process abnormally terminates, there may be a file create with the exception of the problem – the files may be names *.out (or *.err) can’t recall which. Normally the files have 0 size, but on termination will have some text. Are you seeing those files and do they point to the

RE: Accumulo 1.10.0

I would suspect that the metadata table became corrupted when the system went unstable and two tablet servers somehow ended up both thinking that they were responsible for the same extents(s) This should not be because of the balancer running. If you scan the accumulo.metadata table using the

RE: Accumulo 1.10.0

elp message.. Thanks in advance, -S ____ From: dev1 mailto:d...@etcoleman.com>> Sent: Tuesday, April 12, 2022 10:07 AM To: 'user@accumulo.apache.org' mailto:user@accumulo.apache.org>> Subject: [External] RE: Accumulo 1.10.0 I would suspect that the metadata ta

RE: minor compaction same as flush

Flush and compactions are different actions. Flush - sorts and writes current, in-memory changes to a file. This can reduce the amount of recovery in case of a failure because the flushed entries do not need to be processed from the WAL. Compactions combine multiple files into a single file.

RE: accumulo 1.10 replication table issues

Looking at the code, I do not see anything other than the manager (master) calling things that would set the table online. Were the manager(s) started when the replication table was ONLINE? You could try stopping the managers, setting the replication table offline and then restarting the manag

RE: [EXTERNAL EMAIL] - RE: accumu 1.10.0 master log connection refised error

You need to kill any running masters so that the FATE command can get the master lock. Once deleted, restart them. Ed Coleman -Original Message- From: Shailesh Ligade via user Sent: Monday, October 17, 2022 9:22 AM To: user@accumulo.apache.org Subject: RE: [EXTERNAL EMAIL] - RE: accum

RE: very-high latency updates through thrift proxy server

Could it be data dependent? For example, if you have a lot of data that has passed its TTL you may be scanning across a lot of data to find data that is eligible to return. Similar situations could have to do with visibilities that you can’t access… Or, maybe it’s related to your scan range?

RE: Debug logging of accumulo classes

You can change the logging configuration levels for Accumulo processes editing the log4j-service.properties file. Ed Coleman From: Samudrala, Ranganath [USA] via user Sent: Wednesday, January 18, 2023 9:41 AM To: user@accumulo.apache.org Subject: Debug logging of accumulo classes Hello I am tr

RE: Exception during accumulo init

Why do you keep using "-upload-accumulo-props~? During initialization Accumulo will automatically read your site config file and then set properties in ZooKeeper accordingly. Ed Coleman From: Samudrala, Ranganath [USA] via user Sent: Friday, January 20, 2023 10:48 AM To: user@accumulo.apache.

RE: accumulo shell compact --cancel -t mytable does not work

Which Accumulo version are you using? The command should work - it may not stop the compations that are currently writing files, but it should stop new ones from starting and cancel once those write complete. Can you look at the number of FATE operations you have running and if there are confl

ASF Trip opportunity - Berlin Buzzwords

In case anyone is interested. Hi All, The ASF Travel Assistance Committee is supporting taking up to six (6) people to attend Berlin Buzzwords In June this year. This includes Conference passes, and travel & accommodation as needed. Please see our website at https://tac.apache.org for more inf