Re: Accumulo 1.10.0

2022-04-13 Thread Ligade, Shailesh [USA]
apache.org' Subject: Re: Accumulo 1.10.0 Thanks i noticed that issue with the write permission, anyway i was able to run all delete. In reality i ran CheckForMetadatProblems to generate script and ran all delete Shell didn't throw any error Then i restarted my both masters, but then master cam

Re: Accumulo 1.10.0

2022-04-13 Thread Ligade, Shailesh [USA]
apache.org' Subject: [External] RE: Accumulo 1.10.0 If you still have an issue – check that your user has WRITE permissions on the metadata table (even root needs to be added) – if you grant permission(s) you likely would want to remove what you added once you are done to prevent inadvert

RE: Accumulo 1.10.0

2022-04-13 Thread dev1
mistake with a command intended for another table. (Besides being a good security practice to operate with the minimum required permissions) From: Ligade, Shailesh [USA] Sent: Wednesday, April 13, 2022 8:41 AM To: 'user@accumulo.apache.org' Subject: Re: Accumulo 1.10.0 I think i figu

Re: Accumulo 1.10.0

2022-04-13 Thread Ligade, Shailesh [USA]
;user@accumulo.apache.org' Subject: Re: Accumulo 1.10.0 Thanks Ed, a quick question, Now that i want to delete those duplicates (there are many of those 🙁) the scan output is a;: loc:a [] tablet1:9997 a;: loc:zz [] tablet2:9997 What is the right delete command, when i issue

Re: Accumulo 1.10.0

2022-04-13 Thread Ligade, Shailesh [USA]
AM To: 'user@accumulo.apache.org' Subject: [External] RE: Accumulo 1.10.0 I would suspect that the metadata table became corrupted when the system went unstable and two tablet servers somehow ended up both thinking that they were responsible for the same extents(s) This should not be

Re: Accumulo 1.10.0

2022-04-12 Thread Ligade, Shailesh [USA]
Thanks Ed, Stooping tserver didn't help weird issue i saw is that both location on the same tserver. So i guess i have to do it hard way...:-( -S From: dev1 Sent: Tuesday, April 12, 2022 10:07 AM To: 'user@accumulo.apache.org' Subject: [Externa

RE: Accumulo 1.10.0

2022-04-12 Thread dev1
I would suspect that the metadata table became corrupted when the system went unstable and two tablet servers somehow ended up both thinking that they were responsible for the same extents(s) This should not be because of the balancer running. If you scan the accumulo.metadata table using the

RE: [External] Re: accumulo 1.10.0 unassigned tablets issue

2022-03-04 Thread dev1
ginal Message- From: Christopher Sent: Friday, March 4, 2022 9:14 AM To: accumulo-user Subject: Re: [External] Re: accumulo 1.10.0 unassigned tablets issue On Fri, Mar 4, 2022 at 8:36 AM Ligade, Shailesh [USA] wrote: > EDIT again: Thanks Chris[topher], > > Appreciate your support! &g

Re: [External] Re: accumulo 1.10.0 unassigned tablets issue

2022-03-04 Thread Christopher
an figure out how to get to your target setup from your current setup. > > I will have to look at each and every property to ensure it makes sense for > sure.. > > Thanks > > -S > > -Original Message- > From: Christopher > Sent: Wednesday, March 2, 2022

RE: [External] Re: accumulo 1.10.0 unassigned tablets issue

2022-03-04 Thread Ligade, Shailesh [USA]
, 2022 3:09 PM To: accumulo-user Subject: Re: [External] Re: accumulo 1.10.0 unassigned tablets issue On Wed, Mar 2, 2022 at 1:51 PM Ligade, Shailesh [USA] wrote: > EDIT > Thanks Chris[topher], > > I do have instance.volume.replacement overridden > > Does that mean it

Re: [External] Re: accumulo 1.10.0 unassigned tablets issue

2022-03-02 Thread Christopher
question regarding them, you can ask here, but I would start by reviewing your configs against the docs. > > Thanks > > -S > > > > From: Christopher > Sent: Wednesday, March 2, 2022 1:32 PM > To: accumulo-user > Subject: [External

Re: [External] Re: accumulo 1.10.0 unassigned tablets issue

2022-03-02 Thread Ligade, Shailesh [USA]
set table.suspend.duration, and stop/reboot tserver. After i am done, i can restore the property. Please advise. Thanks -S From: Christopher Sent: Wednesday, March 2, 2022 1:32 PM To: accumulo-user Subject: [External] Re: accumulo 1.10.0 unassigned tablets

Re: accumulo 1.10.0 unassigned tablets issue

2022-03-02 Thread Christopher
The replacements message should only appear if you have instance.volumes.replacements set in your configuration. On Wed, Mar 2, 2022 at 11:02 AM Ligade, Shailesh [USA] wrote: > > Hello, > > I need reboot a tserver with 34k hosted tablets. > > I set table.supend.duration to 15 min and stop tserver

Re: accumulo 1.10.0 masters won't start

2022-02-16 Thread Ligade, Shailesh [USA]
structure as is.. -S From: dev1 Sent: Wednesday, February 16, 2022 10:49 AM To: 'user@accumulo.apache.org' Subject: [External] RE: accumulo 1.10.0 masters won't start I would use importdirectory [src_dir] [fail_dir] -t new_table false I would move

RE: accumulo 1.10.0 masters won't start

2022-02-16 Thread dev1
;user@accumulo.apache.org' Subject: Re: accumulo 1.10.0 masters won't start Thanks Since hdfs fsck is fine on /accumulo, I can backup my tables to some location within hdfs (not under accumulo) and reinitialize accumulo. then I can recreate my tables/users on new instances. What will be command to impor

Re: accumulo 1.10.0 masters won't start

2022-02-16 Thread Ligade, Shailesh [USA]
Sent: Wednesday, February 16, 2022 9:29 AM To: 'user@accumulo.apache.org' Subject: [External] Re: accumulo 1.10.0 masters won't start > What happens if I let the #tablets grown? It sounds like you might be in the worst case now? There is over head for each tablet - at wha

Re: accumulo 1.10.0 masters won't start

2022-02-16 Thread dev1
arger is reasonable. Ed Coleman From: Ligade, Shailesh [USA] Sent: Wednesday, February 16, 2022 8:05 AM To: 'user@accumulo.apache.org' Subject: Re: accumulo 1.10.0 masters won't start Thanks Ed, We have two clusters, one is source other is peer. I am te

Re: accumulo 1.10.0 masters won't start

2022-02-16 Thread dev1
Tservers will hold locks. From: Ligade, Shailesh [USA] Sent: Wednesday, February 16, 2022 9:10 AM To: 'user@accumulo.apache.org' Subject: Re: accumulo 1.10.0 masters won't start Well now master doesn't come up throwing all sorts of zo

Re: accumulo 1.10.0 masters won't start

2022-02-16 Thread Ligade, Shailesh [USA]
that normal? Appreciate your help -S From: Ligade, Shailesh [USA] Sent: Wednesday, February 16, 2022 8:05 AM To: 'user@accumulo.apache.org' Subject: Re: accumulo 1.10.0 masters won't start Thanks Ed, We have two clusters, one is source other i

Re: accumulo 1.10.0 masters won't start

2022-02-16 Thread Ligade, Shailesh [USA]
grow tablets, what is the worst case scenerio? How it may affect system performance? Thanks -S From: dev1 Sent: Tuesday, February 15, 2022 3:11 PM To: 'user@accumulo.apache.org' Subject: [External] RE: accumulo 1.10.0 masters won't start Can you com

RE: accumulo 1.10.0 masters won't start

2022-02-15 Thread dev1
, February 15, 2022 1:11 PM To: user@accumulo.apache.org Subject: Re: accumulo 1.10.0 masters won't start Well, I am trying to merge a large table (8T with 72k tablets, with default tablet size 1G) Since I keep getting those zookeeper errors realted to size, i keep on bumping the jute.maxbuffer ad

Re: accumulo 1.10.0 masters won't start

2022-02-15 Thread Ligade, Shailesh [USA]
9:20 AM To: user@accumulo.apache.org Subject: Re: accumulo 1.10.0 masters won't start Thanks Ed, That saved the day. The confusing part setting up that property is documentation if it needs hex or bytes etc. Even the example they provided here https://solr.apache.org/guide/7_4/setting-up-a

Re: accumulo 1.10.0 masters won't start

2022-02-10 Thread Ligade, Shailesh [USA]
_ From: dev1 Sent: Wednesday, February 9, 2022 5:54 PM To: user@accumulo.apache.org Subject: [External] Re: accumulo 1.10.0 masters won't start You might want to set the accumulo (zookeeper client) side - by setting ACCUMULO_JAVA_OPTS that is processed in accumulo-env.sh (or just edit

Re: accumulo 1.10.0 masters won't start

2022-02-09 Thread dev1
want to see if 4194304 (or larger) as a value works, From: dev1 Sent: Wednesday, February 9, 2022 5:25 PM To: user@accumulo.apache.org Subject: Re: accumulo 1.10.0 masters won't start jute.maxbuffer is a ZooKeeper property - it needs to be set on the zook

Re: accumulo 1.10.0 masters won't start

2022-02-09 Thread dev1
uary 9, 2022 5:02 PM To: user@accumulo.apache.org Subject: RE: accumulo 1.10.0 masters won't start Thanks Even if I set jute.maxbuffer on zookeeper in conf/java.env file to -Djute.maxbuffer=30 I see in accumulo master log as INFO: jute.maxbuffer value is 1048575 Bytesnot su

RE: accumulo 1.10.0 masters won't start

2022-02-09 Thread Shailesh Ligade
But still get those zookeeper KeeperErrorCode errors -S From: dev1 Sent: Wednesday, February 9, 2022 4:27 PM To: user@accumulo.apache.org Subject: [EXTERNAL EMAIL] - Re: accumulo 1.10.0 masters won't start I would not recommend setting the goal state directly unlit there are no other alterna

Re: accumulo 1.10.0 masters won't start

2022-02-09 Thread dev1
eed it - but if you don't have it and you need it, well... The table names and id are all in ZooKeeper. Ed Coleman From: Shailesh Ligade Sent: Wednesday, February 9, 2022 3:47 PM To: user@accumulo.apache.org Subject: RE: accumulo 1.10.0 masters won't

RE: accumulo 1.10.0 masters won't start

2022-02-09 Thread Shailesh Ligade
From: dev1 Sent: Wednesday, February 9, 2022 3:36 PM To: user@accumulo.apache.org Subject: [EXTERNAL EMAIL] - Re: accumulo 1.10.0 masters won't start For values in zoo.cfg see: https://zookeeper.apache.org/doc/r3.5.9/zookeeperAdmin.html#sc_advancedConfiguration<https

Re: accumulo 1.10.0 masters won't start

2022-02-09 Thread dev1
quests, excluding ping requests. 0b000100 : Unused, reserved ... zookeeper.apache.org From: Ligade, Shailesh [USA] Sent: Wednesday, February 9, 2022 3:03 PM To: user@accumulo.apache.org Subject: Re: accumulo 1.10.0 masters won't start thanks for response, no

Re: accumulo 1.10.0 masters won't start

2022-02-09 Thread Ligade, Shailesh [USA]
: [External] Re: accumulo 1.10.0 masters won't start Have you tried to increase the zoo session timeout value? I think it's zookeeper.session.timeout.ms From: Ligade, Shailesh [USA] Sent: Wednesday, February 9, 2022 2:47 PM To: user@accumulo.apache.org S

Re: accumulo 1.10.0 masters won't start

2022-02-09 Thread dev1
Have you tried to increase the zoo session timeout value? I think it's zookeeper.session.timeout.ms From: Ligade, Shailesh [USA] Sent: Wednesday, February 9, 2022 2:47 PM To: user@accumulo.apache.org Subject: Re: accumulo 1.10.0 masters won't star

Re: accumulo 1.10.0 masters won't start

2022-02-09 Thread Ligade, Shailesh [USA]
zookeeper…so not sure.. 🙁 -S From: dev1 Sent: Wednesday, February 9, 2022 2:22 PM To: user@accumulo.apache.org Subject: [External] Re: accumulo 1.10.0 masters won't start The is a utility - SetGoalState that can be run from the command line acc

Re: accumulo 1.10.0 masters won't start

2022-02-09 Thread dev1
9, 2022 1:54 PM To: user@accumulo.apache.org Subject: Re: accumulo 1.10.0 masters won't start Well, i just went ahead and deleted fate in zookeeper and restarted the master..it was doing better but then i am getting different error ERROR: Problem getting real goal state from zook

Re: accumulo 1.10.0 masters won't start

2022-02-09 Thread Ligade, Shailesh [USA]
org.apache.accumulo.core.master.thrift.MasterGoalState I hope i didn't delete goal_state accidently ...;-( currently ls on goal_state is [], is there a way to add some value there? -S From: dev1 Sent: Wednesday, February 9, 2022 1:32 PM To: user@accumulo.apache.org Subject: [External] Re: accumulo 1

Re: accumulo 1.10.0 masters won't start

2022-02-09 Thread Ligade, Shailesh [USA]
user@accumulo.apache.org Subject: [External] Re: accumulo 1.10.0 masters won't start Did you try setting the increased size in the zkCli.sh command (or wherever it gets it environment from?) The ZK docs indicate that it needs to be set to the same size on all servers and clients. You should be a

Re: accumulo 1.10.0 masters won't start

2022-02-09 Thread dev1
node named debug - doing a get on that should show the op name. Ed Coleman From: Ligade, Shailesh [USA] Sent: Wednesday, February 9, 2022 12:54 PM To: user@accumulo.apache.org Subject: Re: accumulo 1.10.0 masters won't start Thanks I added -Djute

Re: accumulo 1.10.0 masters won't start

2022-02-09 Thread Ligade, Shailesh [USA]
-S From: dev1 Sent: Wednesday, February 9, 2022 12:26 PM To: user@accumulo.apache.org Subject: [External] Re: accumulo 1.10.0 masters won't start Does the monitor or any of the logs show errors that relate to exceeding the ZooKeeper jute buffer size? Is so,

Re: accumulo 1.10.0 masters won't start

2022-02-09 Thread dev1
Does the monitor or any of the logs show errors that relate to exceeding the ZooKeeper jute buffer size? Is so, have you tried increasing the ZooKeeper jute.maxbuffer limit(https://zookeeper.apache.org/doc/r3.5.9/zookeeperAdmin.html#Unsafe+Options)? Ed Coleman

Re: accumulo 1.10.0 log rotate

2021-11-23 Thread Christopher
You should not rely on the start-daemon.sh script's strategy for rotating logs in production. That is more or less a hack for scenarios where you're running "out of the tarball" rather than a properly integrated and configured deployment. You're right that it only rotates when that script is run to

Re: accumulo 1.10.0 log rotate

2021-11-23 Thread Mike Miller
Are you using Log4j? Log4j has https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/RollingFileAppender.html On Tue, Nov 23, 2021 at 11:15 AM Ligade, Shailesh [USA] < ligade_shail...@bah.com> wrote: > Hello, > > Currently logs (all tserver, master) are getting huge and sometimes fills >