Re: How to set up RocksDB

2015-11-10 Thread Navina Ramesh
Hi Lukas, My reply to your question was stuck in draft.. :( Pasting it below just for reference. You are right. I guess your option is to create a symlink. *Thanks for clarifying that you are not using Yarn. * *In Samza container implementation, we don't seem to allow any configuration related to

Re: How to set up RocksDB

2015-11-10 Thread Lukáš Havrlant
Hi, I managed to start Samza after all. I am not sure what has helped eventually. But still I had to create a folder /opt/samza/state (/opt/samza is Samza's Home folder) and I don't want to have this folder there. Is there any option for it? I looked at the configuration page ( http://samza.apache.

Re: How to set up RocksDB

2015-11-09 Thread Lukáš Havrlant
Hi Navina, I am sorry I didn't mentioned it before. We run Samza as a local job: job.factory.class=org.apache.samza.job.local.ThreadJobFactory But thank for your answer regarding YARN, I'll be setting up Samza on YARN next week! If I remove all the ROCKSDB_SHAREDLIB_DIR stuff I get this error: Ex

Re: How to set up RocksDB

2015-11-09 Thread Navina Ramesh
Hi Lukas, Are you running the Samza job on Yarn? If so, the state stores can be found in the working directory of a yarn application. Samza doesn't configure any special location. Unless you have configured the local-dir location in yarn, it should use the default, which is *${hadoop.tmp.dir}/nm-l

Re: How to set up RocksDB

2015-11-09 Thread Lukáš Havrlant
Yes I think I saw something like "permission denied" in the logs. But I cannot find the config that tells the Samza the destination folder. We tried to set different -Djava.io.tmpdir but it didn't help. Can you please give me a hint what config should it be? 2015-11-09 18:05 GMT+01:00 Tommy Becker

Re: How to set up RocksDB

2015-11-09 Thread Tommy Becker
The native RocksDB libs are included in the jar file, but they are extracted at runtime before they can be loaded. Make sure the user running the job has permission to write to...wherever it is that it tries to put it (sorry don't know offhand). But you shouldn't have to set any environment var