I tried to run camus with the properties file in the examples dir:
java -cp ...... com.linkedin.camus.etl.kafka.CamusJob -P
myproperties.properties
then it says that my output dir does not exist:
~/tools/camus/camus-etl-kafka$ java -cp
target/camus-etl-kafka-0.1.0-SNAPSHOT.jar
com.linkedin.camus.etl.kafka.CamusJob -P camus.properties
Starting Kafka ETL Job
The blacklisted topics: []
The whitelisted topics: []
Dir Destination set to: /camus/out
Getting the base paths.
The execution base path does not exist. Creating the directory
The history base path does not exist. Creating the directory.
Exception in thread "main" java.io.FileNotFoundException: File /camus/exec
does not exist.
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:397)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
at org.apache.hadoop.fs.FileSystem.getContentSummary(FileSystem.java:801)
at com.linkedin.camus.etl.kafka.CamusJob.run(CamusJob.java:223)
at com.linkedin.camus.etl.kafka.CamusJob.run(CamusJob.java:556)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
I tried changing the dir to hdfs://localhost/camus/out it says:
$ java -cp target/camus-etl-kafka-0.1.0-SNAPSHOT.jar
com.linkedin.camus.etl.kafka.CamusJob -P camus.properties
Starting Kafka ETL Job
The blacklisted topics: []
The whitelisted topics: []
Dir Destination set to: hdfs://localhost/camus/out
Getting the base paths.
Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS:
hdfs://localhost/camus/exec, expected: file:///
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381)
at
org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55)
at org.apache.hadoop.fs.LocalFileSystem.pathToFile(LocalFileSystem.java:61)
at org.apache.hadoop.fs.LocalFileSystem.exists(LocalFileSystem.java:51)
at com.linkedin.camus.etl.kafka.CamusJob.run(CamusJob.java:211)
at com.linkedin.camus.etl.kafka.CamusJob.run(CamusJob.java:556)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
so how can I let Camus know that this is in a hdfs environment, and the
code in CamusJob.java:140
private Job createJob(Properties props) throws IOException {
Job job = new Job(getConf());
get a conf that points to a hdfs setup?
I already set my env HADOOP_CONF_DIR to my running hadoop
thanks
Yang
