[
https://issues.apache.org/jira/browse/HBASE-7912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572296#comment-15572296
]
stack commented on HBASE-7912:
------------------------------
My CLASSPATH is fine enough to do cluster start/stop, loadings, shell, MR bulk
jobs. It is pasted below. Tell me what is missing. I can add my config if
that'd help. How did you test against a cluster? What was your setup like?
{code}
stack@ve0524:~$ ./hbase/bin/hbase --config ~/conf_hbase classpath
/home/stack/conf_hbase:/home/stack/bin/jdk//lib/tools.jar:/home/stack/hbase/bin/..:/home/stack/hbase/bin/../lib/activation-1.1.jar:/home/stack/hbase/bin/../lib/antisamy-1.4.3.jar:/home/stack/hbase/bin/../lib/aopalliance-1.0.jar:/home/stack/hbase/bin/../lib/apacheds-i18n-2.0.0-M15.jar:/home/stack/hbase/bin/../lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/stack/hbase/bin/../lib/api-asn1-api-1.0.0-M20.jar:/home/stack/hbase/bin/../lib/api-util-1.0.0-M20.jar:/home/stack/hbase/bin/../lib/asm-3.1.jar:/home/stack/hbase/bin/../lib/avro-1.7.4.jar:/home/stack/hbase/bin/../lib/batik-css-1.7.jar:/home/stack/hbase/bin/../lib/batik-ext-1.7.jar:/home/stack/hbase/bin/../lib/batik-util-1.7.jar:/home/stack/hbase/bin/../lib/bsh-core-2.0b4.jar:/home/stack/hbase/bin/../lib/commons-beanutils-1.7.0.jar:/home/stack/hbase/bin/../lib/commons-beanutils-core-1.7.0.jar:/home/stack/hbase/bin/../lib/commons-cli-1.2.jar:/home/stack/hbase/bin/../lib/commons-codec-1.9.jar:/home/stack/hbase/bin/../lib/commons-collections-3.2.2.jar:/home/stack/hbase/bin/../lib/commons-compress-1.4.1.jar:/home/stack/hbase/bin/../lib/commons-configuration-1.6.jar:/home/stack/hbase/bin/../lib/commons-daemon-1.0.13.jar:/home/stack/hbase/bin/../lib/commons-digester-1.8.jar:/home/stack/hbase/bin/../lib/commons-el-1.0.jar:/home/stack/hbase/bin/../lib/commons-fileupload-1.2.jar:/home/stack/hbase/bin/../lib/commons-httpclient-3.1.jar:/home/stack/hbase/bin/../lib/commons-io-2.4.jar:/home/stack/hbase/bin/../lib/commons-lang-2.6.jar:/home/stack/hbase/bin/../lib/commons-logging-1.2.jar:/home/stack/hbase/bin/../lib/commons-math-2.2.jar:/home/stack/hbase/bin/../lib/commons-math3-3.1.1.jar:/home/stack/hbase/bin/../lib/commons-net-3.1.jar:/home/stack/hbase/bin/../lib/curator-client-2.7.1.jar:/home/stack/hbase/bin/../lib/curator-framework-2.7.1.jar:/home/stack/hbase/bin/../lib/curator-recipes-2.7.1.jar:/home/stack/hbase/bin/../lib/disruptor-3.3.0.jar:/home/stack/hbase/bin/../lib/esapi-2.1.0.jar:/home/stack/hbase/bin/../lib/findbugs-annotations-1.3.9-1.jar:/home/stack/hbase/bin/../lib/gson-2.2.4.jar:/home/stack/hbase/bin/../lib/guava-12.0.1.jar:/home/stack/hbase/bin/../lib/guice-3.0.jar:/home/stack/hbase/bin/../lib/guice-servlet-3.0.jar:/home/stack/hbase/bin/../lib/hadoop-annotations-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-auth-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-client-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-common-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-distcp-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-hdfs-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-mapreduce-client-app-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-mapreduce-client-common-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-mapreduce-client-core-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-mapreduce-client-jobclient-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-mapreduce-client-shuffle-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-yarn-api-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-yarn-client-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-yarn-common-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-yarn-server-common-2.7.1.jar:/home/stack/hbase/bin/../lib/hbase-annotations-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-annotations-2.0.0-SNAPSHOT-tests.jar:/home/stack/hbase/bin/../lib/hbase-client-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-common-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-common-2.0.0-SNAPSHOT-tests.jar:/home/stack/hbase/bin/../lib/hbase-examples-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-external-blockcache-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-hadoop2-compat-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-hadoop-compat-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-it-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-it-2.0.0-SNAPSHOT-tests.jar:/home/stack/hbase/bin/../lib/hbase-prefix-tree-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-procedure-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-protocol-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-resource-bundle-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-rest-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-server-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-server-2.0.0-SNAPSHOT-tests.jar:/home/stack/hbase/bin/../lib/hbase-shell-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-spark-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-thrift-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/htrace-core-3.1.0-incubating.jar:/home/stack/hbase/bin/../lib/httpclient-4.2.5.jar:/home/stack/hbase/bin/../lib/httpcore-4.4.1.jar:/home/stack/hbase/bin/../lib/jackson-core-asl-1.9.13.jar:/home/stack/hbase/bin/../lib/jackson-jaxrs-1.9.13.jar:/home/stack/hbase/bin/../lib/jackson-mapper-asl-1.9.13.jar:/home/stack/hbase/bin/../lib/jackson-xc-1.9.13.jar:/home/stack/hbase/bin/../lib/jamon-runtime-2.4.1.jar:/home/stack/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/home/stack/hbase/bin/../lib/jasper-runtime-5.5.23.jar:/home/stack/hbase/bin/../lib/javax.inject-1.jar:/home/stack/hbase/bin/../lib/java-xmlbuilder-0.4.jar:/home/stack/hbase/bin/../lib/jaxb-api-2.2.2.jar:/home/stack/hbase/bin/../lib/jaxb-impl-2.2.3-1.jar:/home/stack/hbase/bin/../lib/jcodings-1.0.8.jar:/home/stack/hbase/bin/../lib/jersey-client-1.9.jar:/home/stack/hbase/bin/../lib/jersey-core-1.9.jar:/home/stack/hbase/bin/../lib/jersey-guice-1.9.jar:/home/stack/hbase/bin/../lib/jersey-json-1.9.jar:/home/stack/hbase/bin/../lib/jersey-server-1.9.jar:/home/stack/hbase/bin/../lib/jets3t-0.9.0.jar:/home/stack/hbase/bin/../lib/jettison-1.3.3.jar:/home/stack/hbase/bin/../lib/jetty-6.1.26.jar:/home/stack/hbase/bin/../lib/jetty-sslengine-6.1.26.jar:/home/stack/hbase/bin/../lib/jetty-util-6.1.26.jar:/home/stack/hbase/bin/../lib/joni-2.1.2.jar:/home/stack/hbase/bin/../lib/jruby-complete-1.6.8.jar:/home/stack/hbase/bin/../lib/jsch-0.1.42.jar:/home/stack/hbase/bin/../lib/jsp-2.1-6.1.14.jar:/home/stack/hbase/bin/../lib/jsp-api-2.1-6.1.14.jar:/home/stack/hbase/bin/../lib/jsr305-1.3.9.jar:/home/stack/hbase/bin/../lib/junit-4.12.jar:/home/stack/hbase/bin/../lib/leveldbjni-all-1.8.jar:/home/stack/hbase/bin/../lib/libthrift-0.9.3.jar:/home/stack/hbase/bin/../lib/log4j-1.2.17.jar:/home/stack/hbase/bin/../lib/metrics-core-3.1.2.jar:/home/stack/hbase/bin/../lib/nekohtml-1.9.12.jar:/home/stack/hbase/bin/../lib/netty-all-4.0.30.Final.jar:/home/stack/hbase/bin/../lib/paranamer-2.3.jar:/home/stack/hbase/bin/../lib/protobuf-java-2.5.0.jar:/home/stack/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/home/stack/hbase/bin/../lib/servlet-api-2.5.jar:/home/stack/hbase/bin/../lib/slf4j-api-1.7.7.jar:/home/stack/hbase/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/stack/hbase/bin/../lib/snappy-java-1.1.2.jar:/home/stack/hbase/bin/../lib/spymemcached-2.11.6.jar:/home/stack/hbase/bin/../lib/xalan-2.7.0.jar:/home/stack/hbase/bin/../lib/xercesImpl-2.9.1.jar:/home/stack/hbase/bin/../lib/xml-apis-1.3.03.jar:/home/stack/hbase/bin/../lib/xml-apis-ext-1.3.04.jar:/home/stack/hbase/bin/../lib/xmlenc-0.52.jar:/home/stack/hbase/bin/../lib/xom-1.2.5.jar:/home/stack/hbase/bin/../lib/xz-1.0.jar:/home/stack/hbase/bin/../lib/zookeeper-3.4.6.jar:/home/stack/hadoop-2.7.3-SNAPSHOT/etc/hadoop:/home/stack/hadoop-2.7.3-SNAPSHOT/share/hadoop/common/lib/*:/home/stack/hadoop-2.7.3-SNAPSHOT/share/hadoop/common/*:/home/stack/hadoop-2.7.3-SNAPSHOT/share/hadoop/hdfs:/home/stack/hadoop-2.7.3-SNAPSHOT/share/hadoop/hdfs/lib/*:/home/stack/hadoop-2.7.3-SNAPSHOT/share/hadoop/hdfs/*:/home/stack/hadoop-2.7.3-SNAPSHOT/share/hadoop/yarn/lib/*:/home/stack/hadoop-2.7.3-SNAPSHOT/share/hadoop/yarn/*:/home/stack/hadoop-2.7.3-SNAPSHOT/share/hadoop/mapreduce/lib/*:/home/stack/hadoop-2.7.3-SNAPSHOT/share/hadoop/mapreduce/*:/home/stack/hadoop/contrib/capacity-scheduler/*.jar
{code}
> HBase Backup/Restore Based on HBase Snapshot
> --------------------------------------------
>
> Key: HBASE-7912
> URL: https://issues.apache.org/jira/browse/HBASE-7912
> Project: HBase
> Issue Type: Sub-task
> Reporter: Richard Ding
> Assignee: Vladimir Rodionov
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: Backup-and-Restore-Apache_19Sep2016.pdf,
> Backup-and-Restore-Apache_9Sep2016.pdf, HBaseBackupAndRestore - v0.8.pdf,
> HBaseBackupAndRestore -0.91.pdf, HBaseBackupAndRestore-v0.9.pdf,
> HBaseBackupAndRestore.pdf, HBaseBackupRestore-Jira-7912-DesignDoc-v1.pdf,
> HBaseBackupRestore-Jira-7912-DesignDoc-v2.pdf,
> HBaseBackupRestore-Jira-7912-v4.pdf, HBaseBackupRestore-Jira-7912-v5 .pdf,
> HBaseBackupRestore-Jira-7912-v6.pdf, HBase_BackupRestore-Jira-7912-CLI-v1.pdf
>
>
> Finally, we completed the implementation of our backup/restore solution, and
> would like to share with community through this jira.
> We are leveraging existing hbase snapshot feature, and provide a general
> solution to common users. Our full backup is using snapshot to capture
> metadata locally and using exportsnapshot to move data to another cluster;
> the incremental backup is using offline-WALplayer to backup HLogs; we also
> leverage global distribution rolllog and flush to improve performance; other
> added-on values such as convert, merge, progress report, and CLI commands. So
> that a common user can backup hbase data without in-depth knowledge of hbase.
> Our solution also contains some usability features for enterprise users.
> The detail design document and CLI command will be attached in this jira. We
> plan to use 10~12 subtasks to share each of the following features, and
> document the detail implement in the subtasks:
> * *Full Backup* : provide local and remote back/restore for a list of tables
> * *offline-WALPlayer* to convert HLog to HFiles offline (for incremental
> backup)
> * *distributed* Logroll and distributed flush
> * Backup *Manifest* and history
> * *Incremental* backup: to build on top of full backup as daily/weekly backup
> * *Convert* incremental backup WAL files into hfiles
> * *Merge* several backup images into one(like merge weekly into monthly)
> * *add and remove* table to and from Backup image
> * *Cancel* a backup process
> * backup progress *status*
> * full backup based on *existing snapshot*
> *-------------------------------------------------------------------------------------------------------------*
> *Below is the original description, to keep here as the history for the
> design and discussion back in 2013*
> There have been attempts in the past to come up with a viable HBase
> backup/restore solution (e.g., HBASE-4618). Recently, there are many
> advancements and new features in HBase, for example, FileLink, Snapshot, and
> Distributed Barrier Procedure. This is a proposal for a backup/restore
> solution that utilizes these new features to achieve better performance and
> consistency.
>
> A common practice of backup and restore in database is to first take full
> baseline backup, and then periodically take incremental backup that capture
> the changes since the full baseline backup. HBase cluster can store massive
> amount data. Combination of full backups with incremental backups has
> tremendous benefit for HBase as well. The following is a typical scenario
> for full and incremental backup.
> # The user takes a full backup of a table or a set of tables in HBase.
> # The user schedules periodical incremental backups to capture the changes
> from the full backup, or from last incremental backup.
> # The user needs to restore table data to a past point of time.
> # The full backup is restored to the table(s) or to different table name(s).
> Then the incremental backups that are up to the desired point in time are
> applied on top of the full backup.
> We would support the following key features and capabilities.
> * Full backup uses HBase snapshot to capture HFiles.
> * Use HBase WALs to capture incremental changes, but we use bulk load of
> HFiles for fast incremental restore.
> * Support single table or a set of tables, and column family level backup and
> restore.
> * Restore to different table names.
> * Support adding additional tables or CF to backup set without interruption
> of incremental backup schedule.
> * Support rollup/combining of incremental backups into longer period and
> bigger incremental backups.
> * Unified command line interface for all the above.
> The solution will support HBase backup to FileSystem, either on the same
> cluster or across clusters. It has the flexibility to support backup to
> other devices and servers in the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
