- update /etc/hosts to remove 127.0.1.1
- install may take long time
- $ sudo -u hdfs hadoop fs -mkdir /tmp
$ sudo -u hdfs hadoop fs -chmod -R 1777 /tmp - $ sudo -u hdfs hadoop fs -mkdir /tmp/mapred
$ sudo -u hdfs hadoop fs -chmod -R 1777 /tmp/mapred - $ sudo -u hdfs hadoop fs -chmod -R 1777 /
- data node cannot register to namenode
<property><name>dfs.hosts</name><value></value><final>true</final></property> - java.io.IOException: Cannot create an instance of InputSplit class = org.apache.hadoop.hive.hbase.HBaseSplit:org.apache.hadoop.hive.hbase.HBaseSplit
<property><name>hive.aux.jars.path</name> <value>file:///home/hadoop/cdh3/hive-0.7.0-cdh3u0/lib/hive-hbase-handler-0.7.0-cdh3u0.jar,file:///home/hadoop/cdh3/hive-0.7.0-cdh3u0/lib/hbase-0.90.1-cdh3u0.jar,file:///home/hadoop/cdh3/hive-0.7.0-cdh3u0/lib/zookeeper-3.3.1.jar</value></property>The storage handler is built as an independent module, hive-hbase-handler-x.y.z.jar, which must be available on the Hive client auxpath, along with HBase, Guava and ZooKeeper jars. It also requires the correct configuration property to be set in order to connect to the right HBase master. See the HBase documentation for how to set up an HBase cluster.
Here's an example using CLI from a source build environment, targeting a single-node HBase server. (Note that the jar locations and names have changed in Hive 0.9.0, so for earlier releases, some changes are needed.)
$HIVE_SRC/build/dist/bin/hive –auxpath $HIVE_SRC/build/dist/lib/hive-hbase-handler-0.9.0.jar,$HIVE_SRC/build/dist/lib/hbase-0.92.0.jar,$HIVE_SRC/build/dist/lib/zookeeper-3.3.4.jar,$HIVE_SRC/build/dist/lib/guava-r09.jar -hiveconf hbase.master=hbase.yoyodyne.com:60000
Here's an example which instead targets a distributed HBase cluster where a quorum of 3 zookeepers is used to elect the HBase master:
$HIVE_SRC/build/dist/bin/hive –auxpath $HIVE_SRC/build/dist/lib/hive-hbase-handler-0.9.0.jar,$HIVE_SRC/build/dist/lib/hbase-0.92.0.jar,$HIVE_SRC/build/dist/lib/zookeeper-3.3.4.jar,$HIVE_SRC/build/dist/lib/guava-r09.jar -hiveconf hbase.zookeeper.quorum=zk1.yoyodyne.com,zk2.yoyodyne.com,zk3.yoyodyne.com
The handler requires Hadoop 0.20 or higher, and has only been tested with dependency versions hadoop-0.20.x, hbase-0.92.0 and zookeeper-3.3.4. If you are not using hbase-0.92.0, you will need to rebuild the handler with the HBase jar matching your version, and change the –auxpath above accordingly.