Monday, July 29, 2013

Connecting to a remote HBase Server from the Client's Command Line

Goal: 
Using Bulk Loading feature of Hbase to insert data into it from a client using only the command line.

What's so special about this post:
1. I spent one whole weekend figuring out how to do it.
2. I faced a lot of problems, mostly configuration issues and no blog/hbase trouble shooting page covered all the issues I went through.
3. I will try to cover basics about how HBase/ZooKeeper works, how to solve issues which I faced.
4. RTFM will mostly not work as hbase documentation sucks, no really!!!


HOW HBASE WORKS!
  1. Hbase works on the Master-Slave architecture, the master is called HMaster and the slave is called the HRegionServer(..yes Server).
  2. The HRegion Server holds HRegions. An HRegion is a block which holds data about the rows of the table).
  3. Each HRegion has a limited data threshold and once it crosses that threshold, the HRegion is informed and it will undergo a split in 2 regions. Zookeeper is then notified about this change.
  4. Zookeper is the guy who accepts connections from clients who ask for the resources. Zookeeper keeps track of the HRegion Servers and also knows where the root region is located (will explain what root region is in some time).
  5. The Client firsts connects to the zookeeper cluster/ensemble asking the whereabouts of the region servers.
  6. For the clients to figure the HRegion server (to fetch / insert data), hbase provides two catalog tables, -ROOT- and .META.
  7. The -ROOT- table has data about the regions in the .META. tables. -ROOT- table is stored in the zookeeper. The matching .META. region is then pulled from the -ROOT- table. Finally the user region is presented to the client from the .META. table.
  8. The client caches the -ROOT- .META. UserRegion and then performs its operations on them, but incase of HRegion Server crashes, it goes one level up to .META. and if that is also corrupted, then it goes to the -ROOT- table in its quest for the User Region.
WHY ZOOKEEPER IS IMP!
  1. The writes arriving for a specific table are first written to a commit log(as called write ahead log, aka WAL). 
  2. Now the zookeeper checks for a HRegion Server which has space on it and pulls data from commit log to a location called memstore. 
  3. Once the memstore is full, it creates the HFile(this file is hbase compliant, for efficient data loading into hbase) and flushes it into HDFS.
  4. If a HRegion Server crashes, the Zookeeper takes care of this and helps in HRegion failover.
  5. Also the Zookeeper Ensemble takes this processing out of the Hbase Master which just has configuration files and knows about the HRegionServers.

This is the setup I used for testing purposes. All boxes had 4G RAM and enough processing power / disk space on them.


Here are the properties for hbase, I am skipping the hadoop properties as they are quite straight forward.
<Property> :: <Value>
Master(hadoop1) Configurations:
1. hbase-site.xml:
- hbase.rootdir :: hdfs://hadoop1:9000/hbase (base dir in hdfs, shared by all the slaves)
- hbase.cluster.distributed :: true (we want to run hbase in a fully distributed mode)
- hbase.zookeeper.quorum :: hadoop1,hadoop2,hadoop3 (HRegion server cluster)
- hbase.zookeeper.property.dataDir :: /home/hadoop/hbase/zookeeper (base dir for zookeeper info)
- hbase.zookeeper.property.clientPort :: 2181 (The port at which the clients will connect. By default).

2. regionservers
- hadoop1
- hadoop2
- hadoop3
- hadoop4 (All of them line separated)

Hbase Master did not manage Zoo Keeper. Zoo Keeper managed itself :).

Slave(hadoop2, hadoop3, hadoop4) Configurations :
1. hbase-site.xml : same as that of the master.
2. regionservers : same as that of the master.

Client Configuration :
1.  hbase-site.xml:
- hbase.rootdir :: hdfs://hadoop1:9000/hbase
- hbase.cluster.distributed :: true
- hbase.master :: hadoop1:60000
- hbase.zookeeper.quorum :: hadoop1 (should be given all the region servers but its ok, as the zookeeper is running on the same box as that of the HMaster).
- hbase.zookeeper.property.dataDir :: /data/zookeeper

2. Nothing in region servers.

Important Steps to take care of : 
There were various issues when I tried running it, all of them are solved here.
Issues:
1. Disable ipv6 on both client and master : hadoop might bind to ipv6 addresses[0.0.0.0], we don't want that. In this case the error is often garbled, after some searching I got a clue: http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/#disabling-ipv6
in /etc/sysctl.conf
# disable ipv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

check : cat /proc/sys/net/ipv6/conf/all/disable_ipv6
A return value of 0 means IPv6 is enabled, a value of 1 means disabled (that’s what we want).

2. Turn off hadoop in non safe mode as soon as you strart it : otherwise hbase won't be able to hit the hdfs. I just saw the logs here.
hadoop dfsadmin -safemode leave

3. /etc/hosts file is very important because when the client connects to the zookeeper/hmaster, hmaster will return it its own ip and if localhost points to 127.0.0.1 then client will hit its own box which is not correct hence this file should be taken care of like: 
Client:
172.132.45.180  localhost anirudh-laptop
172.132.45.17 user-desktop hadoop1

Master:
172.132.45.17   hadoop1 localhost user-desktop
172.132.45.29   hadoop2
172.132.45.32   hadoop3
172.132.45.177  hadoop4

Slave:
127.0.0.1       localhost
127.0.1.1       user29-desktop
172.132.45.17   hadoop1
172.132.45.29   hadoop2
172.132.45.32   hadoop3
172.132.45.177  hadoop4

Error I faced was something like :
12/01/02 10:02:11 INFO ipc.Client: Retrying connection to server master/192.162.10.10:9000. Already tried 0 time(s).
12/01/02 10:02:12 INFO ipc.Client: Retrying connection to server master/127.0.0.1:9000. Already tried 1 time(s).
12/01/02 10:02:13 INFO ipc.Client: Retrying connection to server master/127.0.0.1:9000. Already tried 2 time(s).
...
Don't wait for it to retry over and over again hoping it will pass, it won't pass. Think about what you might have gone wrong :)

4. After you start Hadoop Make sure your jps looks like this on master:
20349 Jps
19636 DataNode
19357 NameNode
19915 SecondaryNameNode
20001 JobTracker
20279 TaskTracker

5. After you start HBase Make sure your jps looks like this on master:
19636 DataNode
19357 NameNode
26693 Main
19915 SecondaryNameNode
30210 Jps
26371 HMaster
20001 JobTracker
20279 TaskTracker

6. Make sure you see the logs for errors after hadoop and hbase start. Actually just keep them on, will help you a lot. Delete all logs files before starting afresh, will help you pin down the logs easily. Most important!!!

7. Make sure you are able to log into hbase shell and run some commands like list, scan, etc.

Finally here is how I achieved my goal:
- masterNodeIp is hadoop1.
- port is 9000.
- destination table is test_data. You don't need to mention the column family, hbase is smart enough to understand it from the store file command.
1. Creating a input directory in hdfs to save local file.
${HADOOP_HOME}/bin/hadoop fs -fs 
hdfs://#{masterNodeIp}:#{port} -mkdir #{inputDir}
            
2. Copy local tsv file into hdfs.
${HADOOP_HOME}/bin/hadoop fs -put 
#{loadfilePath} hdfs://#{masterNodeIp}:#{port}#{inputDir}

3. Prepare data for bulk loading
${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.mapreduce.ImportTsv 
-Dimporttsv.columns=#{hbaseImportColumnClause} 
-Dimporttsv.bulk.output=hdfs://#{masterNodeIp}:#{port}#{ouputDir} 
#{destinationTable} 
hdfs://#{masterNodeIp}:#{port}#{inputDir}#{fileName}

4. Load data into hbase
${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles 
#{ouputDir} 
#{destinationTable}

That's all people. Comment/mail me if you have any problems understanding/issues.

Posted by at No comments:
Subscribe to: Posts (Atom)