View source for Installing Hadoop 2.2.0 cluster on Ubuntu 12.04 x86 64 Desktops

[[Main Page|Home]] > [[Ubuntu]] > [[Ubuntu Hadoop cluster setup|Hadoop cluster setup]] > [[Installing Hadoop 2.2.0 cluster on Ubuntu 12.04 x86 64 Desktops]]

To install hadoop 2.2.0 cluster on Ubuntu 12.04 x86_64 Desktops or VMs use:
# Setup proper hostnames on all machines using:
## Edit file '<tt>/etc/hostname</tt>' and enter name '<tt>master</tt>' on master and '<tt>slave1</tt>', '<tt>slave2</tt>', etc. on slaves.
## Find out LAN IP address of master and slaves using '<tt>ifconfig</tt>' command
## Edit file '<tt>/etc/hosts</tt>' and associate LAN IPs with master, slave1 and slave2 on all three machines.  Note that you should be able to ping slave using '<tt>ping slave1</tt>', '<tt>ping slave2</tt>', etc. from master and similarly you should be ping master using '<tt>ping master</tt>' from slave.  
## Update hostname on running machine using '<tt>sudo hostname master</tt>', '<tt>sudo hostname slave1</tt>', <tt>sudo hostname slave2</tt>' etc. 
## Verify that both commands '<tt>hostname</tt>' and '<tt>hostname --fqdn</tt>' return master or slave1 or slave2 respectively.
# Reboot all nodes.  Without reboot the next step of installing java will not succeed with 'No protocol specified', 'Exception in class main' errors due to problem with X11 connection.
# Install Java on all nodes (including master and slaves) using [[Installing Java on Ubuntu 12.04 x86 64 Desktop]].  Install java in same folder such as /opt in all nodes.
# Create user account and group for hadoop using:
#:<pre>
#::      sudo groupadd hadoop
#::      sudo useradd hadoop -b /home -g hadoop -mkU -s /bin/bash
#::      cd /home/hadoop
#::      sudo cp -rp /etc/skel/.[^.]* .
#::      sudo chown -R hadoop:hadoop .
#::      sudo chmod -R o-rwx .
#:</pre>
#::on all nodes.  Node that hadoop user name and group name should match on all nodes.
# Install openssh-server on all nodes using:
#:<pre>
#::      sudo apt-get -y install openssh-server
#:</pre>
# Configure password for 'hadoop' user on all three machines using:
#:<pre>
#::      sudo passwd hadoop
#:</pre>
# Setup password-less ssh from hadoop user of master to hadoop user of master itself and all slaves using:
#:<pre>
#::      sudo su - hadoop
#::      ssh-keygen
#::      cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
#::      chmod 0600 ~/.ssh/authorized_keys
#::      ssh-copy-id hadoop@slave1
#::      ssh-copy-id hadoop@slave2
#::      
#::      #To test configuration, should echo hadoop
#::      ssh hadoop@master "echo $USER"
#::      ssh hadoop@slave1 "echo $USER"
#::      ssh hadoop@slave2 "echo $USER"
#::      exit
#:</pre>
#:: Slave to master password-less is not required.
# Download hadoop source from one of the mirrors linked at https://www.apache.org/dyn/closer.cgi/hadoop/common/   Download the latest stable .tar.gz release from stable folder.  (Ex hadoop-2.2.0.tar.gz).  Copy the same hadoop to slaves using something similar to '<tt>rsync -vaH hadoop-* hadoop@slave1:</tt>'
# Extract hadoop sources in '<tt>/opt/hadoop</tt>' and make hadoop:hadoop its owner:
#:<pre>
#::      sudo mkdir /opt/hadoop 
#::      cd /opt/hadoop/ 
#::      sudo tar xzf <path-to-hadoop-source> 
#::      sudo mv hadoop-2.2.0 hadoop 
#::      sudo chown -R hadoop:hadoop . 
#:</pre>
#::Note that hadoop should be installed at same location in all nodes.
# Configure hadoop cluster setup using these steps on '''all nodes''':
## Login as user hadoop:
##:<pre>
##::          sudo su - hadoop 
##:</pre>
## Edit '~/.bashrc' and append
##:<pre>
##::	  export JAVA_HOME=/opt/jdk1.7.0_40 
##::	  export HADOOP_INSTALL=/opt/hadoop/hadoop 
##::	  export HADOOP_PREFIX=/opt/hadoop/hadoop 
##::	  export HADOOP_HOME=/opt/hadoop/hadoop
##::	  export PATH=$PATH:$HADOOP_INSTALL/bin 
##::	  export PATH=$PATH:$HADOOP_INSTALL/sbin 
##::	  export HADOOP_MAPRED_HOME=$HADOOP_INSTALL 
##::	  export HADOOP_COMMON_HOME=$HADOOP_INSTALL 
##::	  export HADOOP_HDFS_HOME=$HADOOP_INSTALL 
##::	  export YARN_HOME=$HADOOP_INSTALL 
##::	  export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/native 
##::	  export HADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib" 
##::	  export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop 
##::	  export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop 
##:</pre>
## Change folder to '<tt>/opt/hadoop/hadoop/etc/hadoop</tt>'
## Edit '<tt>hadoop-env.sh</tt>' and set proper value for JAVA_HOME such as '<tt>/opt/jdk1.7.0_40</tt>'. Do not leave it as ${JAVA_HOME} as that does not works.
## Edit '<tt>/opt/hadoop/hadoop/libexec/hadoop-config.sh</tt>' and prepend following line at start of script:
##:<pre>
##::	  export JAVA_HOME=/opt/jdk1.7.0_40 
##:</pre>
## Exit from hadoop user and relogin using '<tt>sudo su - hadoop</tt>'. Check hadoop version using '<tt>hadoop version</tt>' command. 
## Again change folder to '<tt>/opt/hadoop/hadoop/etc/hadoop</tt>'
## Use '<tt>mdkir /opt/hadoop/tmp</tt>'
## Edit '<tt>core-site.xml</tt>' and add following between &lt;configuration&gt; and &lt;/configuration&gt;:
##:<pre>
##::          <property> 
##::            <name>fs.default.name</name> 
##::            <value>hdfs://master:9000</value> 
##::          </property> 
##::          <property> 
##::            <name>hadoop.tmp.dir</name> 
##::            <value>/opt/hadoop/tmp</value> 
##::          </property> 
##:</pre>
## Setup folders for HDFS using:
##:<pre>
##::           cd ~ 
##::           mkdir -p mydata/hdfs/namenode 
##::           mkdir -p mydata/hdfs/datanode 
##::           cd /opt/hadoop/hadoop/etc/hadoop 
##:</pre>
## Edit '<tt>hdfs-site.xml</tt>' and add following between &lt;configuration&gt; and &lt;/configuration&gt;
##:<pre>
##::	   <property>
##::	     <name>dfs.replication</name>
##::	     <value>2</value>
##::	   </property>
##::	   <property>
##::	     <name>dfs.permissions</name>
##::	     <value>false</value>
##::	   </property>
##::   	   <property>
##::   	     <name>dfs.namenode.name.dir</name>
##::   	     <value>file:/home/hadoop/mydata/hdfs/namenode</value>
##::   	   </property>
##::   	   <property>
##::   	     <name>dfs.datanode.data.dir</name>
##::   	     <value>file:/home/hadoop/mydata/hdfs/datanode</value>
##::   	   </property>
##:</pre>
## Copy mapred-site.xml template using '<tt>cp mapred-site.xml.template mapred-site.xml</tt>'
## Edit '<tt>mapred-site.xml</tt>' and add following between &lt;configuration&gt; and &lt;/configuration&gt;:
##:<pre>
##::	   <property>
##::	     <name>mapreduce.framework.name</name>
##::	     <value>yarn</value>
##::	   </property>
##:</pre>
## Edit '<tt>yarn-site.xml</tt>' and add following between &lt;configuration&gt; and &lt;/configuration&gt;:
##:<pre>
##::	   <property>
##::	     <name>yarn.nodemanager.aux-services</name>
##::  	     <value>mapreduce_shuffle</value>
##::	   </property>
##::	   <property>
##::	     <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
##::	     <value>org.apache.hadoop.mapred.ShuffleHandler</value>
##::	   </property>
##::	   <property>
##::	     <name>yarn.resourcemanager.resource-tracker.address</name>
##::	     <value>master:8025</value>
##::	   </property>
##::	   <property>
##::	     <name>yarn.resourcemanager.scheduler.address</name>
##::	     <value>master:8030</value>
##::	   </property>
##::	   <property>
##::	     <name>yarn.resourcemanager.address</name>
##::	     <value>master:8040</value>
##::	   </property>
##:</pre>
## Format all namenodes master, slave1, slave2, etc. using '<tt>hdfs namenode -format</tt>'
# Do following '''only on master''' machine:
## Edit '<tt>slaves</tt>' files so that it contains:
##:<pre>
##::	  slave1
##::	  slave2
##:</pre>
##::If master is also expected to serve as datanode (store hdfs files) then add '<tt>master</tt>' to the slaves file as well.
## Run '<tt>start-dfs.sh</tt>' and '<tt>stary-yarn.sh</tt>' commands
## Run '<tt>jps</tt>' and verify on master 'ResourceManager', 'NameNode' and 'SecondaryNameNode' are running.  
##* Run '<tt>jps</tt>' on slaves and verify that 'NodeManager' and 'DataNode' are running.
## Access NameNode at http://master:50070 and ResourceManager at http://master:8088
# Run sample map reduce job using: 
## Setup input file for wordcount using:
##:<pre>
##::cd ~
##::mkdir in
##::cat &gt; in/file <<EOF
##::This is one line
##::This is another one
##::EOF
##:</pre>
## Add input directory to HDFS:
##:<pre>
##::	  hdfs dfs -copyFromLocal in /in
##:</pre>
## Run wordcount example provided:
##:<pre>
##::	  hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount /in /out	  
##:</pre>
## Check the output:
##:<pre>
##::	  hdfs dfs -cat /out/*
##:</pre>
# Stop cluster using:
#:<pre>
#::    stop-yarn.sh
#::    stop-dfs.sh
#:</pre>


Steps learned from http://raseshmori.wordpress.com/2012/10/14/install-hadoop-nextgen-yarn-multi-node-cluster/ and https://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Release-Notes/cdh5rn_topic_3_3.html and [[Installing Hadoop 2.2.0 on single Ubuntu 12.04 x86_64 Desktop]]




[[Main Page|Home]] > [[Ubuntu]] > [[Ubuntu Hadoop cluster setup|Hadoop cluster setup]] > [[Installing Hadoop 2.2.0 cluster on Ubuntu 12.04 x86 64 Desktops]]