Hadoop 简要安装(一)

初始环境 vagrant box https://github.com/CommanderK5/packer-centos-template/releases/download/0.6.7/vagrant-centos-6.7.box

计划构建

node1 : HDFS NameNode + Spark Master //对外服务器

node2 : YARN ResourceManager + JobHistoryServer + ProxyServer //调度服务器 job

node3 : HDFS DataNode + YARN NodeManager + Spark Slave //节点服务器1

node4 : HDFS DataNode + YARN NodeManager + Spark Slave //节点服务器2

Hadoop 个人理解。

Hadoop 只是一个数据框架。基于分布式文件(HDSF)上的一种协议。不要被神话吓到了。就和我们编写程序一样。只是加载了几个类。方便用而已。

Hadoop 安装准备

1.首先安装 JAVA SDK

sudo yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel

vi /etc/profile //新增JDK的环境变量

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk.x86_64/

[root@localhost vagrant]# java -version
openjdk version "1.8.0_71"
OpenJDK Runtime Environment (build 1.8.0_71-b15)
OpenJDK 64-Bit Server VM (build 25.71-b15, mixed mode)

[root@localhost vagrant]# $JAVA_HOME/bin/java -version 
openjdk version "1.8.0_71"
OpenJDK Runtime Environment (build 1.8.0_71-b15)
OpenJDK 64-Bit Server VM (build 25.71-b15, mixed mode)

2.创建hadoop 用户

su root  //切换ROOT用户
useradd -m hadoop -s /bin/bash //创建hadoop 用户
passwd hadoop //修改密码
visudo  //特殊处理 给 Hadoop 增加管理员权限 (个人测试设置,生产环境不敢这样设置)
------------------------------------------------------
>## Allow root to run any commands anywhere
>root    ALL=(ALL)       ALL
>hadoop  ALL=(ALL)       ALL
------------------------------------------------------

3.使用vagrant 打包虚拟机

sudo rm -f /etc/udev/rules.d/70-persistent-net.rules    //删除liuunx 网卡配置
vagrant package --output centos-java-hadoop.box         //母机目录下打包vagrant box

建立一个基础镜像未来将会使用该基础进行搭建HADOOP节点。

4.vagrant 加载基础镜像

vagrant box add hadoop-base ./centos-java-hadoop.box

5.创建 vagrant 集群

    Vagrant.configure(2) do |config|
        config.vm.define "service" do |node|
             node.vm.box = "hadoop-base"
          node.vm.network "private_network", ip: "192.168.31.30"
          node.vm.synced_folder "./data", "/hadoop_data"
          node.vm.provider "virtualbox" do |v|
            v.name = "service"
            v.customize ["modifyvm", :id, "--memory", "1024"]
          end
        end
        config.vm.define "service-job" do |node|
         node.vm.box = "hadoop-base"
          node.vm.network "private_network", ip: "192.168.31.31"
          node.vm.synced_folder "./data", "/hadoop_data"
          node.vm.provider "virtualbox" do |v|
            v.name = "service-job"
            v.customize ["modifyvm", :id, "--memory", "1024"]
          end
        end
        config.vm.define "service-node-1" do |node|
          node.vm.box = "hadoop-base"
          node.vm.network "private_network", ip: "192.168.31.36"
          node.vm.synced_folder "./data", "/hadoop_data"
          node.vm.provider "virtualbox" do |v|
            v.name = "service-node-1"
            v.customize ["modifyvm", :id, "--memory", "1024"]
          end
        end
        config.vm.define "service-node-2" do |node|
          node.vm.box = "hadoop-base"
          node.vm.network "private_network", ip: "192.168.31.37"
          node.vm.synced_folder "./data", "/hadoop_data"
          node.vm.provider "virtualbox" do |v|
            v.name = "service-node-2"
            v.customize ["modifyvm", :id, "--memory", "1024"]
          end
        end
    end