brew cleanup hadoop
然后更新homebrew formulaebrew update brew upgrade brew cleanup
检查版本信息brew info hadoop brew info apache-spark brew info sbt brew info scala
如果以上程序没有安装，需要使用brew install app 进行安装。
安装环境 安装hadoop
brew install hadoop
安装sparkbrew install apache-spark scala sbt
设置环境变量 使用vim编辑~/.bash_profile，将以下内容贴到最后
# set environment variables export JAVA_HOME=$(/usr/libexec/java_home) export HADOOP_HOME=/usr/local/Cellar/hadoop/2.5.1 export HADOOP_CONF_DIR=$HADOOP_HOME/libexec/etc/hadoop export SCALA_HOME=/usr/local/Cellar/apache-spark/1.1.0 # set path variables export PATH=$PATH:$HADOOP_HOME/bin:$SCALA_HOME/bin # set alias start & stop scripts alias hstart=$HADOOP_HOME/sbin/start-dfs.sh;$HADOOP_HOME/sbin/start-yarn.sh alias hstop=$HADOOP_HOME/sbin/stop-dfs.sh;$HADOOP_HOME/sbin/stop-yarn.shHadoop必须要使ssh生效，设置ssh
配置文件路径：/etc/sshd_config
生成秘钥：sh-3.2# sudo ssh-keygen -t rsa
Generating public/private rsa key pair. Enter file in which to save the key (/var/root/.ssh/id_rsa): 输入/var/root/.ssh/id_rsa Enter passphrase (empty for no passphrase): ［直接回车］ Enter same passphrase again: ［直接回车］ Your identification has been saved in /var/root/.ssh/id_rsa. Your public key has been saved in /var/root/.ssh/id_rsa.pub. key fingerprint is: 97:e9:5a:5e:91:52:30:63:9e:34:1a:6f:24:64:75:af root@cuican.local The key's randomart image is: +--[ RSA 2048]----+ | .=.X . | | . X B . | | . = . . | | . + o | | S = E | | o . . | | o . | | + . | | . . | +-----------------+ 修改配置文sudo vim /etc/ssh/sshd_config
Port 22 #AddressFamily any #ListenAddress 0.0.0.0 #ListenAddress :: # The default requires explicit activation of protocol 1 Protocol 2 # HostKey for protocol version 1 #HostKey /etc/ssh/ssh_host_key # HostKeys for protocol version 2 #HostKey /etc/ssh/ssh_host_rsa_key #HostKey /etc/ssh/ssh_host_dsa_key #HostKey /etc/ssh/ssh_host_ecdsa_key HostKey /var/root/.ssh/id_rsa # Lifetime and size of ephemeral version 1 server key KeyRegenerationInterval 1h ServerKeyBits 1024 # Logging # obsoletes QuietMode and FascistLogging SyslogFacility AUTHPRIV #LogLevel INFO # Authentication: LoginGraceTime 2m PermitRootLogin yes StrictModes yes #MaxAuthTries 6 #MaxSessions 10 RSAAuthentication yes PubkeyAuthentication yes 启动ssh服务which sshd //查找sshd的位置。
Mac 上sshd的位置在 /usr/sbin/sshd
在终端输入sudo /usr/sbin/sshd即可启动sshd服务。
ssh-keygen -t rsa cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
配置Hadoop 到hadoop的安装路径
cd usr/local/Cellar/hadoop/2.5.1/libexec/
编辑etc/hadoop/hadoop-env.sh
# this fixes the "scdynamicstore" warning export HADOOP_OPTS="$HADOOP_OPTS -Djava.security.krb5.realm= -Djava.security.krb5.kdc="编辑etc/hadoop/core-site.xml
<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value> </property> </configuration>编辑etc/hadoop/hdfs-site.xml
<configuration> <property> <name>dfs.replication</name> <value>1</value> </property> </configuration>编辑etc/hadoop/mapred-site.xml
<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration>编辑etc/hadoop/yarn-site.xml
<configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration>开始启用Hadoop 移动到Hadoop的root directory
cd /usr/local/Cellar/hadoop/2.5.1
格式化Hadoop HDFS./bin/hdfs namenode -format
启动NameNode和DataNode daemon./sbin/start-dfs.sh
从网页中查看http://localhost:50070/
启动ResourceManager和NodeManager daemon./sbin/start-yarn.sh
检查所有的守护线程是不是已经在运行jps
从网页中查看ResourceManagerhttp://localhost:8088/
创建HDFS目录./bin/hdfs dfs -mkdir -p /user/{username}
启动一个MapReduce的例子
\#calculate pi ./bin/hadoop jar libexec/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.1.jar pi 10 100启动spark
到Spark的安装目录
cd /usr/local/Cellar/apache-spark/1.1.0
启动Spark的例子./bin/run-example SparkPi
在网页中查看Spark任务http://localhost:4040/
也可以使用Spark-submit来提交任务
# pattern to launch an application in yarn-cluster mode ./bin/spark-submit --class <path.to.class> --master yarn-cluster [options] <app.jar> [options] # run example application (calculate pi) ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster libexec/lib/spark-examples-*.jar结束
转载于:https://www.cnblogs.com/wangjiyong/p/5173090.html
