文档章节

hadoop学习笔记

清风_
 清风_
发布于 2016/12/10 19:29
字数 2256
阅读 65
收藏 0
点赞 1
评论 0
* vi /etc/hosts
10.204.211.241 JZYH-COLLECTOR-LTEMR3-OSS
* vi /etc/sysconfig/network
#主机名不要使用下划线
127.0.0.1   localhost localhost4 localhost4.localdomain4
** Single Node Cluster
* etc/hadoop/core-site.xml:
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://JZYH-COLLECTOR-LTEMR3-OSS:8020</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/opt/disk/backup/soft/hadoop-2.5.0/data/tmp</value>
    </property>
    <property>
        <name>fs.trash.interval</name>
        <value>10080</value>
    </property>
</configuration>
* etc/hadoop/hdfs-site.xml:
<configuration>
    <!--分布式去掉该属性-->
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>JZYH-COLLECTOR-LTEMR3-OSS:50090</value>
    </property>
</configuration>
* 配置免登陆
$ ssh localhost
$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
往远程机器上拷贝:$ ssh-copy-id 地址
* Format the filesystem:
  $ bin/hdfs namenode -format
* Start NameNode daemon and DataNode daemon:
  $ sbin/start-dfs.sh
  或者
  $ sbin/hadoop-daemon.sh start namenode
  $ sbin/hadoop-daemon.sh start secondarynamenode
  $ sbin/hadoop-daemon.sh start datanode
* Browse the web interface for the NameNode; by default it is available at:
	NameNode - http://localhost:50070/
* Make the HDFS directories required to execute MapReduce jobs:
  $ bin/hdfs dfs -mkdir /user
  $ bin/hdfs dfs -mkdir /user/<username>
* Copy the input files into the distributed filesystem:
  $ bin/hdfs dfs -put etc/hadoop input
* Run some of the examples provided:
  $ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar grep input output 'dfs[a-z.]+'
* Copy the output files from the distributed filesystem to the local filesystem and examine them:
  $ bin/hdfs dfs -get output output
* View the output files on the distributed filesystem:
  $ bin/hdfs dfs -cat output/*
* When you're done, stop the daemons with:
  $ sbin/stop-dfs.sh

**YARN on Single Node
*etc/hadoop/mapred-site.xml:
<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>JZYH-COLLECTOR-LTEMR3-OSS:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>JZYH-COLLECTOR-LTEMR3-OSS:19888</value>
    </property>
</configuration>
*etc/hadoop/yarn-site.xml:
<configuration>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>JZYH-COLLECTOR-LTEMR3-OSS</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
</configuration>
*etc/hadoop/slaves:
JZYH-COLLECTOR-LTEMR3-OSS
*Start ResourceManager daemon and NodeManager daemon:
$ sbin/start-yarn.sh
或者
$ sbin/yarn-daemon.sh start resourcemanager
$ sbin/yarn-daemon.sh start nodemanager
*Browse the web interface for the ResourceManager; by default it is available at:
ResourceManager - http://localhost:8088/
*When you're done, stop the daemons with:
  $ sbin/stop-yarn.sh

*start Mapreduce history daemon
$ sbin/mr-jobhistory-daemon.sh start historyserver
* Aggregation(应用运行完成以后将日志上传到HDFS文件系统)
	etc/hadoop/yarn-site.xml:
<configuration>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>JZYH-COLLECTOR-LTEMR3-OSS</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
    </property>
    <property>
        <name>yarn.log-aggregation.retain-seconds</name>
        <value>3600</value>
    </property>
</configuration>
重启所有服务
$ sbin/yarn-daemon.sh stop resourcemanager
$ sbin/yarn-daemon.sh stop nodemanager
$ sbin/mr-jobhistory-daemon.sh stop historyserver

$ sbin/yarn-daemon.sh start resourcemanager
$ sbin/yarn-daemon.sh start nodemanager
$ sbin/mr-jobhistory-daemon.sh start historyserver


**读取本地文件:bin/hdfs dfs -Dfs.defaultFS=file:/// -ls /
**状态信息:bin/hdfs dfsadmin -report
**安全模式safemode
启用
bin/hdfs dfsadmin -safemode enter
查询
bin/hdfs dfsadmin -safemode get
退出
bin/hdfs dfsadmin -safemode leave

执行wordcount:
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.1.jar wordcount input output
报错:
16/12/11 13:08:01 INFO client.RMProxy: Connecting to ResourceManager at mac/192.168.1.119:8032
16/12/11 13:08:03 INFO input.FileInputFormat: Total input paths to process : 1
16/12/11 13:08:03 INFO mapreduce.JobSubmitter: number of splits:1
16/12/11 13:08:03 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1481432311518_0003
16/12/11 13:08:03 INFO impl.YarnClientImpl: Submitted application application_1481432311518_0003
16/12/11 13:08:03 INFO mapreduce.Job: The url to track the job: http://mac:8088/proxy/application_1481432311518_0003/
16/12/11 13:08:03 INFO mapreduce.Job: Running job: job_1481432311518_0003
16/12/11 13:08:12 INFO mapreduce.Job: Job job_1481432311518_0003 running in uber mode : false
16/12/11 13:08:12 INFO mapreduce.Job:  map 0% reduce 0%
16/12/11 13:08:12 INFO mapreduce.Job: Job job_1481432311518_0003 failed with state FAILED due to: Application application_1481432311518_0003 failed 2 times due to AM Container for appattempt_1481432311518_0003_000002 exited with  exitCode: 127 due to: Exception from container-launch: ExitCodeException exitCode=127: 
ExitCodeException exitCode=127: 
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
	at org.apache.hadoop.util.Shell.run(Shell.java:455)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
查看日志:
hdfs dfs -ls -R /
drwxrwx---   - john supergroup          0 2016-12-11 12:13 /tmp
drwxrwx---   - john supergroup          0 2016-12-10 18:02 /tmp/hadoop-yarn
drwxrwx---   - john supergroup          0 2016-12-11 12:12 /tmp/hadoop-yarn/staging
drwxrwx---   - john supergroup          0 2016-12-10 18:02 /tmp/hadoop-yarn/staging/history
drwxrwx---   - john supergroup          0 2016-12-10 18:02 /tmp/hadoop-yarn/staging/history/done
drwxrwxrwt   - john supergroup          0 2016-12-10 18:02 /tmp/hadoop-yarn/staging/history/done_intermediate
drwx------   - john supergroup          0 2016-12-11 12:12 /tmp/hadoop-yarn/staging/john
drwx------   - john supergroup          0 2016-12-11 13:08 /tmp/hadoop-yarn/staging/john/.staging
drwx------   - john supergroup          0 2016-12-11 12:12 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0001
-rw-r--r--  10 john supergroup     270368 2016-12-11 12:12 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0001/job.jar
-rw-r--r--  10 john supergroup        112 2016-12-11 12:12 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0001/job.split
-rw-r--r--   1 john supergroup         17 2016-12-11 12:12 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0001/job.splitmetainfo
-rw-r--r--   1 john supergroup      80320 2016-12-11 12:13 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0001/job.xml
drwx------   - john supergroup          0 2016-12-11 12:19 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0002
-rw-r--r--  10 john supergroup     270368 2016-12-11 12:19 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0002/job.jar
-rw-r--r--  10 john supergroup        112 2016-12-11 12:19 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0002/job.split
-rw-r--r--   1 john supergroup         17 2016-12-11 12:19 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0002/job.splitmetainfo
-rw-r--r--   1 john supergroup      80320 2016-12-11 12:19 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0002/job.xml
drwx------   - john supergroup          0 2016-12-11 12:51 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0003
-rw-r--r--  10 john supergroup     270368 2016-12-11 12:51 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0003/job.jar
-rw-r--r--  10 john supergroup        112 2016-12-11 12:51 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0003/job.split
-rw-r--r--   1 john supergroup         17 2016-12-11 12:51 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0003/job.splitmetainfo
-rw-r--r--   1 john supergroup      80320 2016-12-11 12:51 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0003/job.xml
drwx------   - john supergroup          0 2016-12-11 12:52 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0004
-rw-r--r--  10 john supergroup     270368 2016-12-11 12:52 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0004/job.jar
-rw-r--r--  10 john supergroup        112 2016-12-11 12:52 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0004/job.split
-rw-r--r--   1 john supergroup         17 2016-12-11 12:52 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0004/job.splitmetainfo
-rw-r--r--   1 john supergroup      80320 2016-12-11 12:52 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0004/job.xml
drwx------   - john supergroup          0 2016-12-11 12:53 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0005
-rw-r--r--  10 john supergroup     270368 2016-12-11 12:53 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0005/job.jar
-rw-r--r--  10 john supergroup        112 2016-12-11 12:53 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0005/job.split
-rw-r--r--   1 john supergroup         17 2016-12-11 12:53 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0005/job.splitmetainfo
-rw-r--r--   1 john supergroup      80320 2016-12-11 12:53 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0005/job.xml
drwx------   - john supergroup          0 2016-12-11 12:59 /tmp/hadoop-yarn/staging/john/.staging/job_1481432311518_0001
-rw-r--r--  10 john supergroup     270368 2016-12-11 12:59 /tmp/hadoop-yarn/staging/john/.staging/job_1481432311518_0001/job.jar
-rw-r--r--  10 john supergroup        112 2016-12-11 12:59 /tmp/hadoop-yarn/staging/john/.staging/job_1481432311518_0001/job.split
-rw-r--r--   1 john supergroup         17 2016-12-11 12:59 /tmp/hadoop-yarn/staging/john/.staging/job_1481432311518_0001/job.splitmetainfo
-rw-r--r--   1 john supergroup      80315 2016-12-11 12:59 /tmp/hadoop-yarn/staging/john/.staging/job_1481432311518_0001/job.xml
drwx------   - john supergroup          0 2016-12-11 13:07 /tmp/hadoop-yarn/staging/john/.staging/job_1481432311518_0002
-rw-r--r--  10 john supergroup     270368 2016-12-11 13:07 /tmp/hadoop-yarn/staging/john/.staging/job_1481432311518_0002/job.jar
-rw-r--r--  10 john supergroup        112 2016-12-11 13:07 /tmp/hadoop-yarn/staging/john/.staging/job_1481432311518_0002/job.split
-rw-r--r--   1 john supergroup         17 2016-12-11 13:07 /tmp/hadoop-yarn/staging/john/.staging/job_1481432311518_0002/job.splitmetainfo
-rw-r--r--   1 john supergroup      80320 2016-12-11 13:07 /tmp/hadoop-yarn/staging/john/.staging/job_1481432311518_0002/job.xml
drwx------   - john supergroup          0 2016-12-11 13:08 /tmp/hadoop-yarn/staging/john/.staging/job_1481432311518_0003
-rw-r--r--  10 john supergroup     270368 2016-12-11 13:08 /tmp/hadoop-yarn/staging/john/.staging/job_1481432311518_0003/job.jar
-rw-r--r--  10 john supergroup        112 2016-12-11 13:08 /tmp/hadoop-yarn/staging/john/.staging/job_1481432311518_0003/job.split
-rw-r--r--   1 john supergroup         17 2016-12-11 13:08 /tmp/hadoop-yarn/staging/john/.staging/job_1481432311518_0003/job.splitmetainfo
-rw-r--r--   1 john supergroup      80320 2016-12-11 13:08 /tmp/hadoop-yarn/staging/john/.staging/job_1481432311518_0003/job.xml
drwxrwxrwt   - john supergroup          0 2016-12-11 12:13 /tmp/logs
drwxrwx---   - john supergroup          0 2016-12-11 12:13 /tmp/logs/john
drwxrwx---   - john supergroup          0 2016-12-11 13:08 /tmp/logs/john/logs
drwxrwx---   - john supergroup          0 2016-12-11 12:13 /tmp/logs/john/logs/application_1481426363207_0001
-rw-r-----   1 john supergroup        509 2016-12-11 12:13 /tmp/logs/john/logs/application_1481426363207_0001/mac_51897
drwxrwx---   - john supergroup          0 2016-12-11 12:19 /tmp/logs/john/logs/application_1481426363207_0002
-rw-r-----   1 john supergroup        509 2016-12-11 12:19 /tmp/logs/john/logs/application_1481426363207_0002/mac_51897
drwxrwx---   - john supergroup          0 2016-12-11 12:51 /tmp/logs/john/logs/application_1481426363207_0003
-rw-r-----   1 john supergroup        509 2016-12-11 12:51 /tmp/logs/john/logs/application_1481426363207_0003/mac_51897
drwxrwx---   - john supergroup          0 2016-12-11 12:52 /tmp/logs/john/logs/application_1481426363207_0004
-rw-r-----   1 john supergroup        509 2016-12-11 12:52 /tmp/logs/john/logs/application_1481426363207_0004/mac_51897
drwxrwx---   - john supergroup          0 2016-12-11 12:53 /tmp/logs/john/logs/application_1481426363207_0005
-rw-r-----   1 john supergroup        509 2016-12-11 12:53 /tmp/logs/john/logs/application_1481426363207_0005/mac_51897
drwxrwx---   - john supergroup          0 2016-12-11 12:59 /tmp/logs/john/logs/application_1481432311518_0001
-rw-r-----   1 john supergroup        509 2016-12-11 12:59 /tmp/logs/john/logs/application_1481432311518_0001/mac_53087
drwxrwx---   - john supergroup          0 2016-12-11 13:07 /tmp/logs/john/logs/application_1481432311518_0002
-rw-r-----   1 john supergroup        509 2016-12-11 13:07 /tmp/logs/john/logs/application_1481432311518_0002/mac_53087
drwxrwx---   - john supergroup          0 2016-12-11 13:08 /tmp/logs/john/logs/application_1481432311518_0003
-rw-r-----   1 john supergroup        509 2016-12-11 13:08 /tmp/logs/john/logs/application_1481432311518_0003/mac_53087
drwx------   - john supergroup          0 2016-12-11 11:50 /user
drwx------   - john supergroup          0 2016-12-11 12:12 /user/john
drwx------   - john supergroup          0 2016-12-11 11:50 /user/john/.Trash
drwx------   - john supergroup          0 2016-12-11 12:12 /user/john/.Trash/Current
drwx------   - john supergroup          0 2016-12-11 11:50 /user/john/.Trash/Current/test
-rw-r--r--   1 john supergroup      15458 2016-12-10 18:43 /user/john/.Trash/Current/test/LICENSE.txt
drwxr-xr-x   - john supergroup          0 2016-12-11 12:08 /user/john/.Trash/Current/test1481429418166
-rw-r--r--   1 john supergroup        101 2016-12-10 19:15 /user/john/.Trash/Current/test1481429418166/NOTICE.txt
drwx------   - john supergroup          0 2016-12-11 12:12 /user/john/.Trash/Current/user
drwx------   - john supergroup          0 2016-12-11 12:12 /user/john/.Trash/Current/user/john
drwxr-xr-x   - john supergroup          0 2016-12-11 12:08 /user/john/.Trash/Current/user/john/output
drwxr-xr-x   - john supergroup          0 2016-12-11 12:08 /user/john/input
-rw-r--r--   1 john supergroup         11 2016-12-11 11:53 /user/john/input/wc.input
hdfs dfs -cat /tmp/logs/john/logs/application_1481432311518_0003/mac_53087
??h??׶9?A@???P	VERSIONAPPLICATION_ACL
MODIFY_APPVIEW_APP APPLICATION_OWNERjohn(&container_1481432311518_0003_01_000001Gstderr48/bin/bash: /bin/java: No such file or directory
stdout0(&container_1481432311518_0003_02_000001Gstderr48/bin/bash: /bin/java: No such file or directory
stdout0
	VERSION*(&container_1481432311518_0003_02_000001none?;?;data:BCFile.indexnone͇

                                                                                     data:TFile.indexnone?Q66data:TFile.metanone?K???h??׶9?A@???PJohndeiMac:Desktop john$ 
找到问题原因,创建软链接解决问题:
sudo ln -s /usr/bin/java /bin/java

运行自己写的jar包
yarn jar jars/mr-wordcount.jar /user/john/input /user/john/wordcount

 

© 著作权归作者所有

共有 人打赏支持
清风_

清风_

粉丝 31
博文 38
码字总数 23170
作品 1
沈阳
程序员

暂无相关文章

Java集合类总结笔记

一、集合类的层次关系 主要容器集合类的特点: ArrayList 一种可以动态增长和缩减的索引序列 LinkedList 一种可以在任何位置进行高效地插入和删除的有序序列 ArrayDeque 一种用循环数组实现的...

edwardGe ⋅ 5分钟前 ⋅ 0

spring RMI远程调用

RMI https://www.cnblogs.com/wdh1995/p/6792407.html

BobwithB ⋅ 10分钟前 ⋅ 0

Jenkins实践2 之基本配置

1 插件管理 系统管理->插件管理 在可选插件中可以自主安装插件 2 管理用户 系统管理->管理用户->新建用户 3 安全配置 系统管理->全局安全配置 授权策略 选择安全矩阵 然后添加现有的用户,赋...

晨猫 ⋅ 10分钟前 ⋅ 0

c++智能指针

1、是一种泛型类,针对指针类型的泛型类,会保存指针 2、重载了符号 *和-> 对智能指针使用这两个符号,相当于对保存的泛型使用这两个符号 3、当智能指针引用计数为0时,会去释放指针指向的资...

国仔饼 ⋅ 11分钟前 ⋅ 0

Spring Boot错误处理机制

1)、SpringBoot默认的错误处理机制 默认效果: 1)、浏览器,返回一个默认的错误页面 浏览器发送请求的请求头: 2)、如果是其他客户端,默认响应一个json数据 原理: 可以参照ErrorMvcAut...

小致dad ⋅ 13分钟前 ⋅ 0

ftp连接不上的终极办法 SFTP

假如FTP由于各种原因就是连不上,那么用SFTP协议吧,使用登录服务器的账号密码。

sskill ⋅ 17分钟前 ⋅ 0

Unity 围绕旋转角度限制(Transform.RotateAround)

在 Unity 中可以利用 Transform.RotateAround 围绕指定物体进行旋转,但某些情况下可能需要对旋转角度进行控制。我是先计算出预设角度大小,然后判断是否在限定角度范围内是则进行旋转。 相关...

大轩 ⋅ 18分钟前 ⋅ 0

阿里沙箱环境支付宝测试demo

阿里支付宝支付和微信支付,包括:阿里沙箱环境支付宝测试demo,支付宝支付整合到spring+springmvc+mybatis环境和微信整合到如上环境,功能非常齐全,只需要修改对应的配置文件即可,帮助文档...

码代码的小司机 ⋅ 21分钟前 ⋅ 0

JDK1.6和JDK1.7中,Collections.sort的区别,

背景 最近,项目正在集成测试阶段,项目在服务器上运行了一段时间,点击表格的列进行排序的时候,有的列排序正常,有的列在排序的时候,在后台会抛出如下异常,查询到不到数据,而且在另外一...

tsmyk0715 ⋅ 38分钟前 ⋅ 0

C++ 中命名空间的 5 个常见用法

相信小伙伴们对C++已经非常熟悉,但是对命名空间经常使用到的地方还不是很明白,这篇文章就针对命名空间这一块做了一个叙述。 命名空间在1995年被引入到 c++ 标准中,通常是这样定义的: 命名...

柳猫 ⋅ 42分钟前 ⋅ 0

没有更多内容

加载失败,请刷新页面

加载更多

下一页

返回顶部
顶部