文档章节

hadoop学习笔记

清风_
 清风_
发布于 2016/12/10 19:29
字数 2256
阅读 69
收藏 0
* vi /etc/hosts
10.204.211.241 JZYH-COLLECTOR-LTEMR3-OSS
* vi /etc/sysconfig/network
#主机名不要使用下划线
127.0.0.1   localhost localhost4 localhost4.localdomain4
** Single Node Cluster
* etc/hadoop/core-site.xml:
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://JZYH-COLLECTOR-LTEMR3-OSS:8020</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/opt/disk/backup/soft/hadoop-2.5.0/data/tmp</value>
    </property>
    <property>
        <name>fs.trash.interval</name>
        <value>10080</value>
    </property>
</configuration>
* etc/hadoop/hdfs-site.xml:
<configuration>
    <!--分布式去掉该属性-->
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>JZYH-COLLECTOR-LTEMR3-OSS:50090</value>
    </property>
</configuration>
* 配置免登陆
$ ssh localhost
$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
往远程机器上拷贝:$ ssh-copy-id 地址
* Format the filesystem:
  $ bin/hdfs namenode -format
* Start NameNode daemon and DataNode daemon:
  $ sbin/start-dfs.sh
  或者
  $ sbin/hadoop-daemon.sh start namenode
  $ sbin/hadoop-daemon.sh start secondarynamenode
  $ sbin/hadoop-daemon.sh start datanode
* Browse the web interface for the NameNode; by default it is available at:
	NameNode - http://localhost:50070/
* Make the HDFS directories required to execute MapReduce jobs:
  $ bin/hdfs dfs -mkdir /user
  $ bin/hdfs dfs -mkdir /user/<username>
* Copy the input files into the distributed filesystem:
  $ bin/hdfs dfs -put etc/hadoop input
* Run some of the examples provided:
  $ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar grep input output 'dfs[a-z.]+'
* Copy the output files from the distributed filesystem to the local filesystem and examine them:
  $ bin/hdfs dfs -get output output
* View the output files on the distributed filesystem:
  $ bin/hdfs dfs -cat output/*
* When you're done, stop the daemons with:
  $ sbin/stop-dfs.sh

**YARN on Single Node
*etc/hadoop/mapred-site.xml:
<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>JZYH-COLLECTOR-LTEMR3-OSS:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>JZYH-COLLECTOR-LTEMR3-OSS:19888</value>
    </property>
</configuration>
*etc/hadoop/yarn-site.xml:
<configuration>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>JZYH-COLLECTOR-LTEMR3-OSS</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
</configuration>
*etc/hadoop/slaves:
JZYH-COLLECTOR-LTEMR3-OSS
*Start ResourceManager daemon and NodeManager daemon:
$ sbin/start-yarn.sh
或者
$ sbin/yarn-daemon.sh start resourcemanager
$ sbin/yarn-daemon.sh start nodemanager
*Browse the web interface for the ResourceManager; by default it is available at:
ResourceManager - http://localhost:8088/
*When you're done, stop the daemons with:
  $ sbin/stop-yarn.sh

*start Mapreduce history daemon
$ sbin/mr-jobhistory-daemon.sh start historyserver
* Aggregation(应用运行完成以后将日志上传到HDFS文件系统)
	etc/hadoop/yarn-site.xml:
<configuration>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>JZYH-COLLECTOR-LTEMR3-OSS</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
    </property>
    <property>
        <name>yarn.log-aggregation.retain-seconds</name>
        <value>3600</value>
    </property>
</configuration>
重启所有服务
$ sbin/yarn-daemon.sh stop resourcemanager
$ sbin/yarn-daemon.sh stop nodemanager
$ sbin/mr-jobhistory-daemon.sh stop historyserver

$ sbin/yarn-daemon.sh start resourcemanager
$ sbin/yarn-daemon.sh start nodemanager
$ sbin/mr-jobhistory-daemon.sh start historyserver


**读取本地文件:bin/hdfs dfs -Dfs.defaultFS=file:/// -ls /
**状态信息:bin/hdfs dfsadmin -report
**安全模式safemode
启用
bin/hdfs dfsadmin -safemode enter
查询
bin/hdfs dfsadmin -safemode get
退出
bin/hdfs dfsadmin -safemode leave

执行wordcount:
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.1.jar wordcount input output
报错:
16/12/11 13:08:01 INFO client.RMProxy: Connecting to ResourceManager at mac/192.168.1.119:8032
16/12/11 13:08:03 INFO input.FileInputFormat: Total input paths to process : 1
16/12/11 13:08:03 INFO mapreduce.JobSubmitter: number of splits:1
16/12/11 13:08:03 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1481432311518_0003
16/12/11 13:08:03 INFO impl.YarnClientImpl: Submitted application application_1481432311518_0003
16/12/11 13:08:03 INFO mapreduce.Job: The url to track the job: http://mac:8088/proxy/application_1481432311518_0003/
16/12/11 13:08:03 INFO mapreduce.Job: Running job: job_1481432311518_0003
16/12/11 13:08:12 INFO mapreduce.Job: Job job_1481432311518_0003 running in uber mode : false
16/12/11 13:08:12 INFO mapreduce.Job:  map 0% reduce 0%
16/12/11 13:08:12 INFO mapreduce.Job: Job job_1481432311518_0003 failed with state FAILED due to: Application application_1481432311518_0003 failed 2 times due to AM Container for appattempt_1481432311518_0003_000002 exited with  exitCode: 127 due to: Exception from container-launch: ExitCodeException exitCode=127: 
ExitCodeException exitCode=127: 
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
	at org.apache.hadoop.util.Shell.run(Shell.java:455)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
查看日志:
hdfs dfs -ls -R /
drwxrwx---   - john supergroup          0 2016-12-11 12:13 /tmp
drwxrwx---   - john supergroup          0 2016-12-10 18:02 /tmp/hadoop-yarn
drwxrwx---   - john supergroup          0 2016-12-11 12:12 /tmp/hadoop-yarn/staging
drwxrwx---   - john supergroup          0 2016-12-10 18:02 /tmp/hadoop-yarn/staging/history
drwxrwx---   - john supergroup          0 2016-12-10 18:02 /tmp/hadoop-yarn/staging/history/done
drwxrwxrwt   - john supergroup          0 2016-12-10 18:02 /tmp/hadoop-yarn/staging/history/done_intermediate
drwx------   - john supergroup          0 2016-12-11 12:12 /tmp/hadoop-yarn/staging/john
drwx------   - john supergroup          0 2016-12-11 13:08 /tmp/hadoop-yarn/staging/john/.staging
drwx------   - john supergroup          0 2016-12-11 12:12 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0001
-rw-r--r--  10 john supergroup     270368 2016-12-11 12:12 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0001/job.jar
-rw-r--r--  10 john supergroup        112 2016-12-11 12:12 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0001/job.split
-rw-r--r--   1 john supergroup         17 2016-12-11 12:12 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0001/job.splitmetainfo
-rw-r--r--   1 john supergroup      80320 2016-12-11 12:13 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0001/job.xml
drwx------   - john supergroup          0 2016-12-11 12:19 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0002
-rw-r--r--  10 john supergroup     270368 2016-12-11 12:19 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0002/job.jar
-rw-r--r--  10 john supergroup        112 2016-12-11 12:19 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0002/job.split
-rw-r--r--   1 john supergroup         17 2016-12-11 12:19 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0002/job.splitmetainfo
-rw-r--r--   1 john supergroup      80320 2016-12-11 12:19 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0002/job.xml
drwx------   - john supergroup          0 2016-12-11 12:51 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0003
-rw-r--r--  10 john supergroup     270368 2016-12-11 12:51 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0003/job.jar
-rw-r--r--  10 john supergroup        112 2016-12-11 12:51 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0003/job.split
-rw-r--r--   1 john supergroup         17 2016-12-11 12:51 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0003/job.splitmetainfo
-rw-r--r--   1 john supergroup      80320 2016-12-11 12:51 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0003/job.xml
drwx------   - john supergroup          0 2016-12-11 12:52 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0004
-rw-r--r--  10 john supergroup     270368 2016-12-11 12:52 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0004/job.jar
-rw-r--r--  10 john supergroup        112 2016-12-11 12:52 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0004/job.split
-rw-r--r--   1 john supergroup         17 2016-12-11 12:52 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0004/job.splitmetainfo
-rw-r--r--   1 john supergroup      80320 2016-12-11 12:52 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0004/job.xml
drwx------   - john supergroup          0 2016-12-11 12:53 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0005
-rw-r--r--  10 john supergroup     270368 2016-12-11 12:53 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0005/job.jar
-rw-r--r--  10 john supergroup        112 2016-12-11 12:53 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0005/job.split
-rw-r--r--   1 john supergroup         17 2016-12-11 12:53 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0005/job.splitmetainfo
-rw-r--r--   1 john supergroup      80320 2016-12-11 12:53 /tmp/hadoop-yarn/staging/john/.staging/job_1481426363207_0005/job.xml
drwx------   - john supergroup          0 2016-12-11 12:59 /tmp/hadoop-yarn/staging/john/.staging/job_1481432311518_0001
-rw-r--r--  10 john supergroup     270368 2016-12-11 12:59 /tmp/hadoop-yarn/staging/john/.staging/job_1481432311518_0001/job.jar
-rw-r--r--  10 john supergroup        112 2016-12-11 12:59 /tmp/hadoop-yarn/staging/john/.staging/job_1481432311518_0001/job.split
-rw-r--r--   1 john supergroup         17 2016-12-11 12:59 /tmp/hadoop-yarn/staging/john/.staging/job_1481432311518_0001/job.splitmetainfo
-rw-r--r--   1 john supergroup      80315 2016-12-11 12:59 /tmp/hadoop-yarn/staging/john/.staging/job_1481432311518_0001/job.xml
drwx------   - john supergroup          0 2016-12-11 13:07 /tmp/hadoop-yarn/staging/john/.staging/job_1481432311518_0002
-rw-r--r--  10 john supergroup     270368 2016-12-11 13:07 /tmp/hadoop-yarn/staging/john/.staging/job_1481432311518_0002/job.jar
-rw-r--r--  10 john supergroup        112 2016-12-11 13:07 /tmp/hadoop-yarn/staging/john/.staging/job_1481432311518_0002/job.split
-rw-r--r--   1 john supergroup         17 2016-12-11 13:07 /tmp/hadoop-yarn/staging/john/.staging/job_1481432311518_0002/job.splitmetainfo
-rw-r--r--   1 john supergroup      80320 2016-12-11 13:07 /tmp/hadoop-yarn/staging/john/.staging/job_1481432311518_0002/job.xml
drwx------   - john supergroup          0 2016-12-11 13:08 /tmp/hadoop-yarn/staging/john/.staging/job_1481432311518_0003
-rw-r--r--  10 john supergroup     270368 2016-12-11 13:08 /tmp/hadoop-yarn/staging/john/.staging/job_1481432311518_0003/job.jar
-rw-r--r--  10 john supergroup        112 2016-12-11 13:08 /tmp/hadoop-yarn/staging/john/.staging/job_1481432311518_0003/job.split
-rw-r--r--   1 john supergroup         17 2016-12-11 13:08 /tmp/hadoop-yarn/staging/john/.staging/job_1481432311518_0003/job.splitmetainfo
-rw-r--r--   1 john supergroup      80320 2016-12-11 13:08 /tmp/hadoop-yarn/staging/john/.staging/job_1481432311518_0003/job.xml
drwxrwxrwt   - john supergroup          0 2016-12-11 12:13 /tmp/logs
drwxrwx---   - john supergroup          0 2016-12-11 12:13 /tmp/logs/john
drwxrwx---   - john supergroup          0 2016-12-11 13:08 /tmp/logs/john/logs
drwxrwx---   - john supergroup          0 2016-12-11 12:13 /tmp/logs/john/logs/application_1481426363207_0001
-rw-r-----   1 john supergroup        509 2016-12-11 12:13 /tmp/logs/john/logs/application_1481426363207_0001/mac_51897
drwxrwx---   - john supergroup          0 2016-12-11 12:19 /tmp/logs/john/logs/application_1481426363207_0002
-rw-r-----   1 john supergroup        509 2016-12-11 12:19 /tmp/logs/john/logs/application_1481426363207_0002/mac_51897
drwxrwx---   - john supergroup          0 2016-12-11 12:51 /tmp/logs/john/logs/application_1481426363207_0003
-rw-r-----   1 john supergroup        509 2016-12-11 12:51 /tmp/logs/john/logs/application_1481426363207_0003/mac_51897
drwxrwx---   - john supergroup          0 2016-12-11 12:52 /tmp/logs/john/logs/application_1481426363207_0004
-rw-r-----   1 john supergroup        509 2016-12-11 12:52 /tmp/logs/john/logs/application_1481426363207_0004/mac_51897
drwxrwx---   - john supergroup          0 2016-12-11 12:53 /tmp/logs/john/logs/application_1481426363207_0005
-rw-r-----   1 john supergroup        509 2016-12-11 12:53 /tmp/logs/john/logs/application_1481426363207_0005/mac_51897
drwxrwx---   - john supergroup          0 2016-12-11 12:59 /tmp/logs/john/logs/application_1481432311518_0001
-rw-r-----   1 john supergroup        509 2016-12-11 12:59 /tmp/logs/john/logs/application_1481432311518_0001/mac_53087
drwxrwx---   - john supergroup          0 2016-12-11 13:07 /tmp/logs/john/logs/application_1481432311518_0002
-rw-r-----   1 john supergroup        509 2016-12-11 13:07 /tmp/logs/john/logs/application_1481432311518_0002/mac_53087
drwxrwx---   - john supergroup          0 2016-12-11 13:08 /tmp/logs/john/logs/application_1481432311518_0003
-rw-r-----   1 john supergroup        509 2016-12-11 13:08 /tmp/logs/john/logs/application_1481432311518_0003/mac_53087
drwx------   - john supergroup          0 2016-12-11 11:50 /user
drwx------   - john supergroup          0 2016-12-11 12:12 /user/john
drwx------   - john supergroup          0 2016-12-11 11:50 /user/john/.Trash
drwx------   - john supergroup          0 2016-12-11 12:12 /user/john/.Trash/Current
drwx------   - john supergroup          0 2016-12-11 11:50 /user/john/.Trash/Current/test
-rw-r--r--   1 john supergroup      15458 2016-12-10 18:43 /user/john/.Trash/Current/test/LICENSE.txt
drwxr-xr-x   - john supergroup          0 2016-12-11 12:08 /user/john/.Trash/Current/test1481429418166
-rw-r--r--   1 john supergroup        101 2016-12-10 19:15 /user/john/.Trash/Current/test1481429418166/NOTICE.txt
drwx------   - john supergroup          0 2016-12-11 12:12 /user/john/.Trash/Current/user
drwx------   - john supergroup          0 2016-12-11 12:12 /user/john/.Trash/Current/user/john
drwxr-xr-x   - john supergroup          0 2016-12-11 12:08 /user/john/.Trash/Current/user/john/output
drwxr-xr-x   - john supergroup          0 2016-12-11 12:08 /user/john/input
-rw-r--r--   1 john supergroup         11 2016-12-11 11:53 /user/john/input/wc.input
hdfs dfs -cat /tmp/logs/john/logs/application_1481432311518_0003/mac_53087
??h??׶9?A@???P	VERSIONAPPLICATION_ACL
MODIFY_APPVIEW_APP APPLICATION_OWNERjohn(&container_1481432311518_0003_01_000001Gstderr48/bin/bash: /bin/java: No such file or directory
stdout0(&container_1481432311518_0003_02_000001Gstderr48/bin/bash: /bin/java: No such file or directory
stdout0
	VERSION*(&container_1481432311518_0003_02_000001none?;?;data:BCFile.indexnone͇

                                                                                     data:TFile.indexnone?Q66data:TFile.metanone?K???h??׶9?A@???PJohndeiMac:Desktop john$ 
找到问题原因,创建软链接解决问题:
sudo ln -s /usr/bin/java /bin/java

运行自己写的jar包
yarn jar jars/mr-wordcount.jar /user/john/input /user/john/wordcount

 

© 著作权归作者所有

共有 人打赏支持
清风_

清风_

粉丝 31
博文 38
码字总数 23278
作品 1
沈阳
程序员

暂无文章

OSChina 周三乱弹 —— 公司女同事约我

Osc乱弹歌单(2018)请戳(这里) 【今日歌曲】 @莱布妮子:分享水木年华的单曲《蝴蝶花(2002年大提琴版)》 《蝴蝶花(2002年大提琴版)》- 水木年华 手机党少年们想听歌,请使劲儿戳(这里) ...

小小编辑
7分钟前
12
4
Linux环境搭建 | VMware下共享文件夹的实现

在进行程序开发的过程中,我们经常要在主机与虚拟机之间传递文件,比如说,源代码位于虚拟机,而在主机下阅读或修改源代码,这里就需要使用到 「共享文件」 这个机制了。本文介绍了两种共享文...

良许Linux
今天
5
0
JUC锁框架——AQS源码分析

JUC锁介绍 Java的并发框架JUC(java.util.concurrent)中锁是最重要的一个工具。因为锁,才能实现正确的并发访问。而AbstractQueuedSynchronizer(AQS)是一个用来构建锁和同步器的框架,使用A...

长头发-dawn
今天
3
0
docker中安装了RabbitMQ后无法访问其Web管理页面

在官网找了"$ docker run -d --hostname my-rabbit --name some-rabbit -p 8080:15672 rabbitmq:3-management"这条安装命令,在docker上安装了RabbitMQ,,结果输入http://localhost:8080并不......

钟然千落
今天
4
1
spring-cloud | 分布式session共享

写在前面的话 各位小伙伴,你们有福了,这一节不仅教大家怎么实现分布式session的问题,还用kotlin开发,喜欢kotlin的小伙伴是不是很开心! 以前在写Android的时候,就对客户端请求有一定的认...

冯文议
今天
3
0

没有更多内容

加载失败,请刷新页面

加载更多

下一页

返回顶部
顶部