文档章节

使用Windows下的intellij IDEA 运行MapReduce程序远程调用Hadoop的hdfs(非Maven方法)

林海_why
 林海_why
发布于 2017/09/07 21:09
字数 4296
阅读 1.2W
收藏 0

#程序员薪资揭榜#你做程序员几年了?月薪多少?发量还在么?>>>

我使用的环境分别是:

本地:win10_64位+IDEA17.2.3

远程:虚拟机 UbuntuKylin16.10_64位+hadoop2.7.3

  • 前提工作:

hadoop环境已经搭好,HDFS服务已经启动。

本地的intellij已经安装好,Jdk等配置好。

  • 第一步:intellij IDEA的项目配置

1.建Java项目,空白项目即可。

2.添加hadoop依赖包。

用启动hdfs的hadoop文件中的hadoop-x.x.x/share/hadoop/下的所有文件复制出来,这所有都是要添加的依赖包,特别要再把hadoop-x.x.x/share/hadoop/common/lib添加到依赖包。

3.添加Windows下的必要文件

Windows下缺少winutils.exe和hadoop.dll文件,请在GitHub上搜索“winutils”,下载对应版本的文件。

比如我用的是hadoop2.7.3版本,所以:

4.添加HADOOP_HOME并添加到path

将第3步下载的所有的文件放到HADOOP_HOME/bin下即可,并在path下追加%HADOOP_HOME%\bin

然后,重启IDEA才能生效!!!

5.运行配置

这里以WordCount为例,需要写两个参数,输入文件和输入文件

注意:输出目录必须是不存在的目录,否则hadoop报错!!!

像这样:

这里的输入输出写远程的hdfs,按格式。

  • 第二步:代码编写 

很简单,这里拿hadoop源文件里给的wordcount例子来演示

package com.hadoop.mapreduce;

/**
 * Created by Aqc on 2017/9/6.
 */
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import java.io.IOException;
import java.util.StringTokenizer;

public class WordCount {

    public static class TokenizerMapper
            extends Mapper<Object, Text, Text, IntWritable> {

        private final static IntWritable one = new IntWritable(1);
        private Text word = new Text();

        @Override
        public void map(Object key, Text value, Context context
        ) throws IOException, InterruptedException {
            StringTokenizer itr = new StringTokenizer(value.toString());
            while (itr.hasMoreTokens()) {
                word.set(itr.nextToken());
                context.write(word, one);
            }
        }
    }

    public static class IntSumReducer
            extends Reducer<Text, IntWritable, Text, IntWritable> {
        private IntWritable result = new IntWritable();

        @Override
        public void reduce(Text key, Iterable<IntWritable> values,
                           Context context
        ) throws IOException, InterruptedException {
            int sum = 0;
            for (IntWritable val : values) {
                sum += val.get();
            }
            result.set(sum);
            context.write(key, result);
        }
    }

    private static void deleteDir(Configuration conf, String dirPath) throws IOException {
        FileSystem fs = FileSystem.get(conf);
        Path targetPath = new Path(dirPath);
        if (fs.exists(targetPath)) {
            boolean delResult = fs.delete(targetPath, true);
            if (delResult) {
                System.out.println(targetPath + " has been deleted sucessfullly.");
            } else {
                System.out.println(targetPath + " deletion failed.");
            }
        }

    }

    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
      /*  conf.set("fs.defaultFS","hdfs://192.168.56.101:8020");
        String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
        if (otherArgs.length < 2) {
            System.err.println("Usage: wordcount <in> [<in>...] <out>");
            System.exit(2);
        }
        //先删除output目录
        deleteDir(conf, otherArgs[otherArgs.length - 1]);*/
        Job job = Job.getInstance(conf, "word count");
        job.setJarByClass(WordCount.class);
        job.setMapperClass(TokenizerMapper.class);
        job.setCombinerClass(IntSumReducer.class);
        job.setReducerClass(IntSumReducer.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        FileInputFormat.addInputPath(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));
        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }

}

 

  • 第三步:连接远程hdfs,运行程序

 首要:

1.确保你的hdfs监听的方式,在hadoop-x.x.x/etc/hadoop/core-site.xml中,fs.defaultFS的值hdfs://ip:port中的ip是不是127.0.0.1即localhost,如果是,则改为本地ip(或者添加/etc/hosts文件),port没有特别要求。。

2.为了保证没有权限问题在hdfs-site.xml文件中添加:

  <property>
     <name>dfs.permissions</name>
     <value>false</value>
  </property>

其实这样是不安全的,在企业中是不允许的,这里如果不考虑太多的话,可以这么改。

然后运行IDEA的代码就可以了。

IDEA的输出结果展示:

"D:\Program Files\Java\jdk1.7.0_80\bin\java" "-javaagent:D:\Program Files\IntelliJ IDEA 2017.2.3\lib\idea_rt.jar=53165:D:\Program Files\IntelliJ IDEA 2017.2.3\bin" -Dfile.encoding=UTF-8 -classpath "D:\Program Files\Java\jdk1.7.0_80\jre\lib\charsets.jar;D:\Program Files\Java\jdk1.7.0_80\jre\lib\deploy.jar;D:\Program Files\Java\jdk1.7.0_80\jre\lib\ext\access-bridge-64.jar;D:\Program Files\Java\jdk1.7.0_80\jre\lib\ext\dnsns.jar;D:\Program Files\Java\jdk1.7.0_80\jre\lib\ext\jaccess.jar;D:\Program Files\Java\jdk1.7.0_80\jre\lib\ext\localedata.jar;D:\Program Files\Java\jdk1.7.0_80\jre\lib\ext\sunec.jar;D:\Program Files\Java\jdk1.7.0_80\jre\lib\ext\sunjce_provider.jar;D:\Program Files\Java\jdk1.7.0_80\jre\lib\ext\sunmscapi.jar;D:\Program Files\Java\jdk1.7.0_80\jre\lib\ext\zipfs.jar;D:\Program Files\Java\jdk1.7.0_80\jre\lib\javaws.jar;D:\Program Files\Java\jdk1.7.0_80\jre\lib\jce.jar;D:\Program Files\Java\jdk1.7.0_80\jre\lib\jfr.jar;D:\Program Files\Java\jdk1.7.0_80\jre\lib\jfxrt.jar;D:\Program Files\Java\jdk1.7.0_80\jre\lib\jsse.jar;D:\Program Files\Java\jdk1.7.0_80\jre\lib\management-agent.jar;D:\Program Files\Java\jdk1.7.0_80\jre\lib\plugin.jar;D:\Program Files\Java\jdk1.7.0_80\jre\lib\resources.jar;D:\Program Files\Java\jdk1.7.0_80\jre\lib\rt.jar;D:\Users\Aqc\Coding_why\MyHadoop_test\out\production\MyHadoop_test;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\hadoop-nfs-2.7.3.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\hadoop-common-2.7.3.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\hadoop-common-2.7.3-tests.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\hdfs\hadoop-hdfs-2.7.3.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\hdfs\hadoop-hdfs-nfs-2.7.3.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\hdfs\hadoop-hdfs-2.7.3-tests.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\httpfs;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\mapreduce\hadoop-mapreduce-examples-2.7.3.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\mapreduce\hadoop-mapreduce-client-hs-2.7.3.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\mapreduce\hadoop-mapreduce-client-app-2.7.3.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\mapreduce\hadoop-mapreduce-client-core-2.7.3.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\mapreduce\hadoop-mapreduce-client-common-2.7.3.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\mapreduce\hadoop-mapreduce-client-shuffle-2.7.3.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\mapreduce\hadoop-mapreduce-client-jobclient-2.7.3.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\mapreduce\hadoop-mapreduce-client-hs-plugins-2.7.3.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\mapreduce\hadoop-mapreduce-client-jobclient-2.7.3-tests.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\yarn\hadoop-yarn-api-2.7.3.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\yarn\hadoop-yarn-client-2.7.3.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\yarn\hadoop-yarn-common-2.7.3.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\yarn\hadoop-yarn-registry-2.7.3.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\yarn\hadoop-yarn-server-tests-2.7.3.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\yarn\hadoop-yarn-server-common-2.7.3.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\yarn\hadoop-yarn-server-web-proxy-2.7.3.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\yarn\hadoop-yarn-server-nodemanager-2.7.3.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\yarn\hadoop-yarn-server-resourcemanager-2.7.3.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\yarn\hadoop-yarn-server-sharedcachemanager-2.7.3.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\yarn\hadoop-yarn-applications-distributedshell-2.7.3.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\yarn\hadoop-yarn-server-applicationhistoryservice-2.7.3.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\yarn\hadoop-yarn-applications-unmanaged-am-launcher-2.7.3.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\xz-1.0.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\asm-3.2.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\avro-1.7.4.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\gson-2.2.4.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\junit-4.11.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\jsch-0.1.42.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\jsp-api-2.1.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\xmlenc-0.52.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\guava-11.0.2.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\jets3t-0.9.0.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\jettison-1.1.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\jetty-6.1.26.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\jsr305-3.0.0.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\log4j-1.2.17.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\paranamer-2.3.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\activation-1.1.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\commons-io-2.4.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\httpcore-4.2.5.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\jaxb-api-2.2.2.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\stax-api-1.0-2.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\commons-cli-1.2.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\commons-net-3.1.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\jersey-core-1.9.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\jersey-json-1.9.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\servlet-api-2.5.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\zookeeper-3.4.6.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\commons-lang-2.6.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\httpclient-4.2.5.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\slf4j-api-1.7.10.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\commons-codec-1.4.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\hadoop-auth-2.7.3.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\hamcrest-core-1.3.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\jackson-xc-1.9.13.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\jaxb-impl-2.2.3-1.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\jersey-server-1.9.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\jetty-util-6.1.26.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\mockito-all-1.8.5.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\netty-3.6.2.Final.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\api-util-1.0.0-M20.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\commons-math3-3.1.1.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\java-xmlbuilder-0.4.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\protobuf-java-2.5.0.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\snappy-java-1.0.4.1.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\commons-digester-1.8.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\curator-client-2.7.1.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\jackson-jaxrs-1.9.13.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\slf4j-log4j12-1.7.10.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\commons-logging-1.1.3.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\curator-recipes-2.7.1.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\api-asn1-api-1.0.0-M20.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\commons-compress-1.4.1.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\commons-httpclient-3.1.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\apacheds-i18n-2.0.0-M15.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\commons-beanutils-1.7.0.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\curator-framework-2.7.1.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\jackson-core-asl-1.9.13.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\hadoop-annotations-2.7.3.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\commons-collections-3.2.2.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\commons-configuration-1.6.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\jackson-mapper-asl-1.9.13.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\commons-beanutils-core-1.8.0.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\htrace-core-3.1.0-incubating.jar;D:\Users\Coding_why\hadoop-2.7.3\share\hadoop\common\lib\apacheds-kerberos-codec-2.0.0-M15.jar" com.hadoop.mapreduce.WordCount hdfs://192.168.56.101:8020/user/hadoop/input1 hdfs://192.168.56.101:8020/user/hadoop/output2
2017-09-07 21:08:29,673 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1173)) - session.id is deprecated. Instead, use dfs.metrics.session-id
2017-09-07 21:08:29,676 INFO  [main] jvm.JvmMetrics (JvmMetrics.java:init(76)) - Initializing JVM Metrics with processName=JobTracker, sessionId=
2017-09-07 21:08:29,902 WARN  [main] mapreduce.JobResourceUploader (JobResourceUploader.java:uploadFiles(64)) - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2017-09-07 21:08:29,942 WARN  [main] mapreduce.JobResourceUploader (JobResourceUploader.java:uploadFiles(171)) - No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
2017-09-07 21:08:30,003 INFO  [main] input.FileInputFormat (FileInputFormat.java:listStatus(283)) - Total input paths to process : 2
2017-09-07 21:08:30,049 INFO  [main] mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(198)) - number of splits:2
2017-09-07 21:08:30,181 INFO  [main] mapreduce.JobSubmitter (JobSubmitter.java:printTokens(287)) - Submitting tokens for job: job_local2028239795_0001
2017-09-07 21:08:30,353 INFO  [main] mapreduce.Job (Job.java:submit(1294)) - The url to track the job: http://localhost:8080/
2017-09-07 21:08:30,354 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1339)) - Running job: job_local2028239795_0001
2017-09-07 21:08:30,355 INFO  [Thread-5] mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2017-09-07 21:08:30,360 INFO  [Thread-5] output.FileOutputCommitter (FileOutputCommitter.java:<init>(108)) - File Output Committer Algorithm version is 1
2017-09-07 21:08:30,363 INFO  [Thread-5] mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2017-09-07 21:08:30,431 INFO  [Thread-5] mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2017-09-07 21:08:30,432 INFO  [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local2028239795_0001_m_000000_0
2017-09-07 21:08:30,452 INFO  [LocalJobRunner Map Task Executor #0] output.FileOutputCommitter (FileOutputCommitter.java:<init>(108)) - File Output Committer Algorithm version is 1
2017-09-07 21:08:30,458 INFO  [LocalJobRunner Map Task Executor #0] util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(192)) - ProcfsBasedProcessTree currently is supported only on Linux.
2017-09-07 21:08:30,766 INFO  [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(612)) -  Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@285eaab5
2017-09-07 21:08:30,770 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(756)) - Processing split: hdfs://192.168.56.101:8020/user/hadoop/input1/file0:0+20
2017-09-07 21:08:30,817 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1205)) - (EQUATOR) 0 kvi 26214396(104857584)
2017-09-07 21:08:30,818 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(998)) - mapreduce.task.io.sort.mb: 100
2017-09-07 21:08:30,818 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(999)) - soft limit at 83886080
2017-09-07 21:08:30,818 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(1000)) - bufstart = 0; bufvoid = 104857600
2017-09-07 21:08:30,818 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(1001)) - kvstart = 26214396; length = 6553600
2017-09-07 21:08:30,822 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:createSortingCollector(403)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2017-09-07 21:08:31,095 INFO  [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2017-09-07 21:08:31,101 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1460)) - Starting flush of map output
2017-09-07 21:08:31,102 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1482)) - Spilling map output
2017-09-07 21:08:31,102 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1483)) - bufstart = 0; bufend = 36; bufvoid = 104857600
2017-09-07 21:08:31,102 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1485)) - kvstart = 26214396(104857584); kvend = 26214384(104857536); length = 13/6553600
2017-09-07 21:08:31,122 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1667)) - Finished spill 0
2017-09-07 21:08:31,135 INFO  [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(1038)) - Task:attempt_local2028239795_0001_m_000000_0 is done. And is in the process of committing
2017-09-07 21:08:31,145 INFO  [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - map
2017-09-07 21:08:31,145 INFO  [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1158)) - Task 'attempt_local2028239795_0001_m_000000_0' done.
2017-09-07 21:08:31,145 INFO  [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local2028239795_0001_m_000000_0
2017-09-07 21:08:31,145 INFO  [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local2028239795_0001_m_000001_0
2017-09-07 21:08:31,146 INFO  [LocalJobRunner Map Task Executor #0] output.FileOutputCommitter (FileOutputCommitter.java:<init>(108)) - File Output Committer Algorithm version is 1
2017-09-07 21:08:31,147 INFO  [LocalJobRunner Map Task Executor #0] util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(192)) - ProcfsBasedProcessTree currently is supported only on Linux.
2017-09-07 21:08:31,210 INFO  [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(612)) -  Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@50a4a6f0
2017-09-07 21:08:31,212 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(756)) - Processing split: hdfs://192.168.56.101:8020/user/hadoop/input1/file1:0+17
2017-09-07 21:08:31,245 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1205)) - (EQUATOR) 0 kvi 26214396(104857584)
2017-09-07 21:08:31,246 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(998)) - mapreduce.task.io.sort.mb: 100
2017-09-07 21:08:31,246 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(999)) - soft limit at 83886080
2017-09-07 21:08:31,246 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(1000)) - bufstart = 0; bufvoid = 104857600
2017-09-07 21:08:31,246 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(1001)) - kvstart = 26214396; length = 6553600
2017-09-07 21:08:31,247 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:createSortingCollector(403)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2017-09-07 21:08:31,252 INFO  [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2017-09-07 21:08:31,252 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1460)) - Starting flush of map output
2017-09-07 21:08:31,252 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1482)) - Spilling map output
2017-09-07 21:08:31,252 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1483)) - bufstart = 0; bufend = 29; bufvoid = 104857600
2017-09-07 21:08:31,252 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1485)) - kvstart = 26214396(104857584); kvend = 26214388(104857552); length = 9/6553600
2017-09-07 21:08:31,261 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1667)) - Finished spill 0
2017-09-07 21:08:31,270 INFO  [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(1038)) - Task:attempt_local2028239795_0001_m_000001_0 is done. And is in the process of committing
2017-09-07 21:08:31,273 INFO  [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - map
2017-09-07 21:08:31,273 INFO  [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1158)) - Task 'attempt_local2028239795_0001_m_000001_0' done.
2017-09-07 21:08:31,273 INFO  [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local2028239795_0001_m_000001_0
2017-09-07 21:08:31,273 INFO  [Thread-5] mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2017-09-07 21:08:31,276 INFO  [Thread-5] mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2017-09-07 21:08:31,276 INFO  [pool-6-thread-1] mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local2028239795_0001_r_000000_0
2017-09-07 21:08:31,285 INFO  [pool-6-thread-1] output.FileOutputCommitter (FileOutputCommitter.java:<init>(108)) - File Output Committer Algorithm version is 1
2017-09-07 21:08:31,286 INFO  [pool-6-thread-1] util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(192)) - ProcfsBasedProcessTree currently is supported only on Linux.
2017-09-07 21:08:31,360 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1360)) - Job job_local2028239795_0001 running in uber mode : false
2017-09-07 21:08:31,361 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1367)) -  map 100% reduce 0%
2017-09-07 21:08:31,377 INFO  [pool-6-thread-1] mapred.Task (Task.java:initialize(612)) -  Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@1af25560
2017-09-07 21:08:31,381 INFO  [pool-6-thread-1] mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@26f5d2fd
2017-09-07 21:08:31,395 INFO  [pool-6-thread-1] reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(197)) - MergerManager: memoryLimit=1303589632, maxSingleShuffleLimit=325897408, mergeThreshold=860369216, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2017-09-07 21:08:31,397 INFO  [EventFetcher for fetching Map Completion Events] reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local2028239795_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2017-09-07 21:08:31,428 INFO  [localfetcher#1] reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(144)) - localfetcher#1 about to shuffle output of map attempt_local2028239795_0001_m_000000_0 decomp: 46 len: 50 to MEMORY
2017-09-07 21:08:31,432 INFO  [localfetcher#1] reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 46 bytes from map-output for attempt_local2028239795_0001_m_000000_0
2017-09-07 21:08:31,434 INFO  [localfetcher#1] reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(315)) - closeInMemoryFile -> map-output of size: 46, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->46
2017-09-07 21:08:31,439 INFO  [localfetcher#1] reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(144)) - localfetcher#1 about to shuffle output of map attempt_local2028239795_0001_m_000001_0 decomp: 37 len: 41 to MEMORY
2017-09-07 21:08:31,440 INFO  [localfetcher#1] reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 37 bytes from map-output for attempt_local2028239795_0001_m_000001_0
2017-09-07 21:08:31,440 INFO  [localfetcher#1] reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(315)) - closeInMemoryFile -> map-output of size: 37, inMemoryMapOutputs.size() -> 2, commitMemory -> 46, usedMemory ->83
2017-09-07 21:08:31,441 INFO  [EventFetcher for fetching Map Completion Events] reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2017-09-07 21:08:31,441 INFO  [pool-6-thread-1] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2017-09-07 21:08:31,442 INFO  [pool-6-thread-1] reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(687)) - finalMerge called with 2 in-memory map-outputs and 0 on-disk map-outputs
2017-09-07 21:08:31,454 INFO  [pool-6-thread-1] mapred.Merger (Merger.java:merge(606)) - Merging 2 sorted segments
2017-09-07 21:08:31,454 INFO  [pool-6-thread-1] mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 2 segments left of total size: 69 bytes
2017-09-07 21:08:31,459 INFO  [pool-6-thread-1] reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(754)) - Merged 2 segments, 83 bytes to disk to satisfy reduce memory limit
2017-09-07 21:08:31,460 INFO  [pool-6-thread-1] reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(784)) - Merging 1 files, 85 bytes from disk
2017-09-07 21:08:31,461 INFO  [pool-6-thread-1] reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(799)) - Merging 0 segments, 0 bytes from memory into reduce
2017-09-07 21:08:31,461 INFO  [pool-6-thread-1] mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2017-09-07 21:08:31,462 INFO  [pool-6-thread-1] mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 74 bytes
2017-09-07 21:08:31,462 INFO  [pool-6-thread-1] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2017-09-07 21:08:31,494 INFO  [pool-6-thread-1] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1173)) - mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
2017-09-07 21:08:31,600 INFO  [pool-6-thread-1] mapred.Task (Task.java:done(1038)) - Task:attempt_local2028239795_0001_r_000000_0 is done. And is in the process of committing
2017-09-07 21:08:31,605 INFO  [pool-6-thread-1] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2017-09-07 21:08:31,605 INFO  [pool-6-thread-1] mapred.Task (Task.java:commit(1199)) - Task attempt_local2028239795_0001_r_000000_0 is allowed to commit now
2017-09-07 21:08:31,631 INFO  [pool-6-thread-1] output.FileOutputCommitter (FileOutputCommitter.java:commitTask(535)) - Saved output of task 'attempt_local2028239795_0001_r_000000_0' to hdfs://192.168.56.101:8020/user/hadoop/output2/_temporary/0/task_local2028239795_0001_r_000000
2017-09-07 21:08:31,632 INFO  [pool-6-thread-1] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2017-09-07 21:08:31,632 INFO  [pool-6-thread-1] mapred.Task (Task.java:sendDone(1158)) - Task 'attempt_local2028239795_0001_r_000000_0' done.
2017-09-07 21:08:31,632 INFO  [pool-6-thread-1] mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local2028239795_0001_r_000000_0
2017-09-07 21:08:31,632 INFO  [Thread-5] mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2017-09-07 21:08:32,363 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1367)) -  map 100% reduce 100%
2017-09-07 21:08:32,363 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1378)) - Job job_local2028239795_0001 completed successfully
2017-09-07 21:08:32,376 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1385)) - Counters: 35
	File System Counters
		FILE: Number of bytes read=1675
		FILE: Number of bytes written=843672
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=94
		HDFS: Number of bytes written=45
		HDFS: Number of read operations=22
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=5
	Map-Reduce Framework
		Map input records=2
		Map output records=7
		Map output bytes=65
		Map output materialized bytes=91
		Input split bytes=232
		Combine input records=7
		Combine output records=7
		Reduce input groups=6
		Reduce shuffle bytes=91
		Reduce input records=7
		Reduce output records=6
		Spilled Records=14
		Shuffled Maps =2
		Failed Shuffles=0
		Merged Map outputs=2
		GC time elapsed (ms)=0
		Total committed heap usage (bytes)=1005060096
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=37
	File Output Format Counters 
		Bytes Written=45

Process finished with exit code 0

以上。

© 著作权归作者所有

林海_why
粉丝 0
博文 8
码字总数 6442
作品 0
私信 提问
加载中

评论(0)

【转载】MapReduce编程 Intellij Idea配置MapReduce编程环境

目录(?)[-] 一软件环境 二创建maven工程 三添加maven依赖 四配置log4j 五启动Hadoop 六运行WordCount从本地读取文件 七运行WordCount从HDFS读取文件 八代码下载 介绍如何在Intellij Idea中通...

Jonson Li
2017/05/09
0
0
MapReduce 踩坑 - hadoop No FileSystem for scheme: file/hdfs

一、场景 hadoop-3.0.2 + hbase-2.0.0 一个mapreduce任务,在IDEA下本地提交到hadoop集群可以正常运行。 现在需要将IDEA本地项目通过maven打成jar包,从而能够在windows/Linux命令行下,通过...

osc_96xykppu
2018/06/22
2
0
Mac下报错'WARN security.UserGroupInformation: PriviledgedActionException as:用户名...No such file o...

Mac下使用IDEA远程连接Hadoop调试MapReduce程序,参考网上博客,总会出现如题报错,下面是我在mac下的一种解决办法,可以参考。 前期准备 如果想远程调试,需要做一些准备工作,简单罗列下。...

osc_fjmtyahf
2019/10/24
4
0
2.3 基于IDEA开发第一个MapReduce大数据程序WordCount

2.3 基于IDEA开发第一个MapReduce大数据程序WordCount 开源地址 https://github.com/wangxiaoleiAI/big-data 卜算子·大数据 目录 开源“卜算子·大数据”系列文章、源码,面向大数据(分布式...

王小雷-多面手
03/31
0
0
Hadoop-MR实现日志清洗(四)

Hadoop-MR实现日志清洗(四) 6.单机部署Hadoop(伪分布式) 在我们自己购买的阿里云Linux主机上部署单机hadoop,测试我们的mapreduce程序。也可以在windows上安装虚拟机部署。 6.1配置Java环...

osc_bouq2lox
2018/09/04
2
0

没有更多内容

加载失败,请刷新页面

加载更多

【视频】工业互联网在线交流现状、发展、怎么建和规划

PPT介绍目录: 工业互联网的现状 平台介绍 技术路线 人力资源 现实问题 成本中心 工业互联网的发展 两大推动引擎 用户需求的改变 片面与极端 由点到面,点面结合 突破点 现实路径 落脚点 工业...

osc_3ubrwgtm
47分钟前
51
0
单臂-Etrunk-三层交换-路由-DHCP中继-NAT-HTTP服务 实验题

单臂-Etrunk-三层交换-路由-DHCP中继-NAT-HTTP服务 实验题 实验环境 需求描述 pc1、pc2、pc3、pc4可以ping通 虚拟机可以打开网页 了解dhcp中继 了解网络地址转换技术NAT 了解access口跟trunk...

osc_h7zc4umy
48分钟前
24
0
医院防勒索病毒灾备解决方案

医院成勒索病毒重灾区 上述某妇幼保健院并非是个例。近年来,针对医院等医疗系统的网络安全风险一直处于活跃状态且呈现持续上升的态势。医院的业务系统及医学记录、病患资料、预约信息等大量...

osc_gh0ost1g
49分钟前
18
0
2招带你快速获取响应头与发送头

今天带大家用两招快速获取响应头以及发送头。 1、获取请求头 想要获取请求头,就需要会用抓包工具,目前比较普遍的就是fiddler。大家可以在百度搜索下载。首先我们打开fiddler 你会发现左侧这...

osc_3g4j2ghj
51分钟前
20
0
Python Django 数据库基础

对数据库的操作可以直接使用SQL语句,也可以使用图形界面工具。这些一般是DBA(数据库管理人员)做的。开发人员是使用代码链接数据库,对数据库进行操作。 下面进行Python Django框架ORM创建...

osc_itgved4p
53分钟前
26
0

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部