sbt 编译spark 的wordcount 程序
博客专区 > MtrS 的博客 > 博客详情
sbt 编译spark 的wordcount 程序
MtrS 发表于3年前
sbt 编译spark 的wordcount 程序
  • 发表于 3年前
  • 阅读 1303
  • 收藏 0
  • 点赞 0
  • 评论 0

标题:腾讯云 新注册用户域名抢购1元起>>>   

  1. 直接执行 sbt 会在当前目录下面创建 target 目录
  2. sbt 的目录格局一般为 lib/ (该目录下存储与编译相关的 jar 文件)
    project/ src/main/scala/ src/main/test/scala
  3. 复制 jar 文件 spark-assembly *hadoop2.5.1.jar 到 lib 目录下
<!-- lang: shell -->
[root@localhost word]# find ../spark  -name "spark*jar" |grep assem

../spark/assembly/target/scala-2.10/spark-assembly-1.1.2-SNAPSHOT-hadoop2.5.1.jar ../spark/dist/lib/spark-assembly-1.1.2-SNAPSHOT-hadoop2.5.1.jar [root@localhost word]# cp ../spark/dist/lib/spark-assembly-1.1.2-SNAPSHOT-hadoop2.5.1.jar lib/ [root@localhost word]# ls lib spark-assembly-1.1.2-SNAPSHOT-hadoop2.5.1.jar 4. 编辑 wordcount.scala

<!-- lang: scala -->
import org.apache.spark.{SparkContext, SparkConf}
import org.apache.spark.SparkContext._
object wordCount{
    def main(args: Array[String]){
        if (args.length == 0) {
        System.err.println("Usage bin/spark-submit [options] --class wordCount wordCount.jar <file1:URI>")
        System.err.println("Usage bin/spark-submit [options] --class wordCount wordCount.jar hdfs://172.16.1.141:9000/test.txt")
        System.exit(1);
    }
        val conf = new SparkConf().setAppName("WordCount")
        val sc = new SparkContext(conf)
        val doc = sc.textFile(args(0))
        doc.cache()
        val words = doc.flatMap(_.split(""))
        val pairs = words.map( x=> (x,1))
        val res = pairs.reduceByKey(_+_)
        res.collect().foreach(println)
        sc.stop()
    }
}
  1. 编辑 build.sbt

    <!-- lang: css -->

    [root@localhost word]# cat build.sbt name := "wordCount" [blank line] version := "1.0" [blank line] scalaVersion := "2.11.4" 6 . 编译打包 成 jar 文件

    <!-- lang: shell -->

    [root@localhost word]# sbt package  -Dsbt.ivy.home=/root/.ivy2
    

    [info] Set current project to wordCount (in build file:/opt/htt/temp_20140611/java/word/) [info] Updating {file:/opt/htt/temp_20140611/java/word/}word... [info] Resolving jline#jline;2.12 ... [info] Done updating. [info] Compiling 2 Scala sources to /opt/htt/temp_20140611/java/word/target/scala-2.11/classes... [warn] Multiple main classes detected. Run 'show discoveredMainClasses' to see the list [info] Packaging /opt/htt/temp_20140611/java/word/target/scala-2.11/wordcount_2.11-1.0.jar ... [info] Done packaging. [success] Total time: 11 s, completed Jan 5, 2015 8:37:38 AM [root@localhost word]# 7. 编译 class 文件到当前目录

    <!-- lang: shell -->

    scalac src/main/scala/wordCount.scala -cp lib/spark-assembly-1.1.2-SNAPSHOT-hadoop2.5.1.jar

  2. 调用spark 执行

<!-- lang: scala -->
 ../spark/bin/spark-submit   --class wordCount   target/scala-2.11/wordcount_2.11-1.0.jar hdfs://172.16.1.141:9000/opt/old/htt/test/test.txt

参考文章: http://www.aboutyun.com/thread-8587-1-1.html

hadoop 的wordCount 在 文档里面有,就不多说啦;

<!-- lang: shell -->
http://10.255.32.250:60001/hadoop-2.5.1/share/doc/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html#Job_Configuration
共有 人打赏支持
粉丝 29
博文 468
码字总数 351660
×
MtrS
如果觉得我的文章对您有用,请随意打赏。您的支持将鼓励我继续创作!
* 金额(元)
¥1 ¥5 ¥10 ¥20 其他金额
打赏人
留言
* 支付类型
微信扫码支付
打赏金额:
已支付成功
打赏金额: