100.Spark本地版-安装使用

原创
2016/04/22 15:20
阅读数 84

参考地址:http://www.linuxidc.com/Linux/2014-09/106714.htm

1.安装过程

安装:Java/Scala
下载:Spark-1.4.1-bin-hadoop2.6

直接进入spark的根目录

$ ./bin/spark-shell

> val lines = sc.textFile("/Users/jiangzl/Desktop/README.md")
> lines.count()
> lines.first()
> val scLines = lines.filter(line => line.contains("sc"))
> scLines.collect()

2.SparkDemo测试

目录结构 $ tree .

├── simple.sbt
├── src
│   └── main
│       └── scala
│           └── example.scala

$ cat simple.sbt

name := "Simple Project"
version := "1.0"
scalaVersion := "2.11.7"
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.4.1"

$ cat src/main/scala/example.scala

import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._

object example {
  def main(args: Array[String]) {
  val conf = new SparkConf().setMaster("local").setAppName("My App")
  val sc = new SparkContext(conf)
  sc.stop()
  //System.exit(0)
  //sys.exit()
  println("this system exit ok!!!")
  }
}

$ sbt package

[info] Set current project to Simple Project (in build file:/Users/jiangzl/Desktop/test/)
[info] Updating {file:/Users/jiangzl/Desktop/test/}test...
[info] Resolving jline#jline;2.12.1 ...
[info] downloading https://jcenter.bintray.com/asm/asm/3.1/asm-3.1.jar ...
[info]  [SUCCESSFUL ] asm#asm;3.1!asm.jar (1968ms)
[info] downloading https://jcenter.bintray.com/org/codehaus/jackson/jackson-jaxrs/1.8.3/jackson-jaxrs-1.8.3.jar ...
[info]  [SUCCESSFUL ] org.codehaus.jackson#jackson-jaxrs;1.8.3!jackson-jaxrs.jar (1563ms)
[info] downloading https://jcenter.bintray.com/org/codehaus/jackson/jackson-xc/1.8.3/jackson-xc-1.8.3.jar ...
[info]  [SUCCESSFUL ] org.codehaus.jackson#jackson-xc;1.8.3!jackson-xc.jar (3959ms)
[info] Done updating.
[info] Compiling 1 Scala source to /Users/jiangzl/Desktop/test/target/scala-2.11/classes...
[info] 'compiler-interface' not yet compiled for Scala 2.11.7. Compiling...
[info]   Compilation completed in 20.389 s
[info] Packaging /Users/jiangzl/Desktop/test/target/scala-2.11/simple-project_2.11-1.0.jar ...
[info] Done packaging.
[success] Total time: 103 s, completed 2016-4-22 15:01:21

$ tree -L 2 target

target
├── resolution-cache
│   ├── reports
│   └── simple-project
├── scala-2.11
│   ├── classes
│   └── simple-project_2.11-1.0.jar
└── streams
    ├── $global
    └── compile

$ ./bin/spark-submit --class "example" ~/Desktop/test/target/scala-2.11/simple-project_2.11-1.0.jar

说明确实成功执行了!(安装java/Scala请自己百度一下;如果是sbt打包的问题,这个只有慢慢下载,要怪只能怪我们国家牛逼的网管叻)

展开阅读全文
打赏
1
0 收藏
分享
加载中
更多评论
打赏
0 评论
0 收藏
1
分享
返回顶部
顶部