Parallelized Collections
Parallelized Collections
月下独酌100 发表于3年前
Parallelized Collections
  • 发表于 3年前
  • 阅读 6
  • 收藏 0
  • 点赞 0
  • 评论 0


Parallelized collections are created by calling SparkContext’s parallelize method on an existing collection in your driver program (a Scala Seq). The elements of the collection are copied to form a distributed dataset that can be operated on in parallel. For example, here is how to create a parallelized collection holding the numbers 1 to 5:

val data = Array(1, 2, 3, 4, 5) val distData = sc.parallelize(data)

Once created, the distributed dataset (distData) can be operated on in parallel. For example, we might call distData.reduce((a, b) => a + b)to add up the elements of the array. We describe operations on distributed datasets later on.

One important parameter for parallel collections is the number of partitions to cut the dataset into. Spark will run one task for each partition of the cluster. Typically you want 2-4 partitions for each CPU in your cluster. Normally, Spark tries to set the number of partitions automatically based on your cluster. However, you can also set it manually by passing it as a second parameter to parallelize (e.g. sc.parallelize(data, 10)). Note: some places in the code use the term slices (a synonym for partitions) to maintain backward compatibility.

共有 人打赏支持
粉丝 26
博文 186
码字总数 56475
* 金额(元)
¥1 ¥5 ¥10 ¥20 其他金额
* 支付类型