Connect Spark to OSS

EMR provides the following features for accessing OSS:

• Supports MetaService.
• Allows you to access OSS without using an AccessKey pair.
• Allows you to access OSS by explicitly writing an AccessKey pair and an endpoint.
  Note The OSS endpoint must be an internal domain name. For more information, see Regions and endpoints.

The following example shows how Spark reads data from OSS and writes the processed data back to OSS without using an AccessKey pair.

val conf = new SparkConf().setAppName("Test OSS")
    val sc = new SparkContext(conf)
    val pathIn = "oss://bucket/path/to/read"
    val inputData = sc.textFile(pathIn)
    val cnt = inputData.count
    println(s"count: $cnt")
    val outputPath = "oss://bucket/path/to/write"
    val outpuData = inputData.map(e => s"$e has been processed.")
    outpuData.saveAsTextFile(outputPath)