Spark + OSS

Spark access to OSS

Currently，E-MapReduce supports MetaService, which allows you to access OSS without AccessKey. Entering AccessKey and Endpoint is also supported， please use internal domain names for OSS endpoints. For more details, refer to OSS Endpoint.

This example demonstrates how to read data from OSS in Spark, and write the processed data back to OSS.

    val conf = new SparkConf().setAppName("Test OSS")
    val sc = new SparkContext(conf)
    val pathIn = "oss://bucket/path/to/read"
    val inputData = sc.textFile(pathIn)
    val cnt = inputData.count
    println(s"count: $cnt")
    val outputPath = "oss://bucket/path/to/write"
    val outpuData = inputData.map(e => s"$e has been processed.")
    outpuData.saveAsTextFile(outputPath)

Appendix

For a complete sample code, refer to:

• Spark access to OSS