E-MapReduce:Connect Spark to OSS

Use Spark Shell (Scala)

1. Log on to the master node via SSH.

2. Start Spark Shell:

   spark-shell

3. Run the following code. Replace <yourBucket> with your OSS bucket name.

   scala> val pathIn = "oss://<yourBucket>/path/to/read"
   scala> val inputData = sc.textFile(pathIn)
   scala> val cnt = inputData.count
   cnt: Long = ...
   scala> println(s"count: $cnt")
   scala> val outputPath = "oss://<yourBucket>/path/to/write"
   scala> val outputData = inputData.map(e => s"$e has been processed.")
   scala> outputData.saveAsTextFile(outputPath)

   The code reads all lines from the input path, counts them, appends a suffix to each line, and writes the result to the output path. For the complete sample, see SparkOssDemo.scala on GitHub.

Use PySpark

1. Log on to the master node via SSH.

2. Start PySpark:

   pyspark

3. Run the following code. Replace <yourBucket> with your OSS bucket name.

   >>> pathIn = "oss://<yourBucket>/path/to/read"
   >>> df = spark.read.text(pathIn)
   >>> cnt = df.count()
   >>> print(cnt)
   >>> outputPath = "oss://<yourBucket>/path/to/write"
   >>> df.write.format("parquet").mode("overwrite").save(outputPath)

   The code reads the input path as a text DataFrame, prints the row count, and writes the result in Parquet format to the output path.

Use Spark SQL

1. Log on to the master node via SSH.

2. Start the Spark SQL CLI:

   spark-sql

3. Create a database stored in OSS, create a CSV table, and insert a row:

   Parameter	Description
   delimiter	The character used to separate fields in the CSV file.
   header	Set to true if the first row contains column names; false otherwise.

   CREATE DATABASE test_db LOCATION "oss://<yourBucket>/test_db";
   USE test_db;
   CREATE TABLE student (id INT, name STRING, age INT)
       USING CSV OPTIONS ("delimiter"=";", "header"="true");
   INSERT INTO student VALUES(1, "ab", 12);
   SELECT * FROM student;

   Replace <yourBucket> with your OSS bucket name. The SELECT statement returns:

   1    ab    12

4. To verify the result, check the CSV file in OSS. The file uses semicolon delimiters, with the first row as headers:

   id;name;age
   1;ab;12

Step 1: Remove the password-free configuration

Remove the fs.oss.credentials.provider parameter from the core-site.xml file of the Hadoop-Common service.

Step 2: Verify that password-free access is removed

Run the following command:

hadoop fs -ls oss://<yourBucket>/test_db

If the removal succeeded, you see:

ls: ERROR: not found login secrets, please configure the accessKeyId and accessKeySecret.

Step 3: Add the AccessKey parameters to core-site.xml

In the core-site.xml file of the Hadoop-Common service, add the following parameters:

Step 4: Verify the AccessKey configuration

Run the following command:

hadoop fs -ls oss://<yourBucket>/test_db

If the AccessKey pair is configured correctly, the output lists the OSS path:

drwxrwxrwx   - root root          0 2025-02-24 11:45 oss://<yourBucket>/test_db/student

Step 5: Restart Spark services

Restart all Spark-related services. After they are running, use Spark Shell, PySpark, or Spark SQL to read from and write to OSS.