Simple operations on OSS files

Dependency conflict when using OSS SDK

If you cannot directly perform operations on the files in OSS through OSS SDK in Spark or Hadoop jobs, it is because there are conflicts between the http-client-4.4.x version that OSS SDK is dependent on and the http-client version in your Spark or Hadoop running environment. If such operations are required, you must solve the dependency conflicts first. By selecting E-MapReduce, Spark and Hadoop are already seamlessly compatibile with OSS, and you can operate on OSS files as you use HDFS.

The Internet endpoints of OSS are only needed when you perform local testing, so that you can access OSS data locally.

All endpoints of OSS can be found here.

Recommended action

Follow these steps to query files under OSS directories:

   [Scala] 
   import org.apache.hadoop.conf.Configuration
   import org.apache.hadoop.fs.{Path, FileSystem}
   val dir = "oss://bucket/dir"
   val path = new Path(dir)
   val conf = new Configuration()
   conf.set("fs.oss.impl", "com.aliyun.fs.oss.nat.NativeOssFileSystem")
   val fs = FileSystem.get(path.toUri, conf)
   val fileList = fs.listStatus(path)
   ...
   [Java]
   import org.apache.hadoop.conf.Configuration;
   import org.apache.hadoop.fs.Path;
   import org.apache.hadoop.fs.FileStatus;
   import org.apache.hadoop.fs.FileSystem;
   String dir = "oss://bucket/dir";
   Path path = new Path(dir);
   Configuration conf = new Configuration();
   conf.set("fs.oss.impl", "com.aliyun.fs.oss.nat.NativeOssFileSystem");
   FileSystem fs = FileSystem.get(path.toUri(), conf);
   FileStatus[] fileList = fs.listStatus(path);
   ...