This topic describes how to upload a JAR package or resource file exceeding 10 MB to DataWorks when running a MapReduce job.
- Upload resources exceeding 10 MB through the MaxCompute CLI client. Before you can upload the resource, first complete the following procedure.
- Currently, resources uploaded through the MaxCompute CLI client cannot be viewed in the resource list in the DataWorks console. You can view and confirm the resources by running the list resources command.
//View resources. list resources;
- Reduce the JAR file size and run the MapReduce jobs on DataWorks on your local server. You need to keep only one main function.
jar -resources test_mr.jar,test_ab.jar --The list resources command directly runs after the function is registered on the MaxCompute CLI client. -classpath test_mr.jar --Size reduction policy: A mapper and a reducer that are related to the main function are required to submit the size reduction policy to a gateway. The third-party dependency is not required. Other resources can be stored on the resources directory. com.aliyun.odps.examples.mr.test_mr wc_in wc_out;
With the preceding methods, you can use the scheduling feature to run MapReduce jobs exceeding 10 MB on DataWorks regularly.