Spark + Log Service

Last Updated: Jun 11, 2017

Spark access to Log Service

This example demonstrates how Spark Streaming consumes the log data in Log Service and calculates the number of logs.

  1. if (args.length < 6) {
  2. System.err.println(
  3. """Usage: TestLoghub <sls project> <sls logstore> <loghub group name>
  4. | <sls endpoint> <receiver number> <batch interval seconds>
  5. """.stripMargin)
  6. System.exit(1)
  7. }
  8. val logserviceProject = args(0) // Log Service 中 project 名
  9. val logStoreName = args(1) // Log Service 中 logstore 名
  10. val loghubGroupName = args(2) // Jobs with the same loghubGroupName will consume data in the logstore together.
  11. val loghubEndpoint = args(3) // The data class API endpoint of Alibaba Cloud log service
  12. val accessKeyId = “<accessKeyId>” // The AccessKeyId for accessing the log service
  13. val accessKeySecret = “<accessKeySecret>” // The AccessKeySecret for accessing the log service
  14. val numReceivers = args(4).toInt // How many receivers have started to read data in the logstore
  15. val batchInterval = Milliseconds(args(5).toInt * 1000) // The interval between processing batches in Spark Streaming [A1]
  16. val conf = new SparkConf().setAppName("Test Loghub Streaming")
  17. val ssc = new StreamingContext(conf, batchInterval)
  18. val loghubStream = LoghubUtils.createStream(
  19. ssc,
  20. loghubProject,
  21. logStream,
  22. loghubGroupName,
  23. endpoint,
  24. numReceivers,
  25. accessKeyId,
  26. accessKeySecret,
  27. StorageLevel.MEMORY_AND_DISK)
  28. loghubStream.foreachRDD(rdd => println(rdd.count()))
  29. ssc.start()
  30. ssc.awaitTermination()

Description

  • E-MapReduce machines cannot connect to the Internet (except the master node).
  • Take caution when using intranet endpoints provided by the Log Service. Otherwise the requests to the Log Service will fail.
  • For more details about Log Service, refer to these Related Documents.

Appendix

For a complete sample code, see:

Thank you! We've received your feedback.