在线时间:8:00-16:00
迪恩网络APP
随时随地掌握行业动态
扫描二维码
关注迪恩网络微信公众号
开源软件名称(OpenSource Name):faizanahemad/spark-gradle-template开源软件地址(OpenSource Url):https://github.com/faizanahemad/spark-gradle-template开源编程语言(OpenSource Language):Scala 100.0%开源软件介绍(OpenSource Introduction):Spark-Gradle-TemplateA barebones project with scala, apache spark built using gradle. Spark-shell provides PrerequisitesBuild and Demo processClone the Repo
Build
Run
All Together
What the demo does?Take a look at src->main->scala->template->spark directory We have two Items here. The trait The file
InitSpark.scala trait InitSpark {
val spark: SparkSession = SparkSession.builder().appName("Spark example").master("local[*]")
.config("spark.some.config.option", "some-value").getOrCreate()
val sc = spark.sparkContext
val sqlContext = spark.sqlContext
def reader = spark.read.option("header",true).option("inferSchema", true).option("mode", "DROPMALFORMED")
def readerWithoutHeader = spark.read.option("header",true).option("inferSchema", true).option("mode", "DROPMALFORMED")
private def init = {
sc.setLogLevel("ERROR")
Logger.getLogger("org").setLevel(Level.ERROR)
Logger.getLogger("akka").setLevel(Level.ERROR)
LogManager.getRootLogger.setLevel(Level.ERROR)
}
init
def close = {
spark.close()
}
} Main.scala final case class Person(firstName: String, lastName: String, country: String, age: Int)
object Main extends InitSpark {
def main(args: Array[String]) = {
import spark.implicits._
val version = spark.version
println("VERSION_STRING = " + version)
val sumHundred = spark.range(1, 101).reduce(_ + _)
println(sumHundred)
val persons = reader.csv("people-example.csv").as[Person]
val averageAge = persons.agg(avg("age")).first.get(0).asInstanceOf[Double]
println(f"Average Age: $averageAge%.2f")
close
}
} Using this RepoJust import it into your favorite IDE as a gradle project. Tested with IntelliJ to work. Or use your favorite editor and build from command line with gradle. Libraries Included
Useful LinksIssues or Suggestions
|
2023-10-27
2022-08-15
2022-08-17
2022-09-23
2022-08-13
请发表评论