#一.优化方向，序列化
##1.官网位置
![](http://img.blog.itpub.net/blog/2019/07/25/dd842c04c9e20c85.png?x-oss-process=style/bb)
##2.解释：
```
默认用的是java序列化，但是会很慢，第二种很快，但是不一定能实现所有序列化
第二种，有些自定义类你需要在代码中注册（Kryo）
```
##3.StorageLevel.MEMORY_ONLY) 方式存储代码
```scala
  def main(args: Array[String]) {
    val sparkConf = new SparkConf()
    val sc = new SparkContext(sparkConf)
    val names = Array[String]("G304","G305","G306")
    val genders = Array[String]("male","female")
    val addresses = Array[String]("beijing","shenzhen","wenzhou","hangzhou")

val infos = new ArrayBuffer[Info]()
    for (i<-1 to 1000000){
      val name = names(Random.nextInt(3))
      val gender = genders(Random.nextInt(2))
      val address = addresses((Random.nextInt(4)))
      infos += Info(name, gender, address)
    }

val rdd = sc.parallelize(infos)
    rdd.persist(StorageLevel.MEMORY_ONLY_SER)
    rdd.count()
    // rdd.persist(StorageLevel.MEMORY_ONLY)

sc.stop()
  }
  case class Info(name:String, gender:String, address:String)
}
```
##4.结果34.3 java序列化默认
![](http://img.blog.itpub.net/blog/2019/07/25/5f140dcef4f5cf22.png?x-oss-process=style/bb)
#二.Kyro序列化
##1.配置文件位置及配置（spark.serialize）在spark-default.conf
![](http://img.blog.itpub.net/blog/2019/07/25/1ec216e9a449eb1f.png?x-oss-process=style/bb)
##2.代码
```scala
 def main(args: Array[String]) {
    val sparkConf = new SparkConf()
    sparkConf.registerKryoClasses(Array(classOf[Info]))
    val sc = new SparkContext(sparkConf)
    val names = Array[String]("G304","G305","G306")
    val genders = Array[String]("male","female")
    val addresses = Array[String]("beijing","shenzhen","wenzhou","hangzhou")
    val infos = new ArrayBuffer[Info]()
    for (i<-1 to 1000000){
      val name = names(Random.nextInt(3))
      val gender = genders(Random.nextInt(2))
      val address = addresses((Random.nextInt(4)))
      infos += Info(name, gender, address)
    }
    val rdd = sc.parallelize(infos)
    rdd.persist(StorageLevel.MEMORY_ONLY_SER)
    rdd.count()
    // rdd.persist(StorageLevel.MEMORY_ONLY_SER)
    sc.stop()
```
##3.没注册情况下截图
![](http://img.blog.itpub.net/blog/2019/07/25/500871e1f893817a.png?x-oss-process=style/bb)
##4.官网没注册结果
![](http://img.blog.itpub.net/blog/2019/07/25/0dd134907d1e4809.png?x-oss-process=style/bb)
##5.加上一句话，否则所有的东西都要加入变大
```scala
sparkConf.registerKryoClasses(Array(classOf[Info]))
```

ITPub博客

spark性能优化几点注意

最新文章