spark shuffle优化-参数调优
spark shuffle优化-参数调优
常用参数列举:
- spark.reducer.maxSizeInFlight
- spark.shuffle.compress
- spark.shuffle.file.buffer
- spark.shuffle.io.maxRetries
- spark.shuffle.io.retryWait
调优参数一定要设置合理,效率才能事半功倍
spark shuffle优化-参数调优
常用参数列举:
调优参数一定要设置合理,效率才能事半功倍