CDH 中hbaese, mapreduce内存控制

1, hbase启动失败:OutOfMemoryError

由于内存不够,导入大批量数据时发生内存溢出,所以需要调整内存分配,在浏览器打开7180集群管理界面,进入到hbase

1, 配置–>输入框搜索:java

CDH 中hbaese, mapreduce内存控制

2,mapreduce运行卡顿

查看/var/log/ hadoop-cmf-yarn-JOBHISTORY-node1.cluster.log.out ,提示gc频繁,所以想到给mapreduce分配的内存不足, 需要加大内存

2019-02-23 08:00:21,593 WARN com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: Failed to publish event: SimpleEvent{attributes={ROLE_TYPE=[JOBHISTORY], CATEGORY=[LOG_MESSAGE], ROLE=[yarn-JOBHISTORY-a6c222f47368c75014f30b4eb3a45e95], SEVERITY=[IMPORTANT], SERVICE=[yarn], HOST_IDS=[f64f5e51-45af-4278-88e1-b18ccaea3de6], SERVICE_TYPE=[YARN], LOG_LEVEL=[WARN], HOSTS=[node1.cluster], EVENTCODE=[EV_L
OG_EVENT]}, content=Detected pause in JVM or host machine (eg GC): pause of approximately 14944ms
GC pool 'ParNew' had collection(s): count=1 time=14631ms, timestamp=1550926817327}
2019-02-23 10:19:51,816 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1009ms
No GCs detected
2019-02-23 10:35:18,805 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5783ms
GC pool 'ParNew' had collection(s): count=1 time=5852ms

修改mapred-site.xml中的map , reduce的内存参数

  • io.sort.mb map缓冲区大小//默认:256m
  • io.sort.factor 并行处理spill的个数//默认:64
    在浏览器打开7180集群管理界面,进入到yarn

1,配置–>输入框搜索:java

会出现对应的java堆内存参数,修改后保存
CDH 中hbaese, mapreduce内存控制
CDH 中hbaese, mapreduce内存控制

2, 配置–>输入框搜索:memory

CDH 中hbaese, mapreduce内存控制