iotop&iostat (load高 IO高的问题排查)
目标:使用iotop&top&jstack命令定位 io高的应用以及线程信息
实现:load过高但是cpu占用率低的模拟与验证
一般load高,说明系统大量的线程在等待执行
cpu us sy 显示都不高,并且wa 占用高,说明系统在等待大量的io操作
1、安装iotop
sudo yum install iotop
2、top命令显示
从top命令可以看出,系统load 6.43,对于4 cpu的系统来说,已经是超载,另外 wa 60%,可以确定系统的线程在等待io操作
其中 线程 2492 比较可疑
top -H -p 2492
3、iostat
iostat ALL :可以查阅 那个磁盘读写频繁
4、sudo iotop
可以查看哪些线程在 做读写io
FileReadWrite 应用:三个线程,每个线程都在每秒36M,42M,75M的速度读写io
试验测试代码为:
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
public class FileReadWrite {
public static void main(String[] args) {
for(int i =0;i<4;i++){
new T().start();
}
}
public static class T extends Thread{
@Override
public void run() {
while (true){
try {
exec();
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
private static void exec() throws Exception{
long start = System.currentTimeMillis();
String fileName = "A"+System.currentTimeMillis();
File f = new File(fileName);
if(!f.exists()){
System.out.println(fileName + " created ");
f.createNewFile();
}
FileOutputStream fi = new FileOutputStream(f);
int i = 0;
while(true){
i++;
byte[] data = new byte[1024*1024*5];
fi.write(data);
if(i>50){
break;
}
}
fi.close();
FileInputStream f2 = new FileInputStream(f);
byte[] data = new byte[1024*1024*1];
int ret = f2.read(data);
while(ret != -1){
data = new byte[1024*1024*1];
ret = f2.read(data);
}
boolean rt = f.delete();
System.out.println(fileName + " deleted " + rt + " time :" + (System.currentTimeMillis() - start));
}
}
jstack 可以看大量的线程阻塞在了read 和write上:
"DestroyJavaVM" prio=10 tid=0x00007fabb4008800 nid=0x1774 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Thread-3" prio=10 tid=0x00007fabb40d6800 nid=0x1784 runnable [0x00007faba7af9000] java.lang.Thread.State: RUNNABLE at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:325) at FileReadWrite.exec(FileReadWrite.java:42) at FileReadWrite.access$000(FileReadWrite.java:6) at FileReadWrite$T.run(FileReadWrite.java:19) "Thread-2" prio=10 tid=0x00007fabb40d4800 nid=0x1783 runnable [0x00007faba7bfa000] java.lang.Thread.State: RUNNABLE at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:243) at FileReadWrite.exec(FileReadWrite.java:54) at FileReadWrite.access$000(FileReadWrite.java:6) at FileReadWrite$T.run(FileReadWrite.java:19) "Thread-1" prio=10 tid=0x00007fabb40d2800 nid=0x1782 runnable [0x00007faba7cfb000] java.lang.Thread.State: RUNNABLE at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:243) at FileReadWrite.exec(FileReadWrite.java:54) at FileReadWrite.access$000(FileReadWrite.java:6) at FileReadWrite$T.run(FileReadWrite.java:19) "Thread-0" prio=10 tid=0x00007fabb40d0800 nid=0x1781 waiting on condition [0x00007faba7dfc000] java.lang.Thread.State: RUNNABLE at FileReadWrite.exec(FileReadWrite.java:53) at FileReadWrite.access$000(FileReadWrite.java:6) at FileReadWrite$T.run(FileReadWrite.java:19)