阅读大文件错误“outofmemoryerror”（java）

Question

对不起，我的英语。我想读一个大文件，但是当我读取错误发生时outOfMemoryError。我不明白如何在应用程序中处理内存。以下代码不起作用：

try { 

    StringBuilder fileData = new StringBuilder(1000); 
    BufferedReader reader = new BufferedReader(new FileReader(file)); 

    char[] buf = new char[8192]; 
    int bytesread = 0, 
     bytesBuffered = 0; 

    while((bytesread = reader.read(buf)) > -1) { 

     String readData = String.valueOf(buf, 0, bytesread); 
     bytesBuffered += bytesread; 

     fileData.append(readData); //this is error 

     if (bytesBuffered > 1024 * 1024) { 
      bytesBuffered = 0; 
     } 
    } 

    System.out.println(fileData.toString().toCharArray()); 
} finally { 

} 

Answer 1

您需要预先分配一个大缓冲区以避免重新分配。

File file = ...; 
StringBuilder fileData = new StringBuilder(file.size()); 

并备有大量堆大小运行：

java -Xmx2G 

====更新

while循环利用缓冲区并不需要太多的内存来运行。将输入视为流，将搜索字符串与流匹配。这是一个非常简单的状态机。如果你需要搜索多个单词，你可以找到一个TrieTree实现（支持流）。

// the match state model 
...xxxxxxabxxxxxaxxxxxabcdexxxx... 
     ab  a  abcd 

    File file = new File("path_to_your_file"); 
    String yourSearchWord = "abcd"; 
    int matchIndex = 0; 
    boolean matchPrefix = false; 
    try (BufferedReader reader = new BufferedReader(new FileReader(file))) { 
     int chr; 
     while ((chr = reader.read()) != -1) { 
      if (matchPrefix == false) { 
       char searchChar = yourSearchWord.charAt(0); 
       if (chr == searchChar) { 
        matchPrefix = true; 
        matchIndex = 0; 
       } 
      } else { 
       char searchChar = yourSearchWord.charAt(++matchIndex); 
       if (chr == searchChar) { 
        if (matchIndex == yourSearchWord.length() - 1) { 
         // match!! 
         System.out.println("match: " + matchIndex); 
         matchPrefix = false; 
         matchIndex = 0; 
        } 
       } else { 
        matchPrefix = false; 
        matchIndex = 0; 
       } 
      } 
     } 
    } 

Answer 2

试试这个。这可能会有所帮助： -

try{ 
    BufferedReader reader = new BufferedReader(new FileReader(file)); 
    String txt = ""; 
    while((txt = reader.read()) != null){ 
     System.out.println(txt); 
    } 
}catch(Exception e){ 
    System.out.println("Error : "+e.getMessage()); 
} 

Answer 3

你不应该在内存中保存这样的大文件，因为你用完了，就像你看到的那样。由于您使用Java 7，因此您需要手动将文件作为流读取，并即时检查内容。否则，你可以使用Java 8的流API。这只是一个例子。它的工作原理，但要记住，该发现字的位置可能会因为编码的问题各不相同，所以这是没有生产代码：

import java.io.File; 
import java.io.FileInputStream; 
import java.io.IOException; 

public class FileReader 
{ 
    private static String wordToFind = "SEARCHED_WORD"; 
    private static File file = new File("YOUR_FILE"); 
    private static int currentMatchingPosition; 
    private static int foundAtPosition = -1; 
    private static int charsRead; 

    public static void main(String[] args) throws IOException 
    { 
     try (FileInputStream fis = new FileInputStream(file)) 
     { 
      System.out.println("Total size to read (in bytes) : " + fis.available()); 

      int c; 
      while ((c = fis.read()) != -1) 
      { 
       charsRead++; 
       checkContent(c); 
      } 

      if (foundAtPosition > -1) 
      { 
       System.out.println("Found word at position: " + (foundAtPosition - wordToFind.length())); 
      } 
      else 
      { 
       System.out.println("Didnt't find the word!"); 
      } 

     } 
     catch (IOException e) 
     { 
      e.printStackTrace(); 
     } 
    } 

    private static void checkContent(int c) 
    { 
     if (currentMatchingPosition >= wordToFind.length()) 
     { 
      //already found.... 
      return; 
     } 

     if (wordToFind.charAt(currentMatchingPosition) == (char)c) 
     { 
      foundAtPosition = charsRead; 
      currentMatchingPosition++; 
     } 
     else 
     { 
      currentMatchingPosition = 0; 
      foundAtPosition = -1; 
     } 
    } 
}