OutOfMemoryException读取Json字符串时

问题描述：

我试图从大小约为1.7 GB的Web Feed中反序列化Json数据。我开始用下面的代码：OutOfMemoryException读取Json字符串时

public override void CreateNewOutputRows() 
{ 

    //Set Webservice URL 
    string wUrl = "webserviceURLgoeshere"; 

    try 
    { 

     RootObject outPutResponse = GetWebServiceResult(wUrl); 

     foreach (Impression imp in outPutResponse.impressions) 
     { 

      ImpressionsSheetOutputBuffer.AddRow(); 
      ImpressionsSheetOutputBuffer.token = imp.token; 
      ImpressionsSheetOutputBuffer.userid = imp.userid; 
      ImpressionsSheetOutputBuffer.itemid = imp.itemid; 
      ImpressionsSheetOutputBuffer.view = imp.view; 
      ImpressionsSheetOutputBuffer.imageguid = imp.imageguid; 
      ImpressionsSheetOutputBuffer.bytes = imp.bytes; 
      ImpressionsSheetOutputBuffer.format = imp.format; 

      ImpressionIDBuffer.AddRow(); 
      ImpressionIDBuffer.oid = imp.imId.oid; 

      ImpressionParamsBuffer.AddRow(); 
      ImpressionParamsBuffer.origformat = imp.imParams.origFormat; 
      ImpressionParamsBuffer.size = imp.imParams.size; 

      ImpressionTimeBuffer.AddRow(); 
      ImpressionTimeBuffer.numLong = Int32.Parse(imp.imTime.numLong); 
     } 
    } 

    catch (Exception e) 
    { 
     FailComponent(e.ToString()); 
    } 
} 

private RootObject GetWebServiceResult(string wUrl) 
{ 

    HttpWebRequest httpWReq = (HttpWebRequest)WebRequest.Create(wUrl); 
    HttpWebResponse httpWResp = (HttpWebResponse)httpWReq.GetResponse(); 
    RootObject jsonResponse = null; 

    try 
    { 

     if (httpWResp.StatusCode == HttpStatusCode.OK) 
     { 

      Stream responseStream = httpWResp.GetResponseStream(); 
      string jsonString = null; 

      using (StreamReader reader = new StreamReader(responseStream)) 
      { 
       jsonString = reader.ReadToEnd(); 
       reader.Close(); 
      } 

      JavaScriptSerializer sr = new JavaScriptSerializer(); 
      jsonResponse = sr.Deserialize<RootObject>(jsonString); 

     } 

     else 
     { 
      FailComponent(httpWResp.StatusCode.ToString()); 

     } 
    } 

    catch (Exception e) 
    { 
     FailComponent(e.ToString()); 
    } 
    return jsonResponse; 
} 

private void FailComponent(string errorMsg) 
{ 
    bool fail = false; 
    IDTSComponentMetaData100 compMetadata = this.ComponentMetaData; 
    compMetadata.FireError(1, "Error Getting Data From Webservice!", errorMsg, "", 0, out fail); 

}

}

public class Id { 

    public string oid { get; set; } 
} 

public class Params { 

    public string origFormat { get; set; } 
    public string size { get; set; } 
} 

public class Time { 

    public string numLong { get; set; } 
} 

public class Impression { 

    public Id imId { get; set; } 
    public string token { get; set; } 
    public string userid { get; set; } 
    public string itemid { get; set; } 
    public string view { get; set; } 
    public string imageguid { get; set; } 
    public int bytes { get; set; } 
    public string format { get; set; } 
    public Params imParams { get; set; } 
    public Time imTime { get; set; } 
} 

public class RootObject { 
    public List<Impression> impressions { get; set; } 
}

然而，StreamReader的ReadToEnd的方法是在异常得到投掷，作为数据的大小太大。

我试图改变该代码如下：

Stream responseStream = httpWResp.GetResponseStream(); 

StreamReader reader = new StreamReader(responseStream); 

using (var myjson = new JsonTextReader(reader)) 
{ 
    JsonSerializer myserialization = new JsonSerializer(); 
    return (List<RootObject>)myserialization.Deserialize(myjson, typeof(List<RootObject>)); 
}

这给了我，我不能隐式转换类型List<RootObject>到RootObject错误。有没有人看到我可能做错了，我无法进行此转换？我使用this question来解决OutOfMemory异常，但现在它不返回反序列化的项目。任何意见将不胜感激。

编辑： JSON数据如下所示：

{ 
"_id": { 
    "$oid": "000000000000000000000000" 
    }, 
"token": "00000000-0000-0000-0000-000000000000", 
"userId": "username", 
"itemId": "00000000-0000-0000-0000-000000000000", 
"view": "view1", 
"imageguid": "00000000-0000-0000-0000-000000000000", 
"bytes": 1000, 
"format": "PNG", 
"params": { 
    "originalFormat": "tif", 
    "size": "50x50" 
    }, 
"time": { 
    "$numberLong": "1458748200000" 
    } 
} 
{ 
"_id": { 
    "$oid": "100000000000000000000000" 
    }, 
"token": "00000000-0000-0000-0000-000000000000", 
"userId": "username", 
"itemId": "00000000-0000-0000-0000-000000000000", 
"view": "view1", 
"imageguid": "00000000-0000-0000-0000-000000000000", 
"bytes": 1000, 
"format": "PNG", 
"params": { 
    "originalFormat": "tif", 
    "size": "50x50" 
    }, 
"time": { 
    "$numberLong": "1458748200000" 
    } 
}

64位应用应该可以正常工作。请确保你没有运行x86。 –

'〜1.7 GB'看起来你内存不足。解？不要这样做。怎么样？更好的设计。像什么？事物的多少。给我一个。逐步处理文件，不要尝试将其全部加载到内存中。怎么样。依靠。这就是为什么人们聘请开发人员：/ – Will

安东尼，如果你的新问题是“不能隐式转换类型'列表'到'RootObject'”那么这应该是你的问题，否则像@Will这样的人会尝试回答“内存不足“而不是你真正的问题，”为什么不是这种反序列化？“。 – Quantic

答

你应该创建一些规则，你如何分隔每个对象，并seperately序列化。

基本上你可以追加stream.ReadLine() 18倍（假设所有对象正是这样写你贴）

如果不是你应该使用stream.ReadLine()算你打开和关闭大括号，直到你达到每年年底对象并将它们分别序列化。

我猜测有更好的方法，但这些都很简单，应该可以解决你的问题...

OutOfMemoryException读取Json字符串时

相关推荐