阅读MSR意译语料库为大熊猫
问题描述:
我从MSR下载MSR复述语料库,并试图把它载入数据帧,但得到了以下错误:阅读MSR意译语料库为大熊猫
import pandas as pd
df = pd.read_csv(r'C:\MSRParaphraseCorpus\msr_paraphrase_test.txt', sep = '\t')
错误:
CParserError Traceback (most recent call last)
<ipython-input-10-35d992467320> in <module>()
----> 1 df = pd.read_csv(r'C:\MSRParaphraseCorpus\msr_paraphrase_test.txt', sep = '\t')
c:\python34\lib\site-packages\pandas\io\parsers.py in parser_f(filepath_or_buffer, sep, dialect, compression, doublequote, escapechar, quotechar, quoting, skipinitialspace, lineterminator, header, index_col, names, prefix, skiprows, skipfooter, skip_footer, na_values, na_fvalues, true_values, false_values, delimiter, converters, dtype, usecols, engine, delim_whitespace, as_recarray, na_filter, compact_ints, use_unsigned, low_memory, buffer_lines, warn_bad_lines, error_bad_lines, keep_default_na, thousands, comment, decimal, parse_dates, keep_date_col, dayfirst, date_parser, memory_map, float_precision, nrows, iterator, chunksize, verbose, encoding, squeeze, mangle_dupe_cols, tupleize_cols, infer_datetime_format, skip_blank_lines)
472 skip_blank_lines=skip_blank_lines)
473
--> 474 return _read(filepath_or_buffer, kwds)
475
476 parser_f.__name__ = name
c:\python34\lib\site-packages\pandas\io\parsers.py in _read(filepath_or_buffer, kwds)
258 return parser
259
--> 260 return parser.read()
261
262 _parser_defaults = {
c:\python34\lib\site-packages\pandas\io\parsers.py in read(self, nrows)
719 raise ValueError('skip_footer not supported for iteration')
720
--> 721 ret = self._engine.read(nrows)
722
723 if self.options.get('as_recarray'):
c:\python34\lib\site-packages\pandas\io\parsers.py in read(self, nrows)
1168
1169 try:
-> 1170 data = self._reader.read(nrows)
1171 except StopIteration:
1172 if nrows is None:
pandas\parser.pyx in pandas.parser.TextReader.read (pandas\parser.c:7566)()
pandas\parser.pyx in pandas.parser.TextReader._read_low_memory (pandas\parser.c:7806)()
pandas\parser.pyx in pandas.parser.TextReader._read_rows (pandas\parser.c:8423)()
pandas\parser.pyx in pandas.parser.TextReader._tokenize_rows (pandas\parser.c:8297)()
pandas\parser.pyx in pandas.parser.raise_parser_error (pandas\parser.c:20715)()
CParserError: Error tokenizing data. C error: Expected 5 fields in line 34, saw 6
所以我看了看线34,看起来非常好。
fname = r'C:\MSRParaphraseCorpus\msr_paraphrase_test.txt'
with open(fname, encoding="utf8") as f:
content = f.readlines()
content[34]
输出:
'0\t1268500\t1268733\tAgainst the Japanese currency, the euro was at 135.92/6.04 yen against the late New York level of 136.03/14.\tThe dollar was at 117.85 yen against the Japanese currency, up 0.1 percent.\n'
有什么想法?
P.S.添加一些更多的细节,因为SO会让我发帖而没有详细信息...
答
问题在于第34行的开引号(正如我的评论中提到的)。通过传入csv.QUOTE_NONE来禁用csv阅读器的引用。尝试:
import csv
import pandas as pd
df = pd.read_csv(r'C:\MSRParaphraseCorpus\msr_paraphrase_test.txt', sep = '\t', quoting=csv.QUOTE_NONE)
34行实际上是内容[33]和该行的问题是,最后一个字段有一个开始报价,但没有结束。你需要做pd.read_csv(r'C:\ MSRParaphraseCorpus \ msr_paraphrase_test.txt',sep ='\ t',quotechar ='')让csv阅读器忽略引号。不作为答案张贴,因为在解决该问题之后尝试加载文件还有其他问题。额外的问题不会跳出来,错误不是很清楚。但我会看起来更多一点。 – clockwatcher