达观杯文本处理任务

达观杯文本处理任务

​ 达观杯文本处理小demo如下图，训练数据与测试数据有2G多，在pycharm里运行demo时一直报错Process finished with exit code -1073740940 (0xC0000374)，解决办法：

• pandas包更新到最新版
• pycharm中后缀为vmoptions的配置文件里Xms、Xmx的值调大（不明白原因）

配置文件：

demo：

import pandas as pd
# 逻辑回归
from sklearn.linear_model import LogisticRegression
# 文本特征提取 词汇在文本中出现的频率
from sklearn.feature_extraction.text import CountVectorizer

df_train = pd.read_csv('./train_set.csv')
df_test = pd.read_csv('./test_set.csv')
# inplace为True直接修改原数组 False将新结果赋给新的数组
df_train.drop(columns=['article', 'id'], inplace=True)
df_test.drop(columns=['article'], inplace=True)

vec = CountVectorizer(ngram_range=(1,2), min_df=3, max_df=0.9, max_features=100000)
vec.fit(df_train['word_seg'])
x_train = vec.transform(df_train['word_seg'])
x_test = vec.transform(df_test['word_seg'])
y_train = df_train['class'] - 1

lg = LogisticRegression(C=4, dual=True)
lg.fit(x_train,y_train)
y_test = lg.predict(x_test)
df_test['class'] = y_test.tolist()
df_test['class'] = df_test['class'] + 1
df_result = df_test.loc[:, ['id', 'class']]
df_result.to_csv('./result.csv', index=False)

print('完成............')