Javascript：使用字典过滤掉字符串中的单词？

Question

我需要从字符串中过滤掉几百个“停止”字词。由于有很多“停”的话，我不认为这将是一个好主意，做这样的事情：

sentence.replace(/\b(?:the|it is|we all|an?|by|to|you|[mh]e|she|they|we...)\b/ig, ''); 

我怎样才能创造这样一个哈希表来存储停止的话吗？在这张图中，一个关键词本身就是一个停词，价值并不重要。然后过滤将导致检查停用词映射中该单词是否不存在。用什么数据结构来构建这样的地图？

Answer 1

没有什么东西可以打败这类工作的正则表达式。但是，他们有两个问题 - 难以维护（您在帖子中指出的内容）以及非常大的性能问题。我不知道一个正则表达式可以处理多少个替代方案，但是我认为在任何情况下都可以达到20-30。

因此，您需要一些代码来从一些数据结构（可以是数组或单个字符串）动态构建正则表达式。我个人更喜欢刺痛，因为它最容易维护。

// taken from http://www.ranks.nl/resources/stopwords.html 
stops = "" 
+"a about above after again against all am an and any are aren't as " 
+"at be because been before being below between both but by can't " 
+"cannot could couldn't did didn't do does doesn't doing don't down " 
+"during each few for from further had hadn't has hasn't have  " 
+"haven't having he he'd he'll he's her here here's hers herself  " 
+"him himself his how how's i i'd i'll i'm i've if in into is isn't " 
+"it it's its itself let's me more most mustn't my myself no nor  " 
+"not of off on once only or other ought our ours ourselves out  " 
+"over own same shan't she she'd she'll she's should shouldn't so " 
+"some such than that that's the their theirs them themselves then " 
+"there there's these they they'd they'll they're they've this  " 
+"those through to too under until up very was wasn't we we'd we'll " 
+"we're we've were weren't what what's when when's where where's  " 
+"which while who who's whom why why's with won't would wouldn't  " 
+"you you'd you'll you're you've your yours yourself yourselves  " 

// how many to replace at a time 
reSize = 20 

// build regexps 
regexes = [] 
stops = stops.match(/\S+/g).sort(function(a, b) { return b.length - a.length }) 
for (var n = 0; n < stops.length; n += reSize) 
    regexes.push(new RegExp("\\b(" + stops.slice(n, n + reSize).join("|") + ")\\b", "gi")); 

一旦你得到了这一点，其余的是很明显的：

regexes.forEach(function(r) { 
    text = text.replace(r, '') 
}) 

您需要reSize值进行实验，找出了正则表达式的长度和正则表达式的总数之间的最佳平衡。如果性能至关重要，您还可以运行一次生成部分，然后在某处缓存结果（即生成的正则表达式）。