effective crowd annotation for 【relation extraction】

crowdsource众包
identify instances of relations
supervised machine learning
distant supervision

crowdsourced data is as good as or better than that annotated by experts

effective crowd annotation for 【relation extraction】

gated instruction GI

Background and Related Work

Distant supervision (DS) is a method for training
extractors that obviates the need for human-labeled
training data by heuristically matching facts from a
background knowledge base (KB) to a large textual
corpus.

3 Gated Instruction Crowdsourcing

effective crowd annotation for 【relation extraction】

3 Interactive Tutorial Design

crowdsource step, most important
ensure workers understand the task
?
effective crowd annotation for 【relation extraction】

3.2 Adaptive Worder Screening

3.3 Motivational Feedback

4 Experimental Results

4.1 Quality of Gated Instruction Training

effective crowd annotation for 【relation extraction】

4.2 Integrating Crowdsourced Data with the Relation Extraction Pipeline

effective crowd annotation for 【relation extraction】

4.3 Effect of Data Quality on Extractor Performance