effective crowd annotation for 【relation extraction】

crowdsource众包
identify instances of relations
supervised machine learning
distant supervision

crowdsourced data is as good as or better than that annotated by experts

gated instruction GI

Background and Related Work

Distant supervision (DS) is a method for training
extractors that obviates the need for human-labeled
training data by heuristically matching facts from a
background knowledge base (KB) to a large textual
corpus.

3 Gated Instruction Crowdsourcing

3 Interactive Tutorial Design

crowdsource step, most important
ensure workers understand the task
?

3.2 Adaptive Worder Screening

3.3 Motivational Feedback

4 Experimental Results

4.1 Quality of Gated Instruction Training

4.2 Integrating Crowdsourced Data with the Relation Extraction Pipeline

4.3 Effect of Data Quality on Extractor Performance