SolrJ - 使用ContentStreamUpdateRequest异步索引文档
问题描述:
我正在使用SolrJ API 4.8将富文档索引到solr。但我想要异步索引这些文档。我做的功能同步发送文件,但我不知道如何改变它使其异步。任何想法?SolrJ - 使用ContentStreamUpdateRequest异步索引文档
功能:
public Boolean indexDocument(HttpSolrServer server, String PathFile, InputReader external)
{
ContentStreamUpdateRequest up = new ContentStreamUpdateRequest("/update/extract");
try {
up.addFile(new File(PathFile), "text");
} catch (IOException e) {
Logger.getLogger(ANOIndexer.class.getName()).log(Level.SEVERE, null, e);
return false;
}
up.setAction(AbstractUpdateRequest.ACTION.COMMIT, true, true);
try {
server.request(up);
} catch (SolrServerException e) {
Logger.getLogger(ANOIndexer.class.getName()).log(Level.SEVERE, null, e);
return false;
} catch (IOException e) {
Logger.getLogger(ANOIndexer.class.getName()).log(Level.SEVERE, null, e);
return false;
}
return true;
}
Solr的服务器:4.8版
答
这听起来像你可能想看看使用的ExecutorService和FutureTask提供这样做:
private static HttpSolrServer server;
private static int threadPoolSize = 4; //Set this to something appropiate for your environment
public static void main(String[] args) {
ExecutorService executor = Executors.newFixedThreadPool(threadPoolSize);
ArrayList<FutureTask<Boolean>> taskList = new ArrayList<FutureTask<Boolean>>();
ArrayList<String> paths = new ArrayList<String>();
//Initialize your list of paths here
for (String path : paths) {
FutureTask<Boolean> futureTask = new FutureTask<Boolean>(new IndexDocumentTask(path));
taskList.add(futureTask);
executor.execute(futureTask);
}
for (int i = 0; i < taskList.size(); i++) {
FutureTask<Boolean> futureTask = taskList.get(i);
try {
System.out.println("Index Task " + i + (futureTask.get() ? " finished successfully." : " encountered an error."));
} catch (ExecutionException e) {
System.out.println("An Execution Exception occurred with Index Task " + i);
} catch (InterruptedException e) {
System.out.println("An Interrupted Exception occurred with Index Task " + i);
}
}
executor.shutdown();
}
static class IndexDocumentTask implements Callable<Boolean> {
private String pathFile;
public IndexDocumentTask(String pathFile) {
this.pathFile = pathFile;
}
@Override
public Boolean call() {
return indexDocument(pathFile);
}
public Boolean indexDocument(String pathFile) {
ContentStreamUpdateRequest up = new ContentStreamUpdateRequest("/update/extract");
try {
up.addFile(new File(pathFile), "text");
} catch (IOException e) {
Logger.getLogger(ANOIndexer.class.getName()).log(Level.SEVERE, null, e);
return false;
}
up.setAction(AbstractUpdateRequest.ACTION.COMMIT, true, true);
try {
server.request(up);
} catch (SolrServerException e) {
Logger.getLogger(ANOIndexer.class.getName()).log(Level.SEVERE, null, e);
return false;
} catch (IOException e) {
Logger.getLogger(ANOIndexer.class.getName()).log(Level.SEVERE, null, e);
return false;
}
return true;
}
}
这是未经测试的代码,所以我不知道是否像这样调用server.request(up)
是线程安全的。我认为只使用一个HttpSolrServer实例会更简洁,但您也可以在每个任务中创建新的HttpSolrServer实例。
如果您愿意,可以增加IndexDocumentTask以实现Callable<Tuple<String, Boolean>>
,以便您可以检索要索引的文档的文件名以及索引是否成功。
尽管我不认为一次向Solr服务器发送多个请求应该是个问题,但您可能想限制您的请求,以免超载Solr服务器。