Spring批处理分区步骤未运行
我试图配置我的第一个多线程作业。我们需要处理大约200,000条记录的主文件。我想将文件分解成10个文件并处理它们。分割文件tasklet工作正常&它将bug文件分成10个小文件。我将作业参数键中的文件路径传递给作业“urlFilesPath”,并将其值设为“file:/ scrapper/spliturl *”。我验证了在MultiResourcePartitioner bean中正在正确读取文件。Spring批处理分区步骤未运行
主步骤在我的配置中运行,但从站步骤不运行。以下是我的配置。
分区程序:
<bean id="partitioner"
class="org.springframework.batch.core.partition.support.MultiResourcePartitioner"
scope="step">
<property name="resources" value="#{jobParameters['urlFilesPath']}" />
</bean>
MultiResourceItemReader:
<bean id="multiResourceItemReader"
class="org.springframework.batch.item.file.MultiResourceItemReader"
scope="step">
<property name="resources" value="#{jobParameters['urlFilesPath']}" />
<property name="delegate" ref="urlFileItemReader" />
<property name="strict" value="true" />
<property name="saveState" value="false" />
</bean>
FlatFileItemWriter:
<bean id="urlFileItemReader" class="org.springframework.batch.item.file.FlatFileItemReader"
scope="step">
<property name="lineMapper" ref="passThroughLineMapper" />
<property name="resource" value="#{stepExecutionContext['fileName']}" />
<property name="saveState" value="true" />
</bean>
作业配置:
<batch:job id="importJob" job-repository="jobRepository">
<batch:step id="fileSplitter" next="readURLFileRunner">
<batch:tasklet ref="fileSplittingTasklet"
transaction-manager="transactionManager" />
</batch:step>
<batch:step id="readURLFileRunner">
<batch:partition step="readURLFile" partitioner="partitioner">
<batch:handler grid-size="10" task-executor="taskExecutor" />
</batch:partition>
</batch:step>
</batch:job>
从步骤配置:
<batch:step id="readURLFile">
<batch:tasklet transaction-manager="transactionManager"
task-executor="taskExecutor" throttle-limit="10">
<batch:chunk reader="multiResourceItemReader" processor="urlFileItemProcessor"
writer="validURLItemWriter" commit-interval="200" skip-limit="100">
<batch:skippable-exception-classes>
<batch:include class="java.net.MalformedURLException" />
<batch:include class="java.net.URISyntaxException" />
<batch:include class="java.net.UnknownHostException" />
</batch:skippable-exception-classes>
</batch:chunk>
<batch:listeners>
<batch:listener ref="malformedURLExceptionListener" />
<batch:listener ref="uriSyntaxExceptionListener" />
<batch:listener ref="unknownHostExceptionListener" />
</batch:listeners>
</batch:tasklet>
<batch:end on="COMPLETED" />
</batch:step>
请告知我在做什么错误。我没有看到处理器urlFileItemProcessor &作家validURLItemWriter正在执行。
更新 我遵循@dimzak给出的答案。但是,我仍然看不到步骤readURLFile正在执行从urlFileItemProcessor & validURLItemWriter记录器不会被打印到控制台。作业挂在后面的日志记录器
org.springframework.batch.repeat.support.TaskExecutorRepeatTemplate - Starting repeat context.
我对配置步骤范围的方式有疑问。
StepScope配置
<bean class="org.springframework.batch.core.scope.StepScope">
<property name="autoProxy" value="true"/>
<property name="proxyTargetClass" value="true"/>
</bean>
在春季论坛,我读过的multiResourceItemReader的委托财产不必一步作用域。当我从urlFileItemReader中删除scope =“step”时,我得到以下例外。
INFO : 26 Sep 2014 00:10:58,811 - org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor - Initializing ExecutorService 'taskExecutor'
INFO : 26 Sep 2014 00:10:59,066 - org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor - Shutting down ExecutorService 'taskExecutor'
Exception in thread "main" org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'urlFileItemReader' defined in class path resource [beansBatchService.xml]: Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanExpressionException: Expression parsing failed; nested exception is org.springframework.expression.spel.SpelEvaluationException: EL1008E:(pos 0): Field or property 'stepExecutionContext' cannot be found on object of type 'org.springframework.beans.factory.config.BeanExpressionContext'
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:547)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:304)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:300)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:195)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:703)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:760)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83)
at com.chw.hma.service.batch.jobrunner.MainJobRunner.run(MainJobRunner.java:29)
at com.chw.hma.service.batch.jobrunner.MainJobRunner.main(MainJobRunner.java:21)
Caused by: org.springframework.beans.factory.BeanExpressionException: Expression parsing failed; nested exception is org.springframework.expression.spel.SpelEvaluationException: EL1008E:(pos 0): Field or property 'stepExecutionContext' cannot be found on object of type 'org.springframework.beans.factory.config.BeanExpressionContext'
at org.springframework.context.expression.StandardBeanExpressionResolver.evaluate(StandardBeanExpressionResolver.java:146)
at org.springframework.beans.factory.support.AbstractBeanFactory.evaluateBeanDefinitionString(AbstractBeanFactory.java:1364)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.evaluate(BeanDefinitionValueResolver.java:214)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:186)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1456)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1197)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:537)
... 12 more
Caused by: org.springframework.expression.spel.SpelEvaluationException: EL1008E:(pos 0): Field or property 'stepExecutionContext' cannot be found on object of type 'org.springframework.beans.factory.config.BeanExpressionContext'
at org.springframework.expression.spel.ast.PropertyOrFieldReference.readProperty(PropertyOrFieldReference.java:217)
at org.springframework.expression.spel.ast.PropertyOrFieldReference.getValueInternal(PropertyOrFieldReference.java:85)
at org.springframework.expression.spel.ast.PropertyOrFieldReference.getValueInternal(PropertyOrFieldReference.java:78)
at org.springframework.expression.spel.ast.CompoundExpression.getValueRef(CompoundExpression.java:49)
at org.springframework.expression.spel.ast.CompoundExpression.getValueInternal(CompoundExpression.java:85)
at org.springframework.expression.spel.ast.SpelNodeImpl.getValue(SpelNodeImpl.java:102)
at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:94)
at org.springframework.context.expression.StandardBeanExpressionResolver.evaluate(StandardBeanExpressionResolver.java:143)
... 18 more
请指教。
更新2014年9月27日
发布任务执行等,由@dimzak
<bean id="taskExecutor"
class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
<property name="corePoolSize" value="10" />
<property name="maxPoolSize" value="10" />
</bean>
<bean id="jobRepository"
class="org.springframework.batch.core.repository.support.JobRepositoryFactoryBean">
<property name="transactionManager" ref="transactionManager" />
<property name="dataSource" ref="dataSource" />
<property name="databaseType" value="mySQL" />
</bean>
<bean id="jobLauncher"
class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
<property name="jobRepository" ref="jobRepository" />
<property name="taskExecutor" ref="taskExecutor" />
</bean>
MultiResourcePartitioner需要file:/scrapper/spliturl*
和分裂样spliturl1,spliturl2,在不同的步骤每个文件建议。
之后,在从属步骤中,您只需读取此步骤的特定文件。 所以你的配置应该没有MultiResourceItemReader
。
<batch:step id="readURLFile">
<batch:tasklet transaction-manager="transactionManager"
task-executor="taskExecutor" throttle-limit="10">
<batch:chunk reader="urlFileItemReader" processor="urlFileItemProcessor"
writer="validURLItemWriter" commit-interval="200" skip-limit="100">
<batch:skippable-exception-classes>
<batch:include class="java.net.MalformedURLException" />
<batch:include class="java.net.URISyntaxException" />
<batch:include class="java.net.UnknownHostException" />
</batch:skippable-exception-classes>
</batch:chunk>
<batch:listeners>
<batch:listener ref="malformedURLExceptionListener" />
<batch:listener ref="uriSyntaxExceptionListener" />
<batch:listener ref="unknownHostExceptionListener" />
</batch:listeners>
</batch:tasklet>
<batch:end on="COMPLETED" />
</batch:step>
我认为这是关于如何设置分区器和2个读卡器中的资源。但我不知道你想要实现什么.'urlFilesPath'就像'file:folder/*',你试图将每个文件分成一个从属步骤?...但是'MultiResourceItemReader'应该读取什么? – dimzak 2014-09-25 10:23:38
@dimzak - 感谢您的评论。我在问题中添加了更多信息。 – AbNig 2014-09-25 11:01:21
最后一个异常是因为您从'urlFileItemReader'中删除了'step = Scope',因此您无法读取'stepExecutionContext'。我真的不明白'multiResourceItemReader'在做什么...将其删除。你的主要读者应该是'urlFileItemReader'。请确保你[正确配置了你的jobrepo](http://docs.spring.io/spring-batch/trunk/reference/html/configureJob.html)进行分区。你的'StepScope'对我来说看起来很好 – dimzak 2014-09-26 08:05:33