文档章节

Spring Batch_Multi-threaded Step

秋风醉了
 秋风醉了
发布于 2014/11/13 19:11
字数 505
阅读 1525
收藏 1
点赞 0
评论 0

Spring Batch_Multi-threaded Step

spring官方文档:http://docs.spring.io/spring-batch/trunk/reference/html/scalability.html

The simplest way to start parallel processing is to add a TaskExecutor to your Step configuration, e.g. as an attribute of the tasklet:

<step id="loading">
    <tasklet task-executor="taskExecutor">...</tasklet>
</step>

In this example the taskExecutor is a reference to another bean definition, implementing the TaskExecutor interface. TaskExecutor is a standard Spring interface, so consult(查阅,参照) the Spring User Guide for details of available implementations. The simplest multi-threaded TaskExecutor is a SimpleAsyncTaskExecutor.

 

The result of the above configuration will be that the Step executes by reading, processing and writing each chunk of items (each commit interval) in a separate thread of execution. Note that this means there is no fixed order for the items to be processed, and a chunk might contain items that are non-consecutive(不连续的) compared to the single-threaded case. In addition to any limits placed by the task executor (e.g. if it is backed by a thread pool), there is a throttle limit in the tasklet configuration which defaults to 4. You may need to increase this to ensure that a thread pool is fully utilised, e.g.

<step id="loading">
	<tasklet task-executor="taskExecutor" throttle-limit="20">...</tasklet>
</step>

Note also that there may be limits placed on concurrency by any pooled resources used in your step, such as a DataSource. Be sure to make the pool in those resources at least as large as the desired number of concurrent threads in the step.

 

There are some practical limitations of using multi-threaded Steps for some common Batch use cases. Many participants in a Step (e.g. readers and writers) are stateful(有状态的), and if the state is not segregated(分离,隔开的) by thread, then those components are not usable in a multi-threaded Step. In particular most of the off-the-shelf(现成的) readers and writers from Spring Batch are not designed for multi-threaded use. It is, however, possible to work with stateless or thread safe readers and writers, and there is a sample (parallelJob) in the Spring Batch Samples that show the use of a process indicator (see Section 6.12, “Preventing State Persistence”) to keep track of items that have been processed in a database input table.

 

Spring Batch provides some implementations of ItemWriter and ItemReader. Usually they say in the Javadocs if they are thread safe or not, or what you have to do to avoid problems in a concurrent environment. If there is no information in Javadocs, you can check the implementation to see if there is any state. If a reader is not thread safe, it may still be efficient(有效率的) to use it in your own synchronizing delegator. You can synchronize the call to read() and as long as the processing and writing is the most expensive part of the chunk your step may still complete much faster than in a single threaded configuration.

=================END=================

© 著作权归作者所有

共有 人打赏支持
秋风醉了
粉丝 228
博文 579
码字总数 407314
作品 0
东城
程序员
Spring Batch_Configuring a Step for Restart

Spring Batch_Configuring a Step for Restart spring官方文档:http://docs.spring.io/spring-batch/trunk/reference/html/configureStep.html#stepRestart 当一个普通的 job 处于complete ......

秋风醉了
2014/11/14
0
0
Spring Batch_Parallel Steps

Spring BatchParallel Steps使用并行的Step spring 官方文档:http://docs.spring.io/spring-batch/trunk/reference/html/scalability.html#scalabilityParallelSteps As long as the applic......

秋风醉了
2014/11/14
0
2
Spring Batch_JOB重启机制

Spring Batch_JOB重启机制 在这一篇文章 对于restart做了试验,http://my.oschina.net/xinxingegeya/blog/344817在这片文章里,我们只是当job成功时,重启了job,对于job失败后,重启job有...

秋风醉了
2014/11/19
0
0
Spring Batch_Step Flow

Spring Batch学习_Step Flow http://docs.spring.io/spring-batch/trunk/reference/html/configureStep.html With the ability to group steps together within an owning job comes the nee......

秋风醉了
2014/10/31
0
0
Spring Batch_实现Tasklet

Spring Batch_实现Tasklet Chunk-oriented processing is not the only way to process in a Step. What if a Step must consist as a simple stored procedure call? You could implement ......

秋风醉了
2014/12/01
0
0
Spring Batch_JobParameters

Spring BatchJobParameters spring batch的JobParameters是设置job运行的参数,同时也具有标志job的作用,就是判断两个job是不是同一个job( "how is one JobInstance distinguished from a...

秋风醉了
2014/11/12
0
0
Spring Batch学习笔记——steps之间共享数据

名词说明: 上下文: 执行: 执行上下文: 案例: 警告:一旦steps共享数据,这些数据就会把这些steps连接起来。努力使steps独立。如果你实在是不能独立他们,才使用下面的技术。你应该把数据...

xhujinjun
2015/04/02
0
0
Spring Batch_ItemReaders and ItemWriters

Spring Batch_ItemReaders and ItemWriters All batch processing can be described in its most simple form as reading in large amounts of data, performing some type of calculation o......

秋风醉了
2014/11/03
0
0
Spring Batch_Intercepting Job Execution

Spring BatchIntercepting Job Execution配置JobExecutionListener 关于spring batch skip 的配置请看:http://my.oschina.net/xinxingegeya/blog/346244 在job的运行期间,我们可以捕捉job......

秋风醉了
2014/11/19
0
0
Spring Batch_Intercepting Step Execution

Spring BatchIntercepting Step Execution配置SkipListener 关于配置skip:http://my.oschina.net/xinxingegeya/blog/346244 先看一下StepListener.java 接口的继承关系图: StepExecutionL......

秋风醉了
2014/11/19
0
3

没有更多内容

加载失败,请刷新页面

加载更多

下一页

java集合元素的默认大小

当底层实现涉及到扩容时,容器或重新分配一段更大的连续内存(如果是离散分配则不需要重新分配,离散分配都是插入新元素时动态分配内存),要将容器原来的数据全部复制到新的内存上,这无疑使...

竹叶青出于蓝
8分钟前
1
0
Java快速开发平台,JEECG 3.7.7闪电版本发布,增加多套主流UI代码生成器模板

JEECG 3.7.7 闪电版本发布,提供5套主流UI代码生成器模板 导读 ⊙平台性能优化,速度闪电般提升 ⊙提供5套新的主流UI代码生成器模板(Bootstrap表单+BootstrapTable列表\ ElementUI列表表单)...

Jeecg
12分钟前
0
0
export 和 module.export 的区别

在浏览器端 js 里面,为了解决各模块变量冲突等问题,往往借助于 js 的闭包把左右模块相关的代码都包装在一个匿名函数里。而 Nodejs 编写模块相当的自由,开发者只需要关注 require,exports,...

孟飞阳
14分钟前
0
0
技术教育的兴起

技术教育的兴起 作者: 阮一峰 1、 有一年,我在台湾环岛旅行。 花莲的海边,我遇到一对台湾青年夫妻,带着女儿在海滩上玩。我们聊了起来。 当时,我还在高校当老师。他们问我,是否觉得台湾...

吕伯文
14分钟前
0
0
Linux服务器下的HTTP抓包分析

说到抓包分析,最简单的办法莫过于在客户端直接安装一个Wireshark或者Fiddler了,但是有时候由于客户端开发人员(可能是第三方)知识欠缺或者其它一些原因,无法顺利的在客户端进行抓包分析,...

mylxsw
19分钟前
0
0
mybatis3-javaapi

sqlSessionFactoryBuilder->sqlSessionFactory->sqlSession<-rowbound<-resultHandler myBatis uses a Java enumeration wrapper for transaction isolation levels, called TransactionIsol......

writeademo
22分钟前
0
0
Java NIO:浅析I/O模型

也许很多朋友在学习NIO的时候都会感觉有点吃力,对里面的很多概念都感觉不是那么明朗。在进入Java NIO编程之前,我们今天先来讨论一些比较基础的知识:I/O模型。下面本文先从同步和异步的概念...

yzbty23
22分钟前
0
0
了解iOS消息推送一文就够:史上最全iOS Push技术详解

本文作者:陈裕发, 腾讯系统测试工程师,由腾讯WeTest整理发表。 1、引言 开发iOS系统中的Push推送,通常有以下3种情况: 1)在线Push:比如QQ、微信等IM界面处于前台时,聊天消息和指令都会...

JackJiang-
24分钟前
0
0
Mysql汉子转拼音

update t_app_city SET CITY_NAME_BEGIN = ELT(INTERVAL(CONV(HEX(LEFT(CONVERT(CITY_NAME USING gbk),1)),16,10), 0xB0A1,0xB0C5,0xB2C1,0xB4EE,0xB6EA,0xB7A2,0xB8C1,0xB9FE,0xBBF7, 0xBFA......

尘叙缘
26分钟前
0
0
大数据构建智慧城市“新引擎”,加速推进新旧动能转换

——“大数据与智慧城市”技术交流分享会——济南站召开 7月13日,“大数据携手智慧城市,助力山东新旧动能转换”技术交流分享会——济南站在山东信息通信技术研究院会议室成功举办,此次会议...

左手的倒影
28分钟前
2
0

没有更多内容

加载失败,请刷新页面

加载更多

下一页

返回顶部
顶部