使用Groovy中的“连接池关闭”,AWS S3 Java SDK上载失败
使用Groovy(Groovy版本:2.4.11 JVM:1.8),我正在使用Amazon Java SDK(最新版本是1.11.147) 0_112供应商:Oracle Corporation操作系统:Mac OS X)将文件上传到S3。使用Amazon的文档中的说明获取Transfer Manager,我能够将内容从一个存储桶复制到另一个存储桶。但是,上传总是失败。我尝试了以下3种方法。我有葡萄抢,因为我在这个堆栈溢出后Apache PoolingHttpClientConnectionManager throwing illegal state exception使用Groovy中的“连接池关闭”,AWS S3 Java SDK上载失败
@Grapes([
@Grab(group='com.amazonaws', module='aws-java-sdk', version='1.11.147'),
// https://mvnrepository.com/artifact/org.apache.commons/commons-compress
@Grab(group='org.apache.commons', module='commons-compress', version='1.13'),
// https://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient
@Grab(group='org.apache.httpcomponents', module='httpclient', version='4.5.3'),
// https://mvnrepository.com/artifact/org.apache.httpcomponents/httpcore
@Grab(group='org.apache.httpcomponents', module='httpcore', version='4.4.6')
])
// creds is String saying which ~/.aws/credentials profile to use
def credentials = new ProfileCredentialsProvider(creds)
def s3client = AmazonS3ClientBuilder.standard().
withCredentials(credentials).
withRegion(region).
build()
def tx = TransferManagerBuilder.standard().
withS3Client(s3client).
build()
def config_path = "/path/to/my/root"
def dir = "key_path"
def f = new File("${config_path}/${dir}/")
// Method 1, upload whole directory in one call
def mfu = tx.uploadDirectory(data_bucket, dir, f, true)
mfu.waitForCompletion() // <-- throws exception, w/o this line, just doesn't upload, but script continues
// Method 2, upload each file separately
def ld
ld = { File file ->
file.listFiles().each { g ->
if (g.isDirectory()) {
ld.call(g)
} else {
def key = g.toString().substring(config_path_length)
def fu = tx.upload(data_bucket, key, g)
def fu = tx.upload(data_bucket, key, g)
fu.waitForCompletion() // <-- throws exception, w/o this line, just doesn't upload, but script continues
}
}
}
ld.call(f)
// Finally Method 3, avoiding TransferManager altogether, and call putObject directly on each file
def ld
ld = { File file ->
file.listFiles().each { g ->
if (g.isDirectory()) {
ld.call(g)
} else {
def key = g.toString().substring(config_path_length)
def fu = tx.upload(data_bucket, key, g)
s3client.putObject(new PutObjectRequest(data_bucket, key, g)) // <-- throws exception
}
}
}
ld.call(f)
读但是,不管我尝试哪一种方法,我总是得到以下堆栈跟踪的HttpClient和的HttpCore的新版本:
捕获:java.lang.IllegalStateException:连接池关闭 java.lang.IllegalStateException:连接池在 关闭org.apache.http.util.Asserts.check(Asserts.java:34)在 org.apache。 http.pool.AbstractConnPool.lease(AbstractConnPool.java:184) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:251) 在 com.amazonaws.http.conn.ClientConnectionManagerFactory $ Handler.invoke(ClientConnectionManagerFactory.java:76) 在com.amazonaws。 http.conn。$ Proxy11.requestConnection(来源不明) 在 org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:175) 在 org.apache.http.impl.execchain.ProtocolExec。执行(ProtocolExec.java:184) 在 org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) 在 org.apache.http.impl.client.CloseableHttpClient.execut E(CloseableHttpClient.java:82) 在 org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) 在 com.amazonaws.http.apache.client.impl.SdkHttpClient.execute( SdkHttpClient.java:72) 在 com.amazonaws.http.AmazonHttpClient $ RequestExecutor.executeOneRequest(AmazonHttpClient.java:1190) 在 com.amazonaws.http.AmazonHttpClient $ RequestExecutor.executeHelper(AmazonHttpClient.java:1030) 在 com.amazonaws.http.AmazonHttpClient $ RequestExecutor.doExecute(AmazonHttpClient.java:742) 在 com.amazonaws.http.AmazonHttpClient $ RequestExecutor.executeWithTimer(AmazonHttpClient.java:716) 在 com.amazonaws.http.AmazonHttpClient $ RequestExecutor.execute(AmazonHttpClient.java:699) 在 com.amazonaws.http.AmazonHttpClient $ RequestExecutor.access $ 500(AmazonHttpClient.java:667) 在 com.amazonaws。 http.AmazonHttpClient $ RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) 在 com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) 在 com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client。 java:4221) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4168) at com.amazonaws.services.s3.AmazonS3Client.putO bject(AmazonS3Client.java:1718) at com.amazonaws.services.s3.AmazonS3 $ putObject.call(Unknown Source) ...
我目前无法确认groovy是否使用更新的httpcomponent库,或者如果这是问题所在。安装在1.8 JDK中的有httpclient_4.2.6和httpclient_4.3.5。任何关于如何取消粘贴的建议将不胜感激。谢谢。 -Vincent
我想通了。 s3client在代码的其他地方连接了另一个TranferManager。当tx.shutdownNow()被调用时,它也关闭了这个TransferManager。
你是如何解决这个问题的?我也得到这个异常? –
@shanelee我停止传递TransferManager参考,而是在每个传递s3文件的函数中使用一个新参数。 –