Presto Cassandra Slow performance slow
问题描述:
我正在使用presto来查询Cassandra记录,它需要大约8分钟来响应结果。需要改善响应时间。Presto Cassandra Slow performance slow
的Presto配置如下:
coordinator=true
node-scheduler.include-coordinator=false
http-server.http.port=8080
query.max-memory=5GB
query.max-memory-per-node=3GB
discovery-server.enabled=true
discovery.uri=http://URL:8080
task.max-worker-threads=10
task.concurrency=32
Worker : 4
coordinator=false
http-server.http.port=8080
query.max-memory=5GB
query.max-memory-per-node=2GB
discovery.uri=http://URL:8080
task.max-worker-threads=16
task.concurrency=32
Cassandra : 4 NODE
片段2 成本:CPU1.98米,输入:17833912行(1.49GB),输出:13089502行(1.31GB)
ScanFilterProject [表=卡桑德拉:卡桑德拉:rasapp:raslog,originalConstraint =(( “bucketid”= CAST( '2017062113' 成本:96.12%,输入:23169736行(22.10MB),输出:17833912行(1.49GB),过滤:23.03%
如何提高响应时间仍然使用分区键哈哈约2300万条记录?采取
CREATE TABLE TEST.TEST_LOG (
bucketId varchar,
id timeuuid,
transaction_id varchar,
ras_transaction_id varchar,
msg_seq_id int,
host_name varchar,
matip_channel_id varchar,
hth_id varchar,
mq_id varchar,
log_point varchar,
entry_time timestamp,
exit_time timestamp,
source_carrier varchar,
destination_carrier varchar,
source_dcs varchar,
destination_dcs varchar,
message_type varchar,
message_direction int,
error_code_business varchar,
exception_code varchar,
exception_description varchar,
scenario varchar,
created_date timestamp,
huborcar varchar,
noof_fanout varchar,
flight_date timestamp,
route_origin varchar,
route_destination varchar,
class_service varchar,
no_of_seats varchar,
ras_host varchar,
cp_host varchar,
PRIMARY KEY(bucketid, created_date, msg_seq_id,message_direction,scenario,source_dcs,exception_code,log_point,transaction_id,id)
) WITH default_time_to_live = 2851200 and CLUSTERING ORDER BY (created_date ASC, msg_seq_id ASC,message_direction ASC,scenario ASC,source_dcs ASC,exception_code ASC,log_point ASC,transaction_id ASC,id ASC);
查询
select
transaction_id,
message_direction,
message_type,
max(exception_code) as exception_code,
min(entry_time) as min_entry,
max(entry_time) as max_entry,
min(exit_time) as min_exit,
max(exit_time) as max_exit
from TEST.TEST_LOG
where bucketid='2017062113'
and (
((msg_seq_id<=2 and message_type='PAOREQ' ) or
(msg_seq_id>2 and message_type='PAORES' )))
group by transaction_id,
message_direction,
message_type
时间:8分钟
感谢,
答
两样东西:0.180版本的Presto将包括不平等的下推谓词聚类键,这将帮助你的查询。另外,您的模式对于您正在运行的查询不起作用。在Cassandra中,最好a)查询特定的分区(你这么做),也可以按照你使用它们的顺序在集群键上拥有谓词(因为这是Cassandra使用的排序顺序)。如果您有主键(bucketid,message_type,msg_seq_id,...),您可能会看到更好的性能。
此外,Presto不会将聚合压入Cassandra(或任何连接器),因此如果您要聚合的数据量很大,并且您不需要Presto用于联合查询,则可能会在Cassandra中执行查询的速度更快。
仅使用Cassandra时查询需要多长时间?什么是您正在运行的查询和表模式(包括哪些列是分区/集群密钥)? –
请检查,更新后 – Augustin
只需卡桑德拉需要多少时间? –