Logstash详解之——output模块
原始学习资料,请参考官方文档:https://www.elastic.co/guide/en/logstash/current/output-plugins.html
Logstash的output模块,相比于input模块来说是一个输出模块,output模块集成了大量的输出插件,可以输出到指定文件,也可输出到指定的网络端口,当然也可以输出数据到ES.
在这里我只介绍如何输出到ES,至于如何输出到端口和指定文件,有很多的文档资料可查找.
elasticsearch{
hosts=>["172.132.12.3:9200"]
action=>"index"
index=>"indextemplate-logstash"
#document_type=>"%{@type}"
document_id=>"ignore"
template=>"/opt/logstash-conf/es-template.json"
template_name=>"es-template.json"
template_overwrite=>true
}
action=>”index” #es要执行的动作 index, delete, create, update
l index:将logstash.时间索引到一个文档
l delete:根据id删除一个document(这个动作需要一个id)
l create:建立一个索引document,如果id存在 动作失败.
l update:根据id更新一个document,有一种特殊情况可以upsert--如果document不是已经存在的情况更新document 。参见upsert选项。
l A sprintf style string to change the action based on the content of the event. The value %{[foo]} would use the foo field for the action
document_id=>” ” 为索引提供document id ,对重写elasticsearch中相同id词目很有用
document_type=>” ”事件要被写入的document type,一般要将相似事件写入同一type,可用%{}引用事件type,默认type=log
index=>”logstash-%{+YYYY,MM.dd}” 事件要被写进的索引,可是动态的用%{foo}语句
hosts=>[“127.0.0.0”] ["127.0.0.1:9200","127.0.0.2:9200"] "https://127.0.0.1:9200"
manage_template=>true 一个默认的es mapping 模板将启用(除非设置为false 用自己的template)
template=>”” 有效的filepath 设置自己的template文件路径,不设置就用已有的
template_name=>”logstash” 在es内部模板的名字
这里需要十分注意的一个问题是,document_id尽量保证值得唯一,这样会解决你面即将面临的ES数据重复问题,切记切记!
输出到文件
input{
beats{
codec => plain{charset => "UTF-8"}
port => "5044"
}
}
filter{
mutate{
remove_field => "@version"
remove_field => "offset"
remove_field => "input_type"
remove_field => "beat"
remove_field => "tags"
}
ruby{
code => "event.timestamp.time.localtime"
}
}
output{
file{
path => "/home/app/logbak/%{+YYYY.MM.dd}-file.txt"
}
}
开始设置的path => "/tmp/%{+YYYY.MM.dd}-%{host}-file.txt",由于没有%{host} 这个字段,导致出现无法创建文件错误,删除这个之后恢复正常。
[2019-04-13T01:17:44,206][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<Errno::EACCES: Permission denied - /home/app/logbak/2019.04.12-{"hostname":"WIN-NN7AS1GRN2F","os":{"build":"9200.0","kernel":"6.2.9200.16384 (win8_rtm.120725-1247)","name":"Windows Server 2012 Standard","family":"windows","version":"6.2","platform":"windows"},"name":"WIN-NN7AS1GRN2F","id":"22067c43-882c-43e7-8a37-7c929f621593","architecture":"x86_64"}-file.txt>, :backtrace=>["org/jruby/RubyFile.java:370:in `initialize'", "org/jruby/RubyIO.java:871:in `new'", "/home/app/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-file-4.0.2/lib/logstash/outputs/file.rb:280:in `open'", "/home/app/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-file-4.0.2/lib/logstash/outputs/file.rb:132:in `multi_receive_encoded'", "org/jruby/RubyHash.java:1342:in `each'", "/home/app/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-file-4.0.2/lib/logstash/outputs/file.rb:131:in `multi_receive_encoded'", "org/jruby/ext/thread/Mutex.java:149:in `synchronize'", "/home/app/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-file-4.0.2/lib/logstash/outputs/file.rb:130:in `multi_receive_encoded'", "/home/app/logstash/logstash-core/lib/logstash/outputs/base.rb:90:in `multi_receive'", "/home/app/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:13:in `multi_receive'", "/home/app/logstash/logstash-core/lib/logstash/output_delegator.rb:47:in `multi_receive'", "/home/app/logstash/logstash-core/lib/logstash/pipeline.rb:420:in `output_batch'", "org/jruby/RubyHash.java:1342:in `each'", "/home/app/logstash/logstash-core/lib/logstash/pipeline.rb:419:in `output_batch'", "/home/app/logstash/logstash-core/lib/logstash/pipeline.rb:365:in `worker_loop'", "/home/app/logstash/logstash-core/lib/logstash/pipeline.rb:330:in `start_workers'"]}
[[email protected] logstash]$ cd ..
修改之后恢复正常