Logstash详解之——output模块

原始学习资料，请参考官方文档：https://www.elastic.co/guide/en/logstash/current/output-plugins.html

Logstash的output模块,相比于input模块来说是一个输出模块,output模块集成了大量的输出插件,可以输出到指定文件,也可输出到指定的网络端口,当然也可以输出数据到ES.
在这里我只介绍如何输出到ES,至于如何输出到端口和指定文件,有很多的文档资料可查找.

  elasticsearch{  
    hosts=>["172.132.12.3:9200"]  
    action=>"index"  
    index=>"indextemplate-logstash"  
    #document_type=>"%{@type}"  
    document_id=>"ignore"  
      
    template=>"/opt/logstash-conf/es-template.json"  
    template_name=>"es-template.json"  
    template_overwrite=>true       
    }

action=>”index” #es要执行的动作 index, delete, create, update
l index:将logstash.时间索引到一个文档
l delete:根据id删除一个document(这个动作需要一个id)
l create:建立一个索引document，如果id存在 动作失败.
l update:根据id更新一个document，有一种特殊情况可以upsert--如果document不是已经存在的情况更新document 。参见upsert选项。
l A sprintf style string to change the action based on the content of the event. The value %{[foo]} would use the foo field for the action
document_id=>” ” 为索引提供document id ，对重写elasticsearch中相同id词目很有用
document_type=>” ”事件要被写入的document type，一般要将相似事件写入同一type，可用%{}引用事件type，默认type=log
index=>”logstash-%{+YYYY,MM.dd}” 事件要被写进的索引，可是动态的用%{foo}语句
hosts=>[“127.0.0.0”] ["127.0.0.1:9200","127.0.0.2:9200"] "https://127.0.0.1:9200" 
manage_template=>true 一个默认的es mapping 模板将启用（除非设置为false 用自己的template）
template=>”” 有效的filepath 设置自己的template文件路径，不设置就用已有的
template_name=>”logstash” 在es内部模板的名字
这里需要十分注意的一个问题是,document_id尽量保证值得唯一,这样会解决你面即将面临的ES数据重复问题,切记切记!

输出到文件

input{
    beats{
        codec => plain{charset => "UTF-8"}
        port => "5044"
    }
}
filter{
    mutate{
        remove_field => "@version"
        remove_field => "offset"
        remove_field => "input_type"
        remove_field => "beat"
        remove_field => "tags"
    }
    ruby{
        code => "event.timestamp.time.localtime"
    }
}
output{
       file{

              path => "/home/app/logbak/%{+YYYY.MM.dd}-file.txt"

       }
}

开始设置的path => "/tmp/%{+YYYY.MM.dd}-%{host}-file.txt"，由于没有%{host} 这个字段，导致出现无法创建文件错误，删除这个之后恢复正常。

[2019-04-13T01:17:44,206][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<Errno::EACCES: Permission denied - /home/app/logbak/2019.04.12-{"hostname":"WIN-NN7AS1GRN2F","os":{"build":"9200.0","kernel":"6.2.9200.16384 (win8_rtm.120725-1247)","name":"Windows Server 2012 Standard","family":"windows","version":"6.2","platform":"windows"},"name":"WIN-NN7AS1GRN2F","id":"22067c43-882c-43e7-8a37-7c929f621593","architecture":"x86_64"}-file.txt>, :backtrace=>["org/jruby/RubyFile.java:370:in `initialize'", "org/jruby/RubyIO.java:871:in `new'", "/home/app/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-file-4.0.2/lib/logstash/outputs/file.rb:280:in `open'", "/home/app/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-file-4.0.2/lib/logstash/outputs/file.rb:132:in `multi_receive_encoded'", "org/jruby/RubyHash.java:1342:in `each'", "/home/app/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-file-4.0.2/lib/logstash/outputs/file.rb:131:in `multi_receive_encoded'", "org/jruby/ext/thread/Mutex.java:149:in `synchronize'", "/home/app/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-file-4.0.2/lib/logstash/outputs/file.rb:130:in `multi_receive_encoded'", "/home/app/logstash/logstash-core/lib/logstash/outputs/base.rb:90:in `multi_receive'", "/home/app/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:13:in `multi_receive'", "/home/app/logstash/logstash-core/lib/logstash/output_delegator.rb:47:in `multi_receive'", "/home/app/logstash/logstash-core/lib/logstash/pipeline.rb:420:in `output_batch'", "org/jruby/RubyHash.java:1342:in `each'", "/home/app/logstash/logstash-core/lib/logstash/pipeline.rb:419:in `output_batch'", "/home/app/logstash/logstash-core/lib/logstash/pipeline.rb:365:in `worker_loop'", "/home/app/logstash/logstash-core/lib/logstash/pipeline.rb:330:in `start_workers'"]}
[[email protected] logstash]$ cd ..

修改之后恢复正常