今天私有云部署了一批虚拟机Server，由于是用虚拟机模板批量部署的，结果运行一段时间后，Kibana Discover显示的数据后面中断了，后面检查集群和Elasticsearch个节点，都正常，磁盘空间使用率很好，后面逐级检查Logstash，发现日志有以下大量报错信息。

Terminal window
[2025-07-11T15:18:08,589][INFO ][logstash.outputs.elasticsearch][koevn_logs][7c39dc6190bbe761f71b0f0b463552b818d13eb5b5b0b9e16dfa801f1463654c] Retrying individual bulk actions that failed or were rejected by the previous bulk request {:count=>78}
[2025-07-11T15:18:08,635][INFO ][logstash.outputs.elasticsearch][koevn_logs][7c39dc6190bbe761f71b0f0b463552b818d13eb5b5b0b9e16dfa801f1463654c] Retrying failed action {:status=>403, :action=>["index", {:_id=>nil, :_index=>"koevn_logs-2025.02.18", :routing=>nil}, {"@version"=>"1", "kafka_topic"=>"koevn_logs", "auditd.process.id"=>"11", "event.type"=>"koevn", "@timestamp"=>2025-02-18T08:49:18.820Z, "auditd.event.id"=>"161", "fields"=>{"env"=>"test", "host_ip"=>"10.68.11.136"}, "tags"=>["koevn_logs"], "auditd.event.time"=>"1739868558.820", "temp_timestamp"=>"2025-02-18T08:49:18.820Z", "message"=>"type=koevn msg=audit(1739868558.820:161): prog-id=11 op=UNLOAD", "log"=>{"offset"=>180412, "file"=>{"path"=>"/var/log/audit/audit.log"}}}], :error=>{"type"=>"cluster_block_exception", "reason"=>"index [koevn_logs-2025.02.18] blocked by: [FORBIDDEN/8/index write (api)];"}}

由于对创建的索引配置了ILM生命周期管理，只有30天内的数据可读写，大于30天的数据仅限只读，所以看日志内容，是有大量大于30天数据在尝试写入只读索引，造成频繁403导致Elasticsearch拒绝所有写入请求，那这种情况就看自己需求了，是需要解除只读限制然后写入，还是直接丢弃。

解除有两者方式 第一种按自己需求解除指定索引

Terminal window
curl -XPUT http://<es-host>:9200/koevn_logs-2025.02.18/_settings -H 'Content-Type: application/json' -d '
{
  "index.blocks.write": false
}'

第二种就是解除全部只读状态索引

Terminal window
curl -XPUT http://<es-host>:9200/_all/_settings -H 'Content-Type: application/json' -d '
{
  "index.blocks.read_only_allow_delete": null,
  "index.blocks.write": false
}'

还有就是在logstash里直接过滤掉大于30天的日志，避免写入Elasticsearch索引导致拒绝所有连接，如以下配置。

Terminal window
filter {
    #   ------- 其他配置省略 -------


    ruby {
      code => "
        require 'time'
        now = Time.now
        cutoff = now - 29 * 24 * 60 * 60
        if event.get('@timestamp')
          ts = Time.parse(event.get('@timestamp').to_s)
          if ts < cutoff
            event.tag('drop_old_event')
          end
        end
      "
    }


    if "drop_old_event" in [tags] {
      drop { }
    }


    #   ------- 其他配置省略 -------
}

然后重载logstash服务，Kibana Discover服务正常显示日志数据。

Logstash日志提示cluster_block_exception错误
https://huoshen.pages.dev/cn/p/b7435430/
作者
Koevn
发布于
许可协议
CC BY-NC-SA 4.0