初试VictoriaLogs集群采集Kubernetes Pod日志

文章目录

Toggle

- VictoriaLogs 介绍
- Helm部署VictoriaLogs Cluster
- Grafana 安装插件
- fluent-bit 采集Kubernetes日志
- Grafana 配置 victoriametrics-logs-datasource 数据源

VictoriaLogs 介绍

VictoriaLogs 是 VictoriaMetrics 团队推出的开源日志解决方案，旨在替代笨重的 Elasticsearch (ELK) 和架构复杂的 Loki

**组件介绍**

- vlinsert 接收来自 Fluent-bit、Vector 或 Logstash 的日志数据
- vlstorage 核心存储，将日志压缩并持久化到磁盘，响应查询请求。
- vlselect 查询网关，处理Grafana或CLI的搜索请求。
- vmauth 认证网关 - 可选，提供统一入口、负载均衡和身份验证。

Helm部署VictoriaLogs Cluster

这里使用从 Kubernetes 中运行的所有容器收集日志，并将数据发送到已安装的集群版 VictoriaLogs。

vlstorage部署了两个实例，每个实例的保留期为180天，PVC 为200Gi。

首先添加helm仓库

helm repo add vm https://victoriametrics.github.io/helm-charts/
helm repo update

使用环境变量的方式，helm VictoriaLogs部署节点

export RETENTION=180d
export PVC_SIZE=200Gi
export VLSTORAGE_REPLICAS=2
export NAMESPACE=tools

# Install victoria-logs-cluster chart
helm install vlc vm/victoria-logs-cluster --namespace $NAMESPACE --wait \
    --set "vlstorage.retentionPeriod=$RETENTION" --set "vlstorage.persistentVolume.size=$PVC_SIZE" \
    --set vmauth.enabled=true \
    --set vlstorage.replicaCount=$VLSTORAGE_REPLICAS

# Install victoria-logs-collector chart
helm install collector vm/victoria-logs-collector --namespace $NAMESPACE \
    --set "remoteWrite[0].url=http://vlc-victoria-logs-cluster-vmauth:8427"

离线本地下载

export RETENTION=180d
export PVC_SIZE=200Gi
export VLSTORAGE_REPLICAS=2
export NAMESPACE=tools

# 基于本地tgz包安装
helm install vlc ./victoria-logs-cluster-0.0.20.tgz \
  --namespace $NAMESPACE --wait \
  --set "vlstorage.retentionPeriod=$RETENTION" \
  --set "vlstorage.persistentVolume.size=$PVC_SIZE" \
  --set vmauth.enabled=true \
  --set vlstorage.replicaCount=$VLSTORAGE_REPLICAS

# 或基于解压后的目录安装（适合本地修改配置后）
helm install vlc ./victoria-logs-cluster \
  --namespace $NAMESPACE --wait \
  --set "vlstorage.retentionPeriod=$RETENTION" \
  --set "vlstorage.persistentVolume.size=$PVC_SIZE" \
  --set vmauth.enabled=true \
  --set vlstorage.replicaCount=$VLSTORAGE_REPLICAS

> 如需镜像加速，可使用abcdocker提供镜像加速文档 https://i4t.com/dockerproxy

修改服务暴露，如果没有ingress，可以和我一样修改SVC类型

#修改写的映射
root@k8s-master-01:~# kubectl edit svc  -n tools vlc-victoria-logs-cluster-vlinsert
  type: NodePort
#修改读的映射
root@k8s-master-01:~# kubectl edit svc -n tools vlc-victoria-logs-cluster-vlselect
  type: NodePort

最终效果

root@k8s-master-01:~# kubectl get svc -n tools|grep vlc
vlc-victoria-logs-cluster-vlinsert    NodePort    10.96.3.162   <none>        9481:31072/TCP                                                                                                    29d
vlc-victoria-logs-cluster-vlselect    NodePort    10.96.1.97    <none>        9471:31708/TCP                                                                                                    29d
vlc-victoria-logs-cluster-vlstorage   ClusterIP   None          <none>        9491/TCP                                                                                                          29d
vlc-victoria-logs-cluster-vmauth      ClusterIP   10.96.0.80    <none>        8427/TCP                                                                                                          29d

Grafana 安装插件

首先在Grafana中需要安装victoriametrics-logs-datasource插件

grafana cli plugins install victoriametrics-logs-datasource

fluent-bit 采集Kubernetes日志

创建ConfigMap

- tools 部署命名空间
- 采集Pod日志/var/log/containers/*.log路径
- 写入日志地址 vlc-victoria-logs-cluster-vlinsert
- 写入日志端口9481

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: tools
  labels:
    app.kubernetes.io/name: fluent-bit
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush        1
        Log_Level    info
        HTTP_Server  On
        HTTP_Listen  0.0.0.0
        HTTP_Port    2020
        Health_Check On

    [INPUT]
        Name            tail
        Tag             kubernetes.*
        Path            /var/log/containers/*.log
        # --- 关键修改在这里 ---
        Exclude_Path    /var/log/containers/fluent-bit-*.log
        Parser          cri
        DB              /var/log/flb_kube.db
        Mem_Buf_Limit   5MB
        Skip_Long_Lines On
        Refresh_Interval 10

    [FILTER]
        Name            kubernetes
        Match           kubernetes.*
        Kube_URL        https://kubernetes.default.svc:443
        Kube_CA_File    /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix kubernetes.var.log.containers.
        Merge_Log       On
        Merge_Log_Key   log_processed
        K8S-Logging.Parser On
        K8S-Logging.Exclude Off

    [OUTPUT]
        Name            http
        Match           *
        Host            vlc-victoria-logs-cluster-vlinsert
        Port            9481
        URI             /insert/jsonline?_msg_field=log&_stream_fields=kubernetes.namespace_name,kubernetes.pod_name,kubernetes.container_name,stream&_time_field=time
        Format          json_lines
        Json_date_key   time
        Json_date_format iso8601
        Header          X-Scope-OrgID 1
        tls             Off
        tls.verify      Off

  parsers.conf: |
    [PARSER]
        Name   cri
        Format regex
        Regex  ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
        Time_Key    time

创建rbac.yaml文件

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
  namespace: tools # 建议使用独立的命名空间，如 logging
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit-read
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods
  - pods/logs
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit-read
subjects:
- kind: ServiceAccount
  name: fluent-bit
  namespace: tools

创建DaemonSet.yaml Pod

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: tools
  labels:
    app.kubernetes.io/name: fluent-bit
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: fluent-bit
  template:
    metadata:
      labels:
        app.kubernetes.io/name: fluent-bit
    spec:
      containers:
      - name: fluent-bit
        image: harbor.frps.cn/tools/fluent-bit:3.0.1
        imagePullPolicy: Always
        ports:
        - containerPort: 2020
        env:
        - name: FLB_PROCESSOR
          value: "OFF"
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
        - name: etcmachineid
          mountPath: /etc/machine-id
          readOnly: true
        resources:
          limits:
            memory: 100Mi
          requests:
            cpu: 10m
            memory: 20Mi
        securityContext:
          runAsUser: 0
          privileged: true
      serviceAccountName: fluent-bit
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config
      - name: etcmachineid
        hostPath:
          path: /etc/machine-id
          type: File
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule

部署完毕后，可以看到服务状态

root@k8s-master-01:~# kubectl get pod -n tools |grep fl
fluent-bit-d9ls4                                      1/1     Running   9 (4d21h ago)     16d
fluent-bit-dmr86                                      1/1     Running   3 (4d20h ago)     9d
fluent-bit-jhs7d                                      1/1     Running   7 (4d21h ago)     14d
fluent-bit-krkh4                                      1/1     Running   9 (4d21h ago)     14d
fluent-bit-t4znj                                      1/1     Running   6 (4d21h ago)     13d
fluent-bit-x9j5c                                      1/1     Running   7 (4d21h ago)     13d

现在我们访问页面就可以看到效果

> node_ip:port

图形化访问的是vlselect

vlc-victoria-logs-cluster-vlselect    NodePort    10.96.1.97    <none>        9471:31708/TCP

Grafana 配置 victoriametrics-logs-datasource 数据源

> Home -->Connections --> Add new connection --> VictoriaLogs -->Add new data source



- http://vlc-victoria-logs-cluster-vlselect:9471

vlc-victoria-logs-cluster-vlselect    NodePort    10.96.1.97    <none>        9471:31708/TCP

保存测试连接



默认Grafana没有提供好的模板选择，需要我们自行导入，这里我让Gemini Pro写了个一个简单的查看日志模板，可以直接导入



导入下方JSON

> 下载地址https://d.frps.cn/file/tools/grafana/VictoriaLogs-Dashboard.json
>
> 打开JSON文件复制上即可



数据源这块别选择❎



最后模板就可以看到日志

- Namespace: 默认 .* (所有)。想看 tools 就直接输 tools。
- Pod: 默认 .* (所有)。想看 blackbox 就直接输 blackbox。
- Search: 默认 * (所有)。想搜错误就输 error 或 exception。
- 时间：默认设为了 Last 15 minutes，这是看实时日志最舒服的范围。