K8s配置log-pilot收集服务日志

一.前言

1.1 简介

Log-Pilot具有自动发现机制: 容器配置好tag后就会被采集组件发现
CheckPoint句柄保持的机制: 对于日志文件的句柄做跟踪
自动日志数据打标: 通过对容器加tag，这个tag会被记录到日志中，取出日志的时候就可以通过这个tag来区分数据
有效应对动态配置: 容器扩缩容的时候能够自动处理
日志重复和丢失以及日志源标记等问题: 通过句柄保持实现

二.部署服务

2.1 配置log-pilot

apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: log-pilot
  name: log-pilot
  namespace: logs
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: log-pilot
  template:
    metadata:
      labels:
        app: log-pilot
    spec:
      containers:
      - env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: PILOT_LOG_PREFIX
          value: aliyun
        - name: LOGGING_OUTPUT
          value: elasticsearch
        - name: ELASTICSEARCH_HOSTS
          value: elasticsearch.logs:9200
        image: harbor.huhuhahei.cn/huhuhahei/log-pilot:filebeat-7.9.2
        imagePullPolicy: Always
        livenessProbe:
          exec:
            command:
            - /pilot/healthz
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        name: log-pilot
        resources:
          limits:
            memory: 500Mi
          requests:
            cpu: 200m
            memory: 200Mi
        securityContext:
          capabilities:
            add:
            - SYS_ADMIN
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/run/docker.sock
          name: sock
        - mountPath: /host
          name: root
          readOnly: true
        - mountPath: /var/lib/filebeat
          name: varlib
        - mountPath: /var/log/filebeat
          name: varlog
        - mountPath: /etc/localtime
          name: localtime
          readOnly: true
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
      volumes:
      - hostPath:
          path: /var/run/docker.sock
          type: ""
        name: sock
      - hostPath:
          path: /
          type: ""
        name: root
      - hostPath:
          path: /var/lib/filebeat
          type: DirectoryOrCreate
        name: varlib
      - hostPath:
          path: /var/log/filebeat
          type: DirectoryOrCreate
        name: varlog
      - hostPath:
          path: /etc/localtime
          type: ""
        name: localtime
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate

交付k8s

kubectl apply -f log-pilot_ds.yaml

查看状态

kubectl get po -n logs | grep log-pi
log-pilot-fs5nt                  1/1     Running   0          46m
log-pilot-nrkwn                  1/1     Running   0          46m
log-pilot-tpvmb                  1/1     Running   0          46m

三.服务内配置日志收集

3.1收集容器stdout

spec:
      containers:
      - env:
        - name: aliyun_logs_note-stdout
          value: stdout

通过容器中添加env实现自动拉取日志

es直接添加索引即可

3.2 收集服务日志

配置如下 需要添加日志路径 且挂载数据卷

- name: aliyun_logs_note-logs
     value: /wiz/logs/*.log
  - mountPath: /wiz/logs
     name: wiz-log
- emptyDir: {}
   name: wiz-log

es中查看