Istio 中监视 API 调用报文 :: Mark Zhu's Blog

概述⌗

API调用报文捕获是进行故障排除的关键工具。

Tcpdump是在ISO第3〜6层工作的工具，许多现代API在第7层（HTTP）工作，许多捕获规则仅在第7层有效。例如。通过URL路径过滤对于tcpdump来说很困难：

tcpdump -i enp0s8 -s 0 -v -n -l | egrep -i "POST /|GET /|Host:"

一般方法：minikube 或 Node Local（物理节点）上用 tcpdump⌗

以下是tcpdump的示例。

[root@host-os ~]# ps -ef | grep -i myServicePS
100       398212  398058  0 Apr07 ?        00:26:08 java ...myServicePS...

set CONTAINER_PID_AT_HOST=398212

[root@host-os ~]# nsenter -t $CONTAINER_PID_AT_HOST  -u -i -n -p -C tcpdump -A -vv
...

在许多情况下，很难在大量的 tcpdump 结果中找到您的API。

Sidecar(Envoy) Filter/Tap 方式⌗

在 Istio 中，Envoy是所有API流量的代理。它支持HTTP过滤器，我们可以在Envoy上创建一些过滤器来捕获负载报文。

有两种方法：

Tap Fitler
Lua Filter

安装⌗

kubectl apply -f istio-envoy-log-filter.yaml

#可选: 检查k8s资源: 
kubectl describe envoyfilters.networking.istio.io logpayload

可选：通过envoy管理端口15000检查envoy配置：

kubectl port-forward $your_pod_name 15000:15000
curl localhost:15000/config_dump | grep -i logPayload

下载: istio-envoy-log-filter.yaml。其中 workloadSelector 与 namespace 需要视你的应用而修改一下.

拦截⌗

方法 1: 用 Tap Fitler 拦截⌗

保证到 pod 的端口转发就绪:

kubectl port-forward $your_pod_name 15000:15000

发起 curl 长连接请求:

curl -XPOST -d 'config_id: test_config_id
tap_config:
  match_config:
    any_match: true
  output_config:
    sinks:
      - format: JSON_BODY_AS_STRING
        streaming_admin: {}
    max_buffered_rx_bytes: 2097152      
    max_buffered_tx_bytes: 2097152' 'http://localhost:15000/tap'

打开另一个终端，访问您的应用程序的API，例如：

curl http://your-service:your-service-port/your-api-path

“curl 长连接请求” 返回API调用报文，如下所示：

{
 "http_buffered_trace": {
  "request": {
   "headers": [
    {
     "key": ":path",
     "value": "/productpage"
    },
    {
     "key": ":method",
     "value": "GET"
    },
    {
     "key": ":scheme",
     "value": "http"
    },
    {
     "key": "user-agent",
     "value": "curl/7.64.1"
    },
    {
     "key": "accept",
     "value": "*/*"
    },
    {
     "key": "x-forwarded-for",
     "value": "10.244.1.1"
    }
   ],
   "trailers": []
  },
  "response": {
   "headers": [
    {
     "key": ":status",
     "value": "200"
    },
    {
     "key": "content-type",
     "value": "text/html; charset=utf-8"
    },
    {
     "key": "content-length",
     "value": "3889"
    },
    {
     "key": "server",
     "value": "istio-envoy"
    },
    {
     "key": "date",
     "value": "Sun, 12 Apr 2020 01:07:40 GMT"
    },
    {
     "key": "x-envoy-upstream-service-time",
     "value": "192"
    },
    {
     "key": "x-envoy-peer-metadata-id",
     "value": "sidecar~10.244.2.37~productpage-v1-5f7b7d4568-hmqzd.default~default.svc.cluster.local"
    }
   ],
   "body": {
    "truncated": false,
    "as_string": "\u003c!DOCTYPE html\u003e\n\u003chtml\u003e\n  \u003chead\u003e\n    \u003ctitle\u003eSimple Bookstore App\u003c/titleon('shown.bs.modal', function () {\n    $('#username').focus();\n  });\n\u003c/script\u003e\n\n  \u003c/body\u003e\n\u003c/html\u003e\n"
   },
   "trailers": []
  }
 }
}

方法 2: 通过 kubectl logs⌗

打开另一个终端，访问您的应用程序的API，例如：

curl http://your-service:your-service-port/your-api-path

查看 sidecar(Envoy) 日志:

kubectl logs -f your-pod -c istio-proxy

日志中已经有调用报文:

[Envoy (Epoch 0)] [2020-04-12 01:07:41.292][29][warning][lua] [external/envoy/source/extensions/filters/http/lua/lua_filter.cc:597] script log: <!DOCTYPE html>
<html>
  <head>
    <title>Simple Bookstore App</title>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">

工作原理⌗

先看看 kubectl apply -f istio-envoy-log-filter.yaml :

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: logpayload
  namespace: default
spec:
  workloadSelector:
    labels:
      app: productpage
  configPatches:
    # The first patch adds the lua filter to the listener/http connection manager
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
#        portNumber: 15001
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
            subFilter:
              name: "envoy.router"
    patch:
      operation: INSERT_BEFORE
      value: # filter specification
        # name: envoy.filters.http.tap
        #   config:
        #     common_config:
        #       static_config:
        #         match_config:
        #           any_match: true
        #         output_config:
        #           sinks:
        #             - file_per_tap:
        #                 path_prefix: taps/any

        name: "envoy.filters.http.tap"
        config:
          common_config:
            admin_config:
              config_id: test_config_id

  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
#        portNumber: 15001
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
            subFilter:
              name: "envoy.router"
    patch:
      operation: INSERT_BEFORE
      value: # filter specification
          name: envoy.lua
          typed_config:
            "@type": type.googleapis.com/envoy.config.filter.http.lua.v2.Lua
            inline_code: |
              function logPayload(handle)
                  handle:logWarn("[logPayload] started")

                  local index = 0
                  for chunk in handle:bodyChunks() do
                      handle:logWarn('[logPayload] showing bodyChunks')

                      local len = chunk:length()
                      if len < 1 then
                        break
                      end
                      local result = chunk:getBytes(index, len)
                      index = index + len

                      handle:logWarn(result)
                  end

                  handle:logWarn("[logPayload] finished")
              end            

              function envoy_on_request(handle)
                logPayload(handle)
              end
              function envoy_on_response(handle)
                logPayload(handle)
              end

首先, 这里用了 Envoy Filter:

HTTP Tap Filter
- https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/tap_filter
Lua Filter
- https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/lua_filter

再之上, 用了 Istio 的对象资源去管理这些 Envoy Filter. Istio 如何管理 Envoy Filter:

https://istio.io/docs/reference/config/networking/envoy-filter/

进一步制定和筛选⌗

根据 service/deployment/replica 去过滤⌗

更新 istio-envoy-log-filter.yaml :

spec:
  workloadSelector:
    labels:
      app: your-app-label

kubectl apply -f istio-envoy-log-filter.yaml

用 URL 去过滤⌗

通过 Tap 方法:⌗

curl -XPOST -d 'config_id: test_config_id
tap_config:
  match_config:
    http_request_headers_match:
       headers:
        - name: ":path"
          exact_match: "/productpage"
        #- name: ":path"
          #safe_regex_match: 
          #  google_re2: 
          #    max_program_size: 200
          #  regex: ".*productpage.*"
        #- name: ":method"
        #  exact_match: "POST"
  output_config:
    sinks:
      - format: JSON_BODY_AS_STRING
        streaming_admin: {}
    max_buffered_rx_bytes: 2097152      
    max_buffered_tx_bytes: 2097152' 'http://localhost:15000/tap'

通过 k8s log 方法:⌗

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: logpayload
  namespace: default
spec:
  workloadSelector:
    labels:
      app: productpage
  configPatches:
    # The first patch adds the lua filter to the listener/http connection manager

  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
#        portNumber: 15001
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
            subFilter:
              name: "envoy.router"
    patch:
      operation: INSERT_BEFORE
      value: # filter specification
          name: envoy.lua
          typed_config:
            "@type": type.googleapis.com/envoy.config.filter.http.lua.v2.Lua
            inline_code: |
              function logPayload(handle)
                    handle:logWarn("[logPayload] started")

                    local headStr = ""
                    for key, value in pairs(handle:headers()) do
                      headStr = headStr .. "\n" .. key .. ":" .. value .. ","
                    end             
                    
                    handle:logWarn("[logPayload:headers] --> " .. headStr)

                    local index = 0
                    for chunk in handle:bodyChunks() do
                        handle:logWarn('[logPayload] showing bodyChunks')

                        local len = chunk:length()
                        if len < 1 then
                          break
                        end
                        local result = chunk:getBytes(index, len)
                        index = index + len

                        handle:logWarn(result)
                    end
              end            

              function envoy_on_request(handle)
                  local path = handle:headers():get(":path")
                  if path ~= nil and path:find("productpage") ~= nil then
                    handle:streamInfo():dynamicMetadata():set("envoy.lua", "request.info", "true")                    
                    logPayload( handle )
                  end              
              end

              function envoy_on_response(handle)
                local path = handle:headers():get(":path")
                if path ~= nil and path:find("productpage") ~= nil then
                  handle:logWarn("[envoy_on_response] find productpage")
                end

                if handle:streamInfo():dynamicMetadata():get("envoy.lua") == nil then
                  return
                end

                local meta = handle:streamInfo():dynamicMetadata():get("envoy.lua")["request.info"]
                if( meta ~= nil ) then
                    handle:logWarn("[envoy_on_request] meta logPayload found")
                    logPayload( handle )
                end
              end

下载 istio-envoy-log-filter-by-path.yaml

Tap to file⌗

TODO

清理⌗

kubectl delete -f istio-envoy-log-filter.yaml

本文主要参考资料⌗

https://medium.com/@mtchkll/solving-microservice-mysteries-with-envoys-tap-filter-fd159c36d0af