Hello Friend

Overview⌗

API Payload capture is a key tool for troubleshooting.

Tcpdump is the tool working at ISO layer 3~6, Many modern API working at Layer 7(HTTP), many capture rules are working only on Layer 7. E.g. filter payload by URL Path is hard for tcpdump:

tcpdump -i enp0s8 -s 0 -v -n -l | egrep -i "POST /|GET /|Host:"

Old school: tcpdump on minikube(FT) or Node local⌗

Below is an example of tcpdump.

[root@host-os ~]# ps -ef | grep -i myServicePS
100       398212  398058  0 Apr07 ?        00:26:08 java ...myServicePS...

set CONTAINER_PID_AT_HOST=398212

[root@host-os ~]# nsenter -t $CONTAINER_PID_AT_HOST  -u -i -n -p -C tcpdump -A -vv
...

In many cases, it is hard to find your API in the network flood from tcpdump.

Sidecar(Envoy) Filter/Tap⌗

In Istio(Service Mesh), Envoy is the sidecar. All API traffic proxy by it. It supports HTTP Filters, we can create some filter on Envoy to capture payload.

Two methods are supported:

  • Tap Fitler
  • Lua Filter

Setup⌗ 

kubectl apply -f istio-envoy-log-filter.yaml

#Optional: check k8s resource: 
kubectl describe envoyfilters.networking.istio.io logpayload

Optional: check envoy configuartion by envoy admin port 15000:

kubectl port-forward $your_pod_name 15000:15000
curl localhost:15000/config_dump | grep -i logPayload

You can download istio-envoy-log-filter.yaml here. workloadSelector and namespace is needed to updated for your case.

Capture⌗

Method 1: Capture by Tap Fitler⌗

Make sure we have port forward to envoy:

kubectl port-forward $your_pod_name 15000:15000

curl long poll:

curl -XPOST -d 'config_id: test_config_id
tap_config:
  match_config:
    any_match: true
  output_config:
    sinks:
      - format: JSON_BODY_AS_STRING
        streaming_admin: {}
    max_buffered_rx_bytes: 2097152      
    max_buffered_tx_bytes: 2097152' 'http://localhost:15000/tap'

Open another terminal, access an API of your application, e.g:

curl http://your-service:your-service-port/your-api-path

The ‘curl long poll’ return the API payload like this:

{
 "http_buffered_trace": {
  "request": {
   "headers": [
    {
     "key": ":path",
     "value": "/productpage"
    },
    {
     "key": ":method",
     "value": "GET"
    },
    {
     "key": ":scheme",
     "value": "http"
    },
    {
     "key": "user-agent",
     "value": "curl/7.64.1"
    },
    {
     "key": "accept",
     "value": "*/*"
    },
    {
     "key": "x-forwarded-for",
     "value": "10.244.1.1"
    }
   ],
   "trailers": []
  },
  "response": {
   "headers": [
    {
     "key": ":status",
     "value": "200"
    },
    {
     "key": "content-type",
     "value": "text/html; charset=utf-8"
    },
    {
     "key": "content-length",
     "value": "3889"
    },
    {
     "key": "server",
     "value": "istio-envoy"
    },
    {
     "key": "date",
     "value": "Sun, 12 Apr 2020 01:07:40 GMT"
    },
    {
     "key": "x-envoy-upstream-service-time",
     "value": "192"
    },
    {
     "key": "x-envoy-peer-metadata-id",
     "value": "sidecar~10.244.2.37~productpage-v1-5f7b7d4568-hmqzd.default~default.svc.cluster.local"
    }
   ],
   "body": {
    "truncated": false,
    "as_string": "\u003c!DOCTYPE html\u003e\n\u003chtml\u003e\n  \u003chead\u003e\n    \u003ctitle\u003eSimple Bookstore App\u003c/titleon('shown.bs.modal', function () {\n    $('#username').focus();\n  });\n\u003c/script\u003e\n\n  \u003c/body\u003e\n\u003c/html\u003e\n"
   },
   "trailers": []
  }
 }
}

Method 2: By kubectl logs⌗

Open another terminal, access an API of your application, e.g:

curl http://your-service:your-service-port/your-api-path

View the log of the sidecar(Envoy):

kubectl logs -f your-pod -c istio-proxy

The paylod in the log:

[Envoy (Epoch 0)] [2020-04-12 01:07:41.292][29][warning][lua] [external/envoy/source/extensions/filters/http/lua/lua_filter.cc:597] script log: <!DOCTYPE html>
<html>
  <head>
    <title>Simple Bookstore App</title>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">

How it work⌗

Let’s have a look at what in kubectl apply -f istio-envoy-log-filter.yaml :

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: logpayload
  namespace: default
spec:
  workloadSelector:
    labels:
      app: productpage
  configPatches:
    # The first patch adds the lua filter to the listener/http connection manager
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
#        portNumber: 15001
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
            subFilter:
              name: "envoy.router"
    patch:
      operation: INSERT_BEFORE
      value: # filter specification
        # name: envoy.filters.http.tap
        #   config:
        #     common_config:
        #       static_config:
        #         match_config:
        #           any_match: true
        #         output_config:
        #           sinks:
        #             - file_per_tap:
        #                 path_prefix: taps/any

        name: "envoy.filters.http.tap"
        config:
          common_config:
            admin_config:
              config_id: test_config_id

  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
#        portNumber: 15001
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
            subFilter:
              name: "envoy.router"
    patch:
      operation: INSERT_BEFORE
      value: # filter specification
          name: envoy.lua
          typed_config:
            "@type": type.googleapis.com/envoy.config.filter.http.lua.v2.Lua
            inline_code: |
              function logPayload(handle)
                  handle:logWarn("[logPayload] started")

                  local index = 0
                  for chunk in handle:bodyChunks() do
                      handle:logWarn('[logPayload] showing bodyChunks')

                      local len = chunk:length()
                      if len < 1 then
                        break
                      end
                      local result = chunk:getBytes(index, len)
                      index = index + len

                      handle:logWarn(result)
                  end

                  handle:logWarn("[logPayload] finished")
              end            

              function envoy_on_request(handle)
                logPayload(handle)
              end
              function envoy_on_response(handle)
                logPayload(handle)
              end

First, Envoy Filter:

Second, How Istio manage the envoy filter resource:

How to customize⌗

filter by service/deployment/replica⌗

update istio-envoy-log-filter.yaml :

spec:
  workloadSelector:
    labels:
      app: your-app-label

kubectl apply -f istio-envoy-log-filter.yaml

More info: https://istio.io/docs/reference/config/networking/envoy-filter/

filter by URL⌗

When capture by Tap:⌗ 
curl -XPOST -d 'config_id: test_config_id
tap_config:
  match_config:
    http_request_headers_match:
       headers:
        - name: ":path"
          exact_match: "/productpage"
        #- name: ":path"
          #safe_regex_match: 
          #  google_re2: 
          #    max_program_size: 200
          #  regex: ".*productpage.*"
        #- name: ":method"
        #  exact_match: "POST"
  output_config:
    sinks:
      - format: JSON_BODY_AS_STRING
        streaming_admin: {}
    max_buffered_rx_bytes: 2097152      
    max_buffered_tx_bytes: 2097152' 'http://localhost:15000/tap'
When capture by k8s log:⌗ 
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: logpayload
  namespace: default
spec:
  workloadSelector:
    labels:
      app: productpage
  configPatches:
    # The first patch adds the lua filter to the listener/http connection manager

  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
#        portNumber: 15001
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
            subFilter:
              name: "envoy.router"
    patch:
      operation: INSERT_BEFORE
      value: # filter specification
          name: envoy.lua
          typed_config:
            "@type": type.googleapis.com/envoy.config.filter.http.lua.v2.Lua
            inline_code: |
              function logPayload(handle)
                    handle:logWarn("[logPayload] started")

                    local headStr = ""
                    for key, value in pairs(handle:headers()) do
                      headStr = headStr .. "\n" .. key .. ":" .. value .. ","
                    end             
                    
                    handle:logWarn("[logPayload:headers] --> " .. headStr)

                    local index = 0
                    for chunk in handle:bodyChunks() do
                        handle:logWarn('[logPayload] showing bodyChunks')

                        local len = chunk:length()
                        if len < 1 then
                          break
                        end
                        local result = chunk:getBytes(index, len)
                        index = index + len

                        handle:logWarn(result)
                    end
              end            

              function envoy_on_request(handle)
                  local path = handle:headers():get(":path")
                  if path ~= nil and path:find("productpage") ~= nil then
                    handle:streamInfo():dynamicMetadata():set("envoy.lua", "request.info", "true")                    
                    logPayload( handle )
                  end              
              end

              function envoy_on_response(handle)
                local path = handle:headers():get(":path")
                if path ~= nil and path:find("productpage") ~= nil then
                  handle:logWarn("[envoy_on_response] find productpage")
                end

                if handle:streamInfo():dynamicMetadata():get("envoy.lua") == nil then
                  return
                end

                local meta = handle:streamInfo():dynamicMetadata():get("envoy.lua")["request.info"]
                if( meta ~= nil ) then
                    handle:logWarn("[envoy_on_request] meta logPayload found")
                    logPayload( handle )
                end
              end

You can download istio-envoy-log-filter-by-path.yaml here

Tap to file⌗

TODO

Clean up⌗ 

kubectl delete -f istio-envoy-log-filter.yaml

Ref.⌗