Debugging Intermittent Kong 503s When the Logs Showed Nothing but the Status Code

This story was originally published on HackerNoon at: https://hackernoon.com/debugging-intermittent-kong-503s-when-the-logs-showed-nothing-but-the-status-code.
Investigating intermittent Kong 503 errors with quiet logs, tracing the failure to an NGINX worker restart caused by Kubernetes memory pressure.
Check more stories related to programming at: https://hackernoon.com/c/programming. You can also check exclusive content about #kong-gateway, #kong, #nginx, #nginx-worker-restart, #kubernetes-oom, #lua-vm-metrics, #api-gateway-debugging, #sre-incident-analysis, and more.

This story was written by: @gudevamsikrishna. Learn more about this writer by checking @gudevamsikrishna's about page, and for more stories, please visit hackernoon.com.

We investigated intermittent Kong 503 responses where the logs showed only the status code and an empty upstream address. Worker-level metrics and dmesg eventually showed that a cgroup OOM event had killed one of the two NGINX workers in the Kong pod.