Ship Kubernetes Microservices Logs to Kafka Topics

Ship Kubernetes Microservices Logs to Kafka Topics

Motivation

In my previous article https://medium.com/@fkarnagi/node-microservices-logging-on-kubernetes-c1768701fddb, I shared how to enable and configure Winston-based NodeJS Microservices logging in Kubernetes, powered by Rancher.

In this article, I am going to share my experience on how to publish and logs captured by Winston, and send it to Kafka.

This article assumes that you are using Rancher as the Kubernetes platform, as our instructions and screenshots are based on Rancher

Let’s Get to Work

First, after login to you cluster, and open your project, you need to enable the Logging.

Choose Kafka as the logging framework

You need to specify the Endpoint (could be more than one) and specify the topic.

Once enabled, you will be able to see the logs in Kafka Topic. I use Kafka Tool to browse the Kakfa topics from my Mac. You can download the tool here http://www.kafkatool.com/.