techs - Cyderable

Using Avro to Publish and Consume in Kafka PubSub Infrastructure

This article shares how to make use of Avro to publish and consume typed-messages to and from Kakfa broker. We begin with creating Avro schema in this content { "namespace": "sg.com.cyder.kafka.log.avro", "type": "record", "name": "ApiLog&

devsecops

Publishing Docker Image to JFrog Docker Registry

This article explains how to publish docker image to JFrog Docker Registry BackgroundBefore proceeding with this article, you might want to complete installation, as explained in this article https://ghost.cyder.com.sg/installing-jfrog-container-registry-on-ubuntu/ and enable NGINX SSL, as explained in this article https://ghost.cyder.com.sg/enabling-https-on-jfrog-artifactory/. Create

devsecops

Enabling HTTPS on JFrog Artifactory

Container Registry needs to be HTTPS accessible. This article explains an easy way to enable HTTPS on JFrog Artifactory BackgroundWe assume that you have completed JFrog Artifactor installation by following this story https://ghost.cyder.com.sg/installing-jfrog-container-registry-on-ubuntu/ Right now we are going to install NGINX, configure it as JFrog

cloudnative

Deploying Zookeeper Cluster

This is a third article in our Running Kafka in Kubernetes publication. Refer to https://medium.com/kafka-in-kubernetes/automating-storage-provisioning-41034e570928 to see the previous article. MotivationKafka uses Zookeeper to manage service discovery for Kafka Brokers that form the cluster. Zookeeper sends changes of the topology to Kafka, so each node in

cloudnative

Automating Storage Provisioning

MotivationTo achieve an automation of the storage provisioning in Kubernetes, we deploy our application using StatefulSet, and paired with Volume Claim Template. The backbone of Volume Claim Template is Storage Class. A StorageClass provides a way for administrators to describe the “classes” of storage they offer. Different classes might map

cloudnative

Provision the K8S Cluster

This article is the first one in the Running Kafka in Kubernetes publication. In this article, you will see the how setup a new Rancher cluster with 2 nodes, and configure NFS mount on each of those nodes. EC2 InstancesWe use EC2 in Oregon region for all our servers. Here

devsecops

Installing JFrog Container Registry on Ubuntu

JFrog Container Registry is an advanced, hybrid Docker registry & Helm registry that leverages the world-class artifact repository, JFrog Artifactory, which powers the world’s most extensive workloads. Preparing Ubuntu VMFirst, we prepare the Ubuntu VM. In our environment, we use AWS EC2. Please note on this hardware requirement before

eventstreaming

Node Microservices Logging on Kubernetes

MotivationNormally we simply use console.log(…) to write our debug messages. Although this message is piped to the Kubernetes managed container log, in /var/log/containers folder in the Kubernetes node, we might want to have custom formatted message, and transported to some other channels, for example File, Kafka, or

eventstreaming

Ship Kubernetes Microservices Logs to Kafka Topics

MotivationIn my previous article https://medium.com/@fkarnagi/node-microservices-logging-on-kubernetes-c1768701fddb, I shared how to enable and configure Winston-based NodeJS Microservices logging in Kubernetes, powered by Rancher. In this article, I am going to share my experience on how to publish and logs captured by Winston, and send it to Kafka. This

eventstreaming

Custom Kafka SinkConnector for ElasticSearch

MotivationWe have some applications running on Rancher Kubernetes platform and we have configured the Kubernetes containers logs to be shipped to Kafka Topics. Right now, we are going to developed a custom Kafka SinkConnector to parse the log records from Kafka Topics to ElaksticSearch. First we need to identity which