Collecting logs from Scalar products on a Kubernetes cluster
This document explains how to deploy Grafana Loki and Promtail on Kubernetes with Helm. After following this document, you can collect logs of Scalar products on your Kubernetes environment.
If you use a managed Kubernetes cluster and you want to use the cloud service features for monitoring and logging, please refer to the following document.
Prerequisites
- Create a Kubernetes cluster.
- Create a Bastion server and set
kubeconfig
. - Deploy Prometheus Operator (we use Grafana to explore collected logs)
Add the grafana helm repository
This document uses Helm for the deployment of Prometheus Operator.
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
Prepare a custom values file
Please get the sample file scalar-loki-stack-custom-values.yaml for loki-stack. For the logging of Scalar products, this sample file's configuration is recommended.
Set nodeSelector in the custom values file (Recommended in the production environment)
In the production environment, it is recommended to add labels to the worker node for Scalar products as follows.
- EKS - Add a label to the worker node that is used for nodeAffinity
- AKS - Add a label to the worker node that is used for nodeAffinity
Since the promtail pods deployed in this document collect only Scalar product logs, it is sufficient to deploy promtail pods only on the worker node where Scalar products are running. So, you should set nodeSelector in the custom values file (scalar-loki-stack-custom-values.yaml) as follows if you add labels to your Kubernetes worker node. See the following examples based on the product you're using.
- ScalarDB
- ScalarDL
Select the ScalarDB product you're using.
- ScalarDB Cluster
- ScalarDB Server (deprecated)
promtail:
nodeSelector:
scalar-labs.com/dedicated-node: scalardb-cluster
promtail:
nodeSelector:
scalar-labs.com/dedicated-node: scalardb
Select the ScalarDL product you're using.
- ScalarDL Ledger
- ScalarDL Auditor
promtail:
nodeSelector:
scalar-labs.com/dedicated-node: scalardl-ledger
promtail:
nodeSelector:
scalar-labs.com/dedicated-node: scalardl-auditor
Set tolerations in the custom values file (Recommended in the production environment)
In the production environment, it is recommended to add taints to the worker node for Scalar products as follows.
- EKS - Add taint to the worker node that is used for toleration
- AKS - Add taint to the worker node that is used for toleration
Since promtail pods are deployed as DaemonSet, you must set tolerations in the custom values file (scalar-loki-stack-custom-values.yaml) as follows if you add taints to your Kubernetes worker node. See the following examples based on the product you're using.
- ScalarDB
- ScalarDL
Select the ScalarDB product you're using.
- ScalarDB Cluster
- ScalarDB Server (deprecated)
promtail:
tolerations:
- effect: NoSchedule
key: scalar-labs.com/dedicated-node
operator: Equal
value: scalardb-cluster
promtail:
tolerations:
- effect: NoSchedule
key: scalar-labs.com/dedicated-node
operator: Equal
value: scalardb
Select the ScalarDL product you're using.
- ScalarDL Ledger
- ScalarDL Auditor
promtail:
tolerations:
- effect: NoSchedule
key: scalar-labs.com/dedicated-node
operator: Equal
value: scalardl-ledger
promtail:
tolerations:
- effect: NoSchedule
key: scalar-labs.com/dedicated-node
operator: Equal
value: scalardl-auditor
Deploy Loki and Promtail
It is recommended to deploy Loki and Promtail on the same namespace monitoring
as Prometheus and Grafana. You have already created the monitoring
namespace in the document Monitoring Scalar products on the Kubernetes cluster.
helm install scalar-logging-loki grafana/loki-stack -n monitoring -f scalar-loki-stack-custom-values.yaml
Check if Loki and Promtail are deployed
If the Loki and Promtail pods are deployed properly, you can see the STATUS
is Running
using the kubectl get pod -n monitoring
command. Since promtail pods are deployed as DaemonSet, the number of promtail pods depends on the number of Kubernetes nodes. In the following example, there are three worker nodes for Scalar products in the Kubernetes cluster.
kubectl get pod -n monitoring
You should see the following output:
NAME READY STATUS RESTARTS AGE
scalar-logging-loki-0 1/1 Running 0 35m
scalar-logging-loki-promtail-2fnzn 1/1 Running 0 32m
scalar-logging-loki-promtail-2pwkx 1/1 Running 0 30m
scalar-logging-loki-promtail-gfx44 1/1 Running 0 32m
View log in Grafana dashboard
You can see the collected logs in the Grafana dashboard as follows.
- Access the Grafana dashboard
- Go to the
Explore
page - Select
Loki
from the top left pull-down - Set conditions to query logs
- Select the
Run query
button at the top right
Please refer to the Monitoring Scalar products on the Kubernetes cluster for more details on how to access the Grafana dashboard.