This document explains how to deploy Grafana Loki and Promtail on Kubernetes with Helm. After following this document, you can collect logs of Scalar products on your Kubernetes environment.

If you use a managed Kubernetes cluster and you want to use the cloud service features for monitoring and logging, please refer to the following document.

Prerequisites

Add the grafana helm repository

This document uses Helm for the deployment of Prometheus Operator.

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

Prepare a custom values file

Please get the sample file scalar-loki-stack-custom-values.yaml for loki-stack. For the logging of Scalar products, this sample file’s configuration is recommended.

In the production environment, it is recommended to add labels to the worker node for Scalar products as follows.

Since the promtail pods deployed in this document collect only Scalar product logs, it is sufficient to deploy promtail pods only on the worker node where Scalar products are running. So, you should set nodeSelector in the custom values file (scalar-loki-stack-custom-values.yaml) as follows if you add labels to your Kubernetes worker node.

  • ScalarDB Cluster Example
    promtail:
      nodeSelector:
        scalar-labs.com/dedicated-node: scalardb-cluster
    
  • (Deprecated) ScalarDB Server Example
    promtail:
      nodeSelector:
        scalar-labs.com/dedicated-node: scalardb
    
  • ScalarDL Ledger Example
    promtail:
      nodeSelector:
        scalar-labs.com/dedicated-node: scalardl-ledger
    
  • ScalarDL Auditor Example
    promtail:
      nodeSelector:
        scalar-labs.com/dedicated-node: scalardl-auditor
    

In the production environment, it is recommended to add taints to the worker node for Scalar products as follows.

Since promtail pods are deployed as DaemonSet, you must set tolerations in the custom values file (scalar-loki-stack-custom-values.yaml) as follows if you add taints to your Kubernetes worker node.

  • ScalarDB Cluster Example
    promtail:
      tolerations:
        - effect: NoSchedule
          key: scalar-labs.com/dedicated-node
          operator: Equal
          value: scalardb-cluster
    
  • (Deprecated) ScalarDB Server Example
    promtail:
      tolerations:
        - effect: NoSchedule
          key: scalar-labs.com/dedicated-node
          operator: Equal
          value: scalardb
    
  • ScalarDL Ledger Example
    promtail:
      tolerations:
        - effect: NoSchedule
          key: scalar-labs.com/dedicated-node
          operator: Equal
          value: scalardl-ledger
    
  • ScalarDL Auditor Example
    promtail:
      tolerations:
        - effect: NoSchedule
          key: scalar-labs.com/dedicated-node
          operator: Equal
          value: scalardl-auditor
    

Deploy Loki and Promtail

It is recommended to deploy Loki and Promtail on the same namespace monitoring as Prometheus and Grafana. You have already created the monitoring namespace in the document Monitoring Scalar products on the Kubernetes cluster.

helm install scalar-logging-loki grafana/loki-stack -n monitoring -f scalar-loki-stack-custom-values.yaml

Check if Loki and Promtail are deployed

If the Loki and Promtail pods are deployed properly, you can see the STATUS is Running using the kubectl get pod -n monitoring command. Since promtail pods are deployed as DaemonSet, the number of promtail pods depends on the number of Kubernetes nodes. In the following example, there are three worker nodes for Scalar products in the Kubernetes cluster.

$ kubectl get pod -n monitoring
NAME                                 READY   STATUS    RESTARTS   AGE
scalar-logging-loki-0                1/1     Running   0          35m
scalar-logging-loki-promtail-2fnzn   1/1     Running   0          32m
scalar-logging-loki-promtail-2pwkx   1/1     Running   0          30m
scalar-logging-loki-promtail-gfx44   1/1     Running   0          32m

View log in Grafana dashboard

You can see the collected logs in the Grafana dashboard as follows.

  1. Access the Grafana dashboard
  2. Go to the Explore page
  3. Select Loki from the top left pull-down
  4. Set conditions to query logs
  5. Select the Run query button at the top right

Please refer to the Monitoring Scalar products on the Kubernetes cluster for more details on how to access the Grafana dashboard.