Run thousand of Browser automation cluster with cost effective

Hello all ,

today we will share how to create selenium grid cluster using kubernetes , so it will expand if the browser are needed but will be shrink if the browser are not needed. so with this setup it will be possible if you are running hundred of automation but you just pay only the resource that you are using. Sometimes if we are using the old static setup , the number of the browser that we are spawning is depend on the resource that we have and might be not all that browser are needed , so it will waste your resource, but using this setup the resource will be following the number of the browser that you are using.

Part 1 : The requirement

We will doing POC using your local PC , so if you want to implement it in the GCP or any cloud service that using Kubernetes , it can be easy for that.

on the requirement , we want to build cluster that can handle so many chrome, but the pods should be grow when needed and shrink when not needed.

and also the selenium grid should be able to be accessed by our selenium test outside the cluster ( like open public service ). so the diagram should be like this one

POC Selenium Kubernetes Diagram — by Argo

on that diagram , our browser are running on the container based inside a cluster , so there will be 1 browser in each pod in the cluster.

Part 2 : The Recipe

in order to do POC , we need this recipe.

1. Docker are installed ( or any other container tools like podman , kvm etc are fine) in this POC we will try to using Docker
2. Minikube : you can download in this link https://minikube.sigs.k8s.io/docs/start
3. Automation for the demo
4. Little bit knowledge on the Selenium grid , you can learn it here https://www.selenium.dev/documentation/grid/

After understanding about the tools that we will use , we can start to install and built your own POC. Please make sure it’s running fine

Part 3 : Installation

Docker

Make sure you are already running docker in your pc , this is important because all of them are bases on the container ( including Kubernetes ) , you can follow this tutorial based on your operating system

https://docs.docker.com/engine/install/

If you have already follow that installation , you can check by running this command

docker run hello-world

you should able to see the result like this screenshot

if the result is different , that’s mean your installation are not correct , please troubleshoot it first before continue to the next steps.

Minikube

Minikube is a tool that makes it easy to run Kubernetes locally. It does this by setting up a virtual machine on your computer and deploying a simple cluster consisting of only one node. This setup is ideal for individuals who are learning Kubernetes or developing applications that will run on it. Minikube supports various operating systems like Linux, macOS, and Windows, and offers a simplified experience for managing the cluster with basic commands to start, stop, and interact with your Kubernetes cluster.

TLDR; minikube is a tools for learning kubernetes in your local, you can learn minikube in their site

https://minikube.sigs.k8s.io/docs/start/

Minikube Logo

After you are successful install the minikube , you can check if it’s working or not by using this command to start the minikube ( make sure the docker are working fine in the previous steps)

minikube start

it will start some container for minikube

you can enable metrics-server , so we can monitor the kubernetes cluster that we are using.

minikube addons enable metrics-server

and you can start the dashboard for better view.

minikube dashboard

after you run that command , it should open the new dashboard with nice view

Below is the some explanation about the dashboard

1. Namespace or the project , so this one imagine you have some team that working together , each team can have 1 name space , so the team can focusing to monitor their service and their resource.
2. Workloads : there are some menu under workloads but the main function of that is to monitor and also manage resources and service inside your kubernetes cluster.
3. PODS : Imagine you have a bunch of toy blocks that you want to keep together so they don’t get lost. In the world of Kubernetes, these toy blocks are like containers, which are little packages of software. A pod is like toys box that can have different containers and work together to do something cool, like help you build a neat website or app!

There are some basic kubernetes that nice to have if you want to build this POC , if you need to learn that , we recommend to use this one from official documentation

Learn Kubernetes Basics

KEDA — Kubernetes Event-driven Autoscaller

This is the main menu that we are talking about , yes , KEDA or we called autoscaller based on teh event driven. So in short keda will listen to the some event or getting some data , if they match to the criteria , KEDA will start new pod based on the autoscaller deployment that already defined in their configuration.

For installation keda , you just type this command your terminal

kubectl apply --server-side -f https://github.com/kedacore/keda/releases/download/v2.14.0/keda-2.14.0.yaml

and after that , you will see new namespace on your kubernetes dashboard

and there will be new pod are spawned in ther , this pod are used by keda for autoscaller

Selenium Grid

after the keda is installed , you can start to prepare the selenium grid , actualy if you having the baremetal server or vm it will very good as long as they are able to communicate with the cluster it’s fine. but in this sample , we will use selenium grid that running on the kubernetes.

first , we need to install service for selenium Grid , you can copy this xml and save with the name selenium-hub-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: selenium-hub
  labels:
    app: selenium-hub
spec:
  ports:
  - port: 4444
    targetPort: 4444
    name: port0
  - port: 4443
    targetPort: 4443
    name: port1
  - port: 4442
    targetPort: 4442
    name: port2
  selector:
    app: selenium-hub
  type: NodePort
  sessionAffinity: None

and then you can run

kubectl apply -f ./selenium-hub-svc.yaml --namespace=keda

Please make sure you are on the same folder that yaml file are downloaded or created.

and the next you can run deployment that contains full of the selenium grid application , you can create new yaml file and copy this sample of yaml. you caan name it selenium-hub-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: selenium-hub
  labels:
    app: selenium-hub
spec:
  replicas: 1
  selector:
    matchLabels:
      app: selenium-hub
  template:
    metadata:
      labels:
        app: selenium-hub
    spec:
      containers:
      - name: selenium-hub
        image: selenium/hub:4.20.0
        ports:
          - containerPort: 4444
          - containerPort: 4443
          - containerPort: 4442
        resources:
          limits:
            memory: "1000Mi"
            cpu: ".5"
        livenessProbe:
          httpGet:
            path: /wd/hub/status
            port: 4444
          initialDelaySeconds: 30
          timeoutSeconds: 5
        readinessProbe:
          httpGet:
            path: /wd/hub/status
            port: 4444
          initialDelaySeconds: 30
          timeoutSeconds: 5

and reapply using this one

kubectl apply -f ./selenium-hub-deployment.yaml --namespace=keda

in this step, you should able to see the selenium grid pods are spawned

you can run this command so you can see the selenium grid

minikube service selenium-hub --namespace keda

you will see this output ,

there will be some url with port , you can accessing the top one , since the port 4444 is in the first place ,

and the last is install the browser. same steps with selenium grid instalation , you can get this sample of xml and put in your directory with name chrome-scalled-object.yaml

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: selenium-grid-chrome-scaledobject
  namespace: keda
  labels:
    deploymentName: selenium-node-chrome
spec:
  maxReplicaCount: 8
  minReplicaCount: 1
  scaleTargetRef:
    name: selenium-node-chrome
  triggers:
    - type: selenium-grid
      metadata:
        url: 'http://selenium-hub.keda:4444/graphql'
        browserName: 'chrome'

kubectl apply -f ./chrome-scalled-object.yaml --namespace=keda

you can change maxReplica and min replica based on your needed , in this sample , the minimum browser that spawned are 1 and max are 8 , so during no automation run , there are 1 browser standby for getting request , if you don’t want to be there , you can adjust by your needed.

after that selenium grid dashboard will be like this one

behind that , the chrome pod are only 1

Testing

you can try to running 2 parallel automation in that selenium grid , at the first time , there will be queue

but after waiting for pods spawn the browser , there will be new browser that ready to handle the queue

in the background the pods are spawn new browser for handling the new queue

after it’s finished , the selenium grid will drain the pods and reduce the pod

and the browser available are back to 1

Part 4 — Conclusion

This proof of concept (POC) demonstrates a significant advancement in resource management within Kubernetes clusters. By enabling dynamic configuration of browsers, it streamlines the process of running automated tests in environments like Selenium or Selenium Grid.

This flexibility means that even if the demand scales to thousands of browsers, adjustments can be made swiftly and efficiently through configuration changes alone, without the need for extensive setup. This approach not only conserves resources but also enhances the agility of the testing infrastructure, making it a robust solution for large-scale automation testing.

You can also combine it with jenkins or any ci tools that you are using in your company and combine it with gcp , amazon aws or any Kubernetes vendor or provider, but please do calculation. since running resource in the cloud sometimes is more expensive than running in the baremetal , we recomended to do some calculations based on their’s price.