You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This repository is for setting up load testing environment on GKE with terraform.
3
+
4
+
![diagram](docs/diagram.png?raw=true"Diagram")
5
+
# Pre Request
6
+
- gcloud >= Google Cloud SDK 349.0.0
7
+
- kubernetes-cli >= 1.22.1
8
+
- terraform >= 1.0.5
9
+
- python >= 3.9 (To generate diagram)
10
+
# How to Set Up on GKE
11
+
## Configure Makefile
12
+
Copy `Makefile.example` and fill out attributes below:
13
+
| value | description |
14
+
|:-- |:--|
15
+
| PROJECT_ID | GCP Project ID |
16
+
| CLUSTER_NAME | Cluster base name. Due to the cluster deletion takes time, this tool add a random texts at the end of the base cluster name |
17
+
| REGION | GCP Region name |
18
+
| ZONE | GCP Zone name |
19
+
| MACHINE_TYPE | Machine type of loading machines. Please see [machine types](https://cloud.google.com/compute/docs/general-purpose-machines) for more details |
20
+
| CREDENTIALS | The full path to the Service Account JSON file. |
21
+
| SERVICE_ACCOUNT_EMAIL | Service Account Email. Eg. `[User name]@[Project name].iam.gserviceaccount.com`|
22
+
| TARGET_HOST | Target host URL |
23
+
24
+
## Set Up Google Kubernetes Cluster (GKE)
25
+
1. Navigate to `deploy` folder.
26
+
27
+
```
28
+
make init_all
29
+
```
30
+
to set up `terraform`
31
+
1. Run
32
+
```
33
+
make build
34
+
```
35
+
to set up a GKE cluster and initialize and `gcloud` command pointing to the created GKE cluster.
36
+
1. Run
37
+
```
38
+
make a_locust
39
+
```
40
+
to set up `locust` and required config maps (storing load test scripts) for performance testing.
41
+
1. Run
42
+
```
43
+
make locust
44
+
```
45
+
This will do port forwarding to the local. Then you can access to `Locust Master` with `localhost:8089`.
46
+
1. Stop `make locust` and Run
47
+
```
48
+
make refresh
49
+
```
50
+
This will refresh the Locust Cluster with updated `main.py` script file and `values.yaml` content. Once the Locust Cluster up and running, connect the master with `make locust`
51
+
## Tear Down GKE Cluster
52
+
Run
53
+
```
54
+
make d_all
55
+
```
56
+
57
+
## Update Code for Load Testing
58
+
At each load testing scripts update, workers need to be redeployed to read the latest config maps where testing scripts are stored according to the Kubernetes specification. This way allows you to update with one command.
59
+
60
+
1. If you are already connecting the load cluster with `make locust`, Ctrl+C to stop it.
61
+
1. All code is stored under `locust` directory. `main.py` is the main logic, and libraries are under the `lib` directory.
62
+
1. Once code is updated, run
63
+
```
64
+
make refresh
65
+
```
66
+
to reload `ConfigMap` and Locust clusters to read the updated config map.
67
+
1. Run `make locust` again to connect the load cluster.
68
+
69
+
## How to Adjust Balance of Workers and Users
70
+
To generate the load at a lower cost, you may want to use as few workers as possible. This is a sample step on how to adjust the number of users and workers appropriately.
71
+
72
+
In the case of generating 10000 RPS, here are the steps that I tried.
73
+
74
+
1. Enable HPA, start from 10 workers with 2000 users, and see how much load the Locust cluster can generate. In this case, Locust generated 3000 RPS and saturated there. No CPU errors are observed in Cloud Logging, which implies CPU is still not pushed to the limit.
75
+
1. Assuming 3 times more users would generate 10000 RPS. Change users to 6000 and run `make refresh` to restore `ConfigMap` and Locust pods.
76
+
1. You observed workers automatically scaled to 15 and the load reached higher than 10000 RPS.
77
+
1. Adjust the initial worker to `15` in the `values.yaml` and `make refresh` to update the Locust pods.
78
+
79
+
## Reference Settings
80
+
In the case where you use `spike_load.py` to generate **10000RPS** with the Locust Cluster on GKE, here is the reference configuration.
81
+
82
+
`spike_load.py` hatches users at once and hold requests until all users are spawned **in each worker** (not across all workers).
83
+
84
+
| parameters | description |
85
+
|:-- | :-- |
86
+
| Machine type of locust worker (`MACHINE_TYPE` in `Makefile`) | e2-standard-2 |
87
+
| Replicas for worker (line 66 of `values.yaml`) | 15 |
88
+
| User amount (line 15 of `spike_load.ph`, `user_amount`) | 10000 |
89
+
90
+
With this settings,
91
+
- The first second RPS is around 600
92
+
- It'll reach 10000RPS in 15 to 20 seconds, and go higher. You may want to pace the access with `constant_pacing` function if you exactly target 10000RPS and dwell (stay) for a while.
93
+
94
+
In `spike_load.py`, the below line configure the dwell load time. This code means dwell 120 seconds with amount of user_amount users. Adust dwell time accordingly.
95
+
```spike_load.py
96
+
targets_with_times = Step(user_amount, 120)
97
+
```
98
+
99
+
# How to Run Locally
100
+
You may want to iterate try and error quickly while building a testing script. Loading the testing script every time on GKE is quite troublesome. For the development phase, you can leverage Docker to run a small cluster locally.
101
+
102
+
Spin up the small locust cluster, run
103
+
```
104
+
docker-compose up --build --scale worker=1
105
+
```
106
+
and you can access to the master from `localhost:8089`
107
+
108
+
# Tips
109
+
110
+
## Test Script Locally first and move on the production.
111
+
112
+
Locust stops with exceptions when syntax errors are included in the loading script. For a faster turnaround, you may want to make sure the script works correctly at the local first and move on to the production.
113
+
## Help of Commands
114
+
Run `make help`
115
+
## How to Access Locust Master Manually
116
+
1. Go to GCP console > `Services & Ingress`
117
+
1. Open `locust-cluster`, scroll down to `Ports`
118
+
1. Click `PORT FORWARDING` button of `master-p3`, with port `8089` row
119
+
1. A dialog will be popped up and displays the port forwarding code in there. Copy & Paste onto the terminal, and run.
120
+
1. You can access the `locust-cluster` master pod with `localhost:8080` from your browser.
121
+
## How to Configure gcloud for The GKE Cluster by Default
122
+
This can be done just run `make build`, but also separately as below:
123
+
1. Build cluster with
124
+
```
125
+
make build_cluster
126
+
```
127
+
128
+
1. Run
129
+
```
130
+
make gcloud_init
131
+
```
132
+
This command will configure your `gcloud` environment pointing to the newly created GKE cluster.
133
+
## How to Generate Diagram
134
+
1. Install `Diagrams` following [this step](https://diagrams.mingrammer.com/docs/getting-started/installation).
135
+
1. Go to `docs` directory and run `python diagram.py`
136
+
137
+
## How to Enable Autoscaling
138
+
Autoscaling is depending on Kubernetes's [Horizontal Pod Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#how-does-the-horizontal-pod-autoscaler-work)(HPA). To enable HPA, Kubernetes manifest needs to include `resource` to sepecify the pod's resource allocation so that Kubernetes can manage the pods based on the CPU usage.
&& kubectl port-forward $(kubectl get pod --selector="app.kubernetes.io/instance=locust-cluster,app.kubernetes.io/name=locust,component=master,load_test=locust-cluster" --output jsonpath='{.items[0].metadata.name}') 8089:8089
0 commit comments