[GPII-3188]: First crack at smoke testing for GCP infra with Locust by natarajaya · Pull Request #93 · gpii-ops/gpii-infra

natarajaya · 2018-08-19T00:17:56Z

To deploy Locust: rake sh['xk up live/dev/locust']
To start the swarm: rake sh['sh -c "TF_VAR_locust_swarm=1 xk up live/dev/locust"']

I don't like the way how chart handles task scripts. There is also an issue if you want to work with multiple hosts (i.e. preferences. and flowmanager.) within the same task set. But otherwise this approach is pretty flexible.

natarajaya · 2018-08-19T05:27:20Z

Typical "failed run":

Starting Locust swarm with 100 users and hatch rate of 10!
{"message": "Swarming started", "success": true}
USERS  RPS     STATUS
0	0	hatching
0	0	hatching
15	0	hatching
15	0	hatching
35	11	hatching
35	11	hatching
55	16	hatching
55	16	hatching
75	20	hatching
75	20	hatching
100	0	running
100	0	running
100	40	running
100	40	running
100	47	running
100	47	running
100	47	running
100	47	running
100	46	running
100	46	running
100	47	running
100	47	running
100	46	running
100	46	running
100	46	running
100	46	running
100	46	running
100	46	running
100	46	running
100	46	running
100	47	running
100	47	running
100	47	running
100	47	running
100	47	running
100	47	running
100	48	running
100	48	running
100	48	running
100	48	running
100	48	running
100	48	running
100	48	running
100	48	running
100	48	running
100	48	running
100	48	running
100	48	running
100	47	running
100	47	running
100	48	running
100	48	running
100	47	running
100	47	running
100	48	running
100	48	running
100	47	running
100	47	running
100	48	running
100	48	running

Swarming complete!
{"message": "Test stopped", "success": true}
Processing stats...

{"errors": [], "stats": [{"median_response_time": 82, "min_response_time": 23, "current_rps": 22.7, "name": "/preferences/carla", "num_failures": 0, "max_response_time": 618, "avg_content_length": 1549, "avg_response_time": 101.65651558073654, "method": "GET", "num_requests": 1412}, {"median_response_time": 84, "min_response_time": 25, "current_rps": 27.0, "name": "/preferences/vladimir", "num_failures": 0, "max_response_time": 577, "avg_content_length": 1811, "avg_response_time": 102.4164402173913, "method": "GET", "num_requests": 1472}, {"median_response_time": 84, "min_response_time": 23, "current_rps": 49.7, "name": "Total", "num_failures": 0, "max_response_time": 618, "avg_content_length": 1682, "avg_response_time": 102.04438280166436, "method": null, "num_requests": 2884}], "fail_ratio": 0.0, "slave_count": 3, "state": "stopped", "user_count": 0, "total_rps": 49.7}

Looks like max_response_time (618) is worse than desired (200)!
This is unacceptable!

Stats distribution:
"Name","# requests","50%","66%","75%","80%","90%","95%","98%","99%","100%"
"GET /preferences/carla",1433,82,100,120,130,190,250,320,370,618
"GET /preferences/vladimir",1492,85,110,130,140,190,240,300,370,577
"None Total",2925,84,100,120,140,190,240,310,370,618
Resetting stats...
ok

amatas · 2018-08-20T10:31:51Z

+
+      echo
+      echo $SESSION_STATS
+      SESSION_SUCCEEDED=1


Do you think that using error codes could be more useful in a future? it's just a suggestion.

Sounds reasonable! Thanks @amatas !

…for ready workers before starting the swarm

mrtyler

This is neat.

I realize it's still early in this work, so just a couple high-level observations below.

Remember that preferences server is supposed to stop being publicly available someday, so you might start thinking about whether/how to expose preferences server for testing, and/or how to do OAuth with flowmanager so we can test preferences through flowmanager.

mrtyler · 2018-08-20T23:54:12Z

+
+  provisioner "local-exec" {
+    command = <<EOF
+      mkdir -p ${var.charts_dir}/locust/tasks


FYI this will create files as root inside a bind mount, which is likely to cause trouble.

Alternative would be to keep locust scripts (which are app-specific) in common chart dir. It is not possible to modify this behavior, since Helm does not give file system access beyond chart dir...

It works well locally, with your changes from #92 merged. I also added cleanup script to remove those links after chart creation to eliminate possible issues with permissions on the host.

Do you think we'll still have troubles on CI workers with this?

Did you test it on Linux, where the root ownership problem occurs?

Another alternative would be to mount a Docker Volume into the expected place inside the container, the same strategy we use to solve the problem for other locations that need to be written from inside the container. You can see examples of how this works in #92.

Nope, I did not test this on Linux yet. Having separate Docker Volume just for Locust tasks is probably okay, but I would be happy to avoid this. Let's see if cleaning up links from inside the container works.

mrtyler · 2018-08-21T00:05:06Z

+    command = <<EOF
+      if [ "${var.locust_swarm}" == "" ]; then
+        echo "Looks like TF_VAR_locust_swarm is unset, terminating!"
+        exit


Should we exit non-zero in this case?

Idea behind this is that there may be a situation, when we want to deploy the Locust, but don't actually want to start the swarm. I probably overcomplicate things as always.

mrtyler · 2018-08-21T00:07:52Z

+
+      RETRIES=10
+      RETRY_COUNT=1
+      while [ "$WORKERS_READY" != "${var.locust_workers}" ]; do


This is a lot of in-line code.

Automation cost.

I don't understand this response.

Probably because I did not understand the question :) What did you mean by "a lot of in-line code"?

This is a little clearer after you talked about this PR a bit in standup today.

This is a lot of code to be in-line. Maybe move it to separate file.

This seems like kind of a lot of (custom) code to run as part of a deployment, but I guess we need some glue to run Locust, collect stats, and assert that various stats were within certain ranges.

This seems like a lot of shell code. It's pretty simple right now, but perhaps a proper test framework that knows how to assert that stats are within a range, run subsets, generate reports, etc. is better than writing our own.

Relatedly, maybe it's worth moving this code to a more featureful language than shell. Python might be a good fit since Locust tasks are in Python. Ruby might be a good fit since gpii-infra is in Ruby.

It is already in a separate file.

Yes.

I was also thinking on this question. There are lots of data in Locust stats. But, while idea to have assertions with customizable thresholds per Locust task, seems to be useful in theory, I still have not decided if this would give us any practical benefit. Most meaningful assertions (on median / max response times, RPS, failure counts) are already in the code. Separate task sets can have separate thresholds too, in case we need them. Locust reports are present in logs as well, we can send them into Bigtable or something, to collect historical data. I think this topic worth a discussion with other Ops (and maybe some Devs as well?)

Usually with this much "in-line code", I would expect it to live in a dedicated file run_swarm_test.sh or something. But you're right, I see what you mean.

\3. Definitely worth a larger discussion.

mrtyler · 2018-08-22T04:02:15Z

+      RETRY_COUNT=1
+      while [ "$WORKERS_READY" != "${var.locust_workers}" ]; do
+        echo "[Try $RETRY_COUNT of $RETRIES] Waiting for all Locust workers to join the master..."
+        WORKERS_READY=$(kubectl -n locust logs deployment/locust-master --tail 1 | grep -oE "Currently \d+ clients" | grep -oE "\d+")


Does locust-master produce any other logs that might prevent us from seeing the "Currently \d+ clients" line?

It produces some other logs, but last messages are always about workers joining the master. I tried different scenarios, changing locust_workers up and down (which scales workers deployment) and target_host (which recreates master pod). It works.

mrtyler · 2018-08-22T04:03:13Z

+      while [ "$WORKERS_READY" != "${var.locust_workers}" ]; do
+        echo "[Try $RETRY_COUNT of $RETRIES] Waiting for all Locust workers to join the master..."
+        WORKERS_READY=$(kubectl -n locust logs deployment/locust-master --tail 1 | grep -oE "Currently \d+ clients" | grep -oE "\d+")
+        if [ "$WORKERS_READY" != "" ]; then


I would refactor this to:

if [ "$WORKERS_READY" == "" ]; then WORKERS_READY=0 fi echo "Number of ready workers: $WORKERS_READY out of ${var.locust_workers}!"

Looks better, thanks!

EDIT: Never mind, I was looking at the wrong set of commits. Sorry for the noise.

Bump

mrtyler · 2018-08-22T04:18:55Z

+
+      RETRIES=10
+      RETRY_COUNT=1
+      while [ "$WORKERS_READY" != "${var.locust_workers}" ]; do


This is a little clearer after you talked about this PR a bit in standup today.

This is a lot of code to be in-line. Maybe move it to separate file.

This seems like kind of a lot of (custom) code to run as part of a deployment, but I guess we need some glue to run Locust, collect stats, and assert that various stats were within certain ranges.

This seems like a lot of shell code. It's pretty simple right now, but perhaps a proper test framework that knows how to assert that stats are within a range, run subsets, generate reports, etc. is better than writing our own.

Relatedly, maybe it's worth moving this code to a more featureful language than shell. Python might be a good fit since Locust tasks are in Python. Ruby might be a good fit since gpii-infra is in Ruby.

mrtyler · 2018-08-22T04:19:37Z

+      echo
+      echo "Resetting stats..."
+      curl -s $LOCUST_URL/stats/reset
+      kill $(pgrep kubectl)


This seems potentially dangerous, especially with terragrunt's parallelism.

I don't think we ever want to run this in parallel with other modules as part of the deployment. I would rather think of this as a separate build step that only deploys locust module and runs the swarm. But yeah, kill $(pgrep -f "kubectl -n locust port-forward").

EDIT: Never mind, I was looking at the wrong set of commits. Sorry for the noise.

Bump

That revised pgrep looks good.

stepanstipl · 2018-08-22T15:20:50Z

What is the end game for this - deploy locust into dev clusters, and run it against local GPII deployment every time TF runs?

natarajaya · 2018-08-23T04:33:17Z

@stepanstipl I am thinking of this as of separate step in CI pipeline (instead of current "smoke tests" that we have for AWS). We could start with deploying locust into the same cluster. Run nightly benchmarks for stg and prd from dev. End game – have "utility" cluster, so we can benchmark from there?

@mrtyler As long as Locust deployed into the same cluster, it can swarm NodePort or ClusterIP targets with no issue. Locust can do OAuth.

mrtyler · 2018-08-25T00:14:40Z

@natarajaya

I am thinking of this as of separate step in CI pipeline

I like this. I think it would come after stg (and could potentially use stg as the cluster-under-test).

(instead of current "smoke tests" that we have for AWS).

I'm not sure (these) Locust tests are a replacement for smoke tests. Something lightweight that runs as part of deployment gives us quick feedback on the health of that deployment.

A sort of compromise might be lightweight Locust tests that run during deployment (so we don't have to write and maintain scripts that run curl), and heavier performance-focused Locust tests that run in a separate step.

I think this is a good addition to the larger discussion of testing we've been trying to have for a few days :).

We could start with deploying locust into the same cluster. Run nightly benchmarks for stg and prd from dev. End game – have "utility" cluster, so we can benchmark from there?

I like doing it from dev in the short-term -- a nice environment that is destroyed and re-created regularly. We can add variables to the nightly build and use them to only run the benchmark step during nightly builds (not during triggered CI builds).

Longer term I like a "utility environment", probably the same environment where the CI worker lives.

@mrtyler As long as Locust deployed into the same cluster, it can swarm NodePort or ClusterIP targets with no issue.

Cool. Hitting the NodePort/ClusterIP directly is a little different from going through the public IP, Google LB, nginx-ingress, etc. I guess it depends on exactly what we're trying to test.

Locust can do OAuth.

That example code looks good.

natarajaya · 2018-08-28T19:48:21Z

So, I think things look a bit better after refactoring.

To start preferences test:
rake sh['sh -c "xk down live/dev/locust && TF_VAR_locust_target_host=http://preferences.$TF_VAR_domain_name TF_VAR_locust_script=preferences.py xk up live/dev/locust"']

To start flowmanager test:
rake sh['sh -c "xk down live/dev/locust && TF_VAR_locust_target_host=http://flowmanager.$TF_VAR_domain_name TF_VAR_locust_script=flowmanager.py TF_VAR_locust_users=30 TF_VAR_locust_desired_total_rps=10 TF_VAR_locust_desired_median_response_time=200 TF_VAR_locust_desired_max_response_time=700 xk up live/dev/locust"']

I configured default thresholds so preferences test can pass without need to change anything. Interestingly, flowmanager shows much worse performance, so thresholds needs to be adjusted accordingly.

natarajaya · 2018-08-28T20:01:20Z

Added tests to CI config.
There is an issue with multiline scripts, I followed a workaround advice that supposed to work, but I have no way to verify.

mrtyler

Couple small rename suggestions.

After those, I think this can be merged. Excited to start collecting this data regularly!

mrtyler · 2018-08-29T18:44:11Z

-      LOCUST_URL=http://127.0.0.1:8089
-      kubectl -n locust port-forward deployment/locust-master 8089:8089 </dev/null &>/dev/null &
+      LOCUST_URL="http://127.0.0.1:8089"
+      KUBECTL_CMD="kubectl -n locust port-forward deployment/locust-master 8089:8089"


I would call this PORT_FORWARD_CMD.

Alright then.

mrtyler · 2018-08-29T18:46:50Z

+
  @task
-  def prefByKey(self):
+  def getPrefByKey(self):


Generally in Python, methods/functions should use snake_case. But maybe Locust dictates this name?

Nope, Locust dictates nothing here, the only requirement is clear connection between task name and requests it is making, because task names are not present in stats. Let's switch to snake_case if you think it better follows Python naming style.

mrtyler · 2018-08-29T18:48:38Z


 variable "locust_swarm_duration" {
-  default = 60
+  default = 50


Right, 50 means 60 because we add 10 to it later.

It's fine the way you've implemented it but you don't think this is weird? Do you think it warrants a comment?

Ok, let's just exit if locust_swarm_duration is unset.

mrtyler · 2018-08-29T18:56:20Z

    - rake configure_serviceaccount_ci_restore
    - rake
+    # Run smoke tests for Preferences
+    - rake sh['sh -c "


This command is pretty complex, and is something that we might want to run ourselves. Can you make two new rake tasks, :smoke_test_preferences and :smoke_test_flowmanager (or something), and call them from here?

Makes sense!

mrtyler

Couple more little things.

mrtyler · 2018-08-30T03:00:43Z

+task :test_preferences => [:set_vars] do
+  sh "#{@exekube_cmd} rake xk[' \
+    xk down live/dev/locust && \
+    sleep 30 && \


Can you add echo Sleeping 30s... here so the user doesn't think the process is stuck?

See next thread.

mrtyler · 2018-08-30T03:00:52Z

+task :test_flowmanager => [:set_vars] do
+  sh "#{@exekube_cmd} rake xk[' \
+    xk down live/dev/locust && \
+    sleep 30 && \


Can you add echo Sleeping 30s... here so the user doesn't think the process is stuck?

See next thread.

mrtyler · 2018-08-30T03:10:00Z

+    sleep 30 && \
+    TF_VAR_locust_target_host=http://preferences.$TF_VAR_domain_name \
+    TF_VAR_locust_script=preferences.py \
+    xk up live/dev/locust',skip_infra,skip_secret_mgmt]"


Do we want to leave Locust running after the test completes? I think it would be better to clean up, especially since we're going to tear everything down before the next test run (which we should continue to do).

It turns out we don't need to destroy Locust module to reconfigure it. Terraform detects template changes and modifies Helm release if we provide TF_VARs with different values. I could add destroy_module command to CI pipeline, but module will be destroyed with the cluster anyway (it does not leave any leftover resources behind).

That's good news about reconfiguration.

Some responses / additional things this topic has made me think about:

I think some smoke tests -- meaning, at least for now, these :test_* tasks -- should be part of the regular rake / rake deploy workflow, as they are on the AWS side. They would provide rapid feedback about the health of a new cluster.

Failing that, I think these :test_* tasks should run as part of CI in stg and prd. Unless you do 1., can you add the relevant calls to the (disabled) gcp-stg and gcp-prd jobs?

I don't think we should leave Locust running in prd. I'm less concerned about leaving it running in stg, but we want stg to be as much like prd as possible. Depending on how you handle the above, how about adding destroy_module to at least the (disabled) gcp-prd job?

Nevermind the good news, there is a pretty good chance that Locust master configuration becomes messed up if we do not destroy module before.

I don't have strong opinion on this. If we include tests to rake / rake deploy workflow, it would add around 5 minutes of run time, which may be not very convenient for dev environment, if you just want to quickly spin everything up to debug your module. We can probably just copy tests from AWS side, and include them, but it will require wait_for code and maybe something else.

Added.

Agree, added.

I agree 5 minutes is a long time.

I think we can table this for now. gcp-* jobs will run a superset of smoke tests as part of CI. We can discuss other options later.

…sholds

mrtyler · 2018-08-30T23:35:50Z

LGTM!

natarajaya · 2018-08-31T00:19:55Z

    sleep 45 && \
    TF_VAR_locust_target_host=http://preferences.$TF_VAR_domain_name \
    TF_VAR_locust_script=preferences.py \
+    TF_VAR_locust_desired_max_response_time=4000 \


@seg10 @mrtyler This is related to GPII-3250.
Surprisingly, I did not see any failed requests in Locust stats, so instead of timing out, first request to Preferences just takes a lot of time to complete, affecting max_response_time stat. Once GPII-3250 issue is solved, this line can be removed. I'll add comment.

@natarajaya - based on my observations of the application on GCP, it looks like couchdb just needs some time to settle down upon initial deployment. When requests are made prematurely before the system is up and running, the preferences server takes a while to complete its request. Recommend that we initially put in a wait for some amount of time before running the smoketests.

Strike the above, this is more related to the indices for queries not being warmed up.

PR for the above now available at: gpii-ops/gpii-dataloader#7

stepanstipl · 2018-08-31T15:38:07Z

Conceptually I'm not quite happy about mixing the provisioning of the locust swarm and actually running the tests. I would think that provisioning should be one thing, idempotent, handled with TF. and actually running the perf. test another. This woudl probably prevent, if I understand correctly above code, things like teraing down the cluster every time before running the test or non-idempotent TF runs.

natarajaya · 2018-08-31T19:38:54Z

@stepanstipl Thanks for review!

As the title of this PR states – it is just a first attempt to introduce Locust into our infra. I agree that there is definitely room for improvement! Maybe, once common code is merged, we can have common "utility" multi-purpose cluster to run smoke tests from. This would've resolved issue from your previous comment and allowed us to have better test results, "unstained" by Locust activity, always benchmark from different zone, etc.

natarajaya added 3 commits August 18, 2018 16:40

Adding locust chart

d22d9b2

Adding basic GPII locust tasks

c5ee411

Adding locust modules

e8ab297

natarajaya self-assigned this Aug 19, 2018

natarajaya added 2 commits August 18, 2018 17:43

Various fixes

28e3df9

locust_workers should be configurable

8a8c966

amatas reviewed Aug 20, 2018

View reviewed changes

natarajaya added 6 commits August 20, 2018 14:58

Adding locust/tasks into .gitignore

eafdd07

Link tasks before applying chart, customizable locust scripts, check …

db27bb7

…for ready workers before starting the swarm

Use EXIT_STATUS

cc35eca

Merge branch 'master' of github.com:gpii-ops/gpii-infra into locust

919d4dd

Small fixes

cd8c89a

Remove tasks from locust chart before creating links

78b558a

mrtyler reviewed Aug 21, 2018

View reviewed changes

natarajaya added 2 commits August 20, 2018 21:54

Cleanup links after chart install, run swarm on every apply

2886528

Don't need quotes here

da38a0b

mrtyler reviewed Aug 22, 2018

View reviewed changes

natarajaya added 3 commits August 22, 2018 18:31

Merge branch 'master' of github.com:gpii-ops/gpii-infra into locust

fa5df5e

Merge branch 'master' of github.com:gpii-ops/gpii-infra into locust

a047c42

Various fixes

6f3e497

natarajaya added 5 commits August 22, 2018 22:00

Sleep more between retries

bcc20cd

Cosmetic tasks update

8f8253b

Use greater equal

602a11d

Do not use greater equal, use less than

9bc8d31

Tweak some variables

8f9f482

natarajaya added 3 commits August 28, 2018 11:27

Set TF_VAR_domain_name in vars.rb

05e108a

Use domain_name instead of parsing dns_zones in flowmanager,preferences

5d307a3

Various fixes

9040090

Add smoke tests to gcp-dev stage of CI config

a034305

mrtyler reviewed Aug 29, 2018

View reviewed changes

natarajaya added 6 commits August 29, 2018 16:21

Merge branch 'master' of github.com:gpii-ops/gpii-infra into locust

be12350

Use snake case for task naming

1f6d9ab

Rename var, exit if swarm_duration empty

9d2f465

Tweak some more vars

58fbbe1

Add test.rake

ae3af4f

Update .gitlab-ci.yml

2867b89

mrtyler reviewed Aug 30, 2018

View reviewed changes

natarajaya added 4 commits August 29, 2018 20:20

Bump flowmanager mrt

a422859

No need to destroy locust module

de6dd99

Destroy module and sleep before running tests, tweak flowmanager thre…

344b4fd

…sholds

Rearrange entries, add test steps to gcp-stg, gcp-prd

7e45220

Bumping preferences mrt to 4000 due to GPII-3250

b7bf75c

natarajaya commented Aug 31, 2018

View reviewed changes

Add comment

b63c9f6

stepanstipl reviewed Aug 31, 2018

View reviewed changes

Comment thread gcp/modules/locust/main.tf

stepanstipl reviewed Aug 31, 2018

View reviewed changes

Comment thread gcp/rakefiles/test.rake

Bump preferences mrt

9950012

natarajaya merged commit 8987350 into gpii-ops:master Aug 31, 2018

natarajaya deleted the locust branch September 10, 2018 17:04

mrtyler mentioned this pull request Apr 8, 2019

[GPII-3841]: Add display_universal_image_info task #344

Merged

Uh oh!

Conversation

natarajaya commented Aug 19, 2018

Uh oh!

natarajaya commented Aug 19, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrtyler left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

natarajaya Aug 23, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrtyler Aug 24, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrtyler Aug 25, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrtyler Aug 25, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stepanstipl commented Aug 22, 2018

Uh oh!

natarajaya commented Aug 23, 2018

Uh oh!

mrtyler commented Aug 25, 2018

Uh oh!

natarajaya commented Aug 28, 2018

Uh oh!

natarajaya commented Aug 28, 2018

Uh oh!

mrtyler left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

natarajaya Aug 23, 2018 •

edited

Loading

mrtyler Aug 24, 2018 •

edited

Loading

mrtyler Aug 25, 2018 •

edited

Loading

mrtyler Aug 25, 2018 •

edited

Loading

mrtyler left a comment •

edited

Loading

natarajaya Aug 30, 2018 •

edited

Loading

natarajaya Aug 30, 2018 •

edited

Loading