"kube-dns" benchmark jobs are actually testing CoreDNS, not kube-dns

[danwinship]

The pull-perf-tests-benchmark-kube-dns and ci-benchmark-kube-dns-master jobs have names suggesting that they test the old kube-dns, but both actually run in clusters with CoreDNS. (In contrast, the SIG Network tests with "kube-dns" in their names actually do test kube-dns specifically, in contrast to the more-standard CoreDNS tests.)

Maybe just remove the kube- from the names? (I would have submitted a PR but I don't want to break dashboards, etc...)

(It seems that these jobs have always tested CoreDNS: the periodic job was created in #10366, a month after CoreDNS became the default for GCE CI clusters in kubernetes/kubernetes#69883.)

(Noticed as part of kubernetes/kubernetes#137556.)

/sig scalability
@kubernetes/sig-scalability

[aojea]

I do no think there are active mainteinrs for those jobs

• https://testgrid.k8s.io/sig-scalability-benchmarks#kube-dns
• https://testgrid.k8s.io/sig-scalability-benchmarks#node-local-dns

[danwinship]

meaning what? nobody cares and we should just delete them?

[aojea]

meaning what? nobody cares and we should just delete them?

I can not say, some teams internally fork this repo and keep running the test, so if we change anything that people will break most probably , a change on the name changes the dashboards ... if nobody uses them then we just run less tests

[Dipanshusinghh]

The actual mismatch here seems pretty clear — the jobs are named as kube-dns benchmarks even though they have effectively been exercising CoreDNS for a long time already-
At the same time, the concern about breaking downstream dashboards or internal consumers also makes sense. Before renaming or removing anything, it would probably help to first check where these jobs are currently referenced in TestGrid/configs and whether they are still actively depended on outside the main repo workflows.

[yunus25jmi1]

Findings from digging into this:

Confirming @danwinship's report, with some additional detail:

1. These jobs have never benchmarked kube-dns. Both jobs call run-e2e.sh kube-dns, which runs perf-tests/dns/run kube-dns → run_perf.py --use-cluster-dns. That flag means "benchmark the cluster's existing DNS service" — no DNS server is ever deployed by the test. Neither job sets CLUSTER_DNS_CORE_DNS=false (unlike SIG Network's genuine kube-dns jobs in sig-network-gce.yaml), so the clusters have run CoreDNS since it became the GCE default (Make CoreDNS default in kubeup and update CoreDNS version/manifest in kubeup and kubeadm kubernetes#69883, Oct 2018). The --use-cluster-dns mode was added to perf-tests in Nov 2018 and the periodic job in Dec 2018 — after the default flipped.

2. The parameter sweep in the results is fictitious. params/kubedns/default.yaml sweeps kubedns_cpu × dnsmasq_cpu × dnsmasq_cache (40 configurations), but runner.py skips all deployment changes when --use-cluster-dns is set. The job benchmarks the same untouched CoreDNS instance 40 times and labels the results with parameter values that were never applied. Data under perfDashPrefix: kube-dns benchmark is therefore mislabeled on two axes. The node-local-dns job has the same inert-params issue via --nodecache-ip.

3. The kube-dns and core-dns scenarios in run-e2e.sh are functionally identical — both pass --use-cluster-dns, differing only in which (partially ignored) params file they load.

Questions for SIG Scalability before any code changes:

• Does anyone own these jobs and consume their output (TestGrid sig-scalability-benchmarks#kube-dns, perf-dash "kube-dns benchmark")? If not, the simplest fix is deletion.
• If they're worth keeping, we propose this migration rather than an in-place rename:
  1. perf-tests: add an implementation-neutral cluster-dns scenario that (a) accepts only params that actually affect dnsperf, rejecting deployment-only ones, (b) detects and records the actual DNS implementation in result metadata, (c) covers the node-local-dns existing-server mode too. Legacy scenario names remain as deprecated aliases temporarily.
  2. test-infra: add ci-benchmark-cluster-dns-master / pull-perf-tests-benchmark-cluster-dns alongside the existing jobs; run both for ~one release cycle with an announced cutoff.
  3. Remove the legacy jobs and tabs, then drop the aliases and params/kubedns/ from perf-tests.

We would not fix this by setting CLUSTER_DNS_CORE_DNS=false — kube-dns is being deprecated (kubernetes/kubernetes#137556), so building CI for it now is backwards.

Happy to send the PRs once there's a keep/delete decision.

/cc @kubernetes/sig-scalability-leads