Skip to content

Add MySQL DBI integration tests#717

Draft
Klarsen04 wants to merge 11 commits into
dbi-mysqlfrom
feature/mysql-dbi-integration-tests
Draft

Add MySQL DBI integration tests#717
Klarsen04 wants to merge 11 commits into
dbi-mysqlfrom
feature/mysql-dbi-integration-tests

Conversation

@Klarsen04

@Klarsen04 Klarsen04 commented Jun 30, 2026

Copy link
Copy Markdown

Summary

Add integration tests for MySQL Database Insights (DBI) pipeline, mirroring the existing PostgreSQL DBI tests.

Changes

  • Test runner (database_insights_mysql_test.go): Validates MySQL counter metrics, DB Load metrics (7 active session dimensions), Top SQL metrics, server-logs and raw-events log groups, process metrics for mysqld, and top_query event attributes.
  • Agent config (database_insights_mysql_config.json): Minimal config with opentelemetry.collect.database_insights.mysql targeting localhost:3306.
  • Setup script (database_insights_mysql_setup.sh): Cross-distro MySQL installation (yum/apt/zypper), performance_schema configuration, monitoring user creation, and passfile setup.
  • Test case generator: Registers the MySQL DBI test with same OS exclusions as PostgreSQL DBI.

Testing

  • Follows same patterns as the PostgreSQL DBI integration test
  • Uses sysbench (with mysqlslap fallback) for workload generation
  • Validates metrics via PromQL with resource attribute labels (db.system.name=mysql, db.instance.name=dbi-mysql-integ-test)
  • Conservative metric selection: only validates metrics that reliably appear in a 5-minute test window

@Klarsen04
Klarsen04 requested a review from a team as a code owner June 30, 2026 18:40
@Klarsen04
Klarsen04 force-pushed the feature/mysql-dbi-integration-tests branch from 5d4cf96 to 5a1f0cc Compare June 30, 2026 18:46
@Klarsen04
Klarsen04 marked this pull request as draft July 1, 2026 16:11
@Klarsen04
Klarsen04 force-pushed the feature/mysql-dbi-integration-tests branch 12 times, most recently from e300e27 to caea3e1 Compare July 10, 2026 14:57
Klarsen04 added 2 commits July 10, 2026 14:59
@Klarsen04
Add MySQL DBI integration tests
363a8b9 
Implement end-to-end integration tests for MySQL database insights,
mirroring the existing PostgreSQL DBI test in test/otel_collect/database_insights.

- Setup script installs MySQL across yum/apt/zypper, enables
  performance_schema with statement/wait consumers, configures the error
  log, and creates a cw_monitor user (SELECT on performance_schema,
  PROCESS, REPLICATION CLIENT) plus a sysbench workload user
- Agent config targets localhost:3306 with a pgpass-style passfile
- Workload generation via sysbench oltp_read_write (mysqlslap fallback)
- Validates counter metrics (enabled-by-default receiver metrics only),
  the 7 DB Load metrics, and Top SQL metrics via PromQL with resource
  attribute labels (db.system.name=mysql, db.instance.name)
- Validates mysqld process metrics from the host metrics process scraper
- Validates server logs and raw events in CloudWatch Logs, including
  db.server.top_query event attributes
- Register in test case generator with ol8/ubuntu-25 exclusions and
  exclude from ARM64 itar/china partitions (same as PostgreSQL DBI tests)
@Klarsen04
Merge remote-tracking branch 'origin/main' into feature/mysql-dbi-int…
233d6ec 
…egration-tests
@Klarsen04
Klarsen04 force-pushed the feature/mysql-dbi-integration-tests branch from caea3e1 to 233d6ec Compare July 10, 2026 14:59
sky333999 and others added 9 commits July 13, 2026 14:09
@sky333999
Standardize EFA test cluster names to cwagent-eks-integ- prefix (#721)
985d7ad 
The efa and otel-multi-efa EKS tests used divergent cluster names
(integ-* and cwagent-mefa-*) while every other integ/e2e EKS test uses
the shared cwagent-eks-integ- prefix. Align both so the cluster-cleaner
allowlist covers them without EFA-specific prefixes.
@sky333999
Use unique SSM parameter names and fix command-completion helper in s…
c2caac0 
…sm_document test (#722)

TestSSMDocument is chronically flaky (fails on nearly every "Test
Artifacts" run on main). Two test-side races fixed:

1. Shared parameter names. Every job in the matrix wrote the same
   SSM parameter names (agentConfig1/agentConfig2) in one account+region,
   so concurrent jobs' cleanup deleted parameters out from under each
   other -> the agent's fetch-config GetParameter (and the test's own
   DeleteParameter) hit ParameterNotFound. Suffix both names with the
   per-run uuid8 already used for the document name, in the unix and
   windows validators. Register each parameter's cleanup defer right
   after its Put (via a shared cleanupSSMParameter helper) so an early
   failure can't orphan a now-uniquely-named parameter.

2. WaitForCommandCompletion masked real failures. It only returned on
   Success and otherwise looped a fixed 60s, so a command that reached
   Failed/Cancelled/TimedOut in ~1s was reported as the misleading
   "commands did not complete within 1 minute". Return immediately on
   terminal non-Success states (surfacing StatusDetails), make the
   timeout configurable (default 2m), and add jitter to polling. The
   signature stays backward-compatible (variadic timeout).
@sky333999
test(ecs_sd): retry log-event validation to de-flake service discover…
4a53efa 
…y scenarios (#723)

ValidateLogGroupFormat retried only on log-group existence, then validated
log events exactly once. Only the first scenario (dockerLabel) survived
reliably -- it waits out log-group creation, incidentally giving the pipeline
~100-120s. Scenarios reusing the existing group (or checking a stream new to
that scenario) got only the flat 60s wait, below the ~125s pipeline minimum
(sd_frequency 1m + scrape_interval 1m + flush + ingestion), so whichever
scenario's SD-tick/scrape alignment fell outside the window failed with
"no log events" -- a race whose victim rotated run to run.

Retry ValidateLogsContent when the stream exists but is still empty, spanning
multiple SD cycles; fail fast on genuine validation errors. Add a typed
ErrNoLogEvents sentinel to AssertLogsNotEmpty so the retry matches on identity
rather than a substring.
@sky333999
fix(test): OTel v0.150 integ compatibility (SDKv2, kubeletstats, jour…
13e9d38 
…nald) (#719)

* fix(journald): re-trigger unit-test service after agent start

The journald units subtest asserts on entries a provisioner-run oneshot
service emits ~9 min before the agent starts. Contrib v0.149.0 made the stanza
journald input pass --lines=0 for the default start_at: end, dropping the
backlog that v0.124's implicit --lines=10 incidentally surfaced -- so on al2023
(newer journalctl) the journald-units stream is never created and GetLogEvents
fails with ResourceNotFoundException.

Re-trigger cwagent-unit-test.service after the agent initializes so fresh
entries land while the receiver is following, matching the priority/regex
subtests. Keeps start_at: end (the correct default; avoids backfilling the
whole journal on install).

* fix(test): align assume_role and credential_chain assertions with aws-sdk-go-v2

Two integ checks asserted on aws-sdk-go v1 artifacts that changed under the SDK
v2 migration. Credentials resolve correctly in both cases (metrics publish, the
access-key-id matches), so these are test-side, not agent regressions.

- assume_role confused_deputy_headers: the log scan keyed on v1 wire-debug
  markers (---[ REQUEST POST-SIGN ]---). Under v2 the STS client logs via
  smithy-go's RequestResponseLogger, which emits no such markers. Parse both
  formats so the X-Amz-Source-Account / X-Amz-Source-Arn headers are still
  detected (with a unit test covering both).
- credential_chain CommonConfigTest: expected provider name
  SharedCredentialsProvider (v1). Under v2 shared credentials resolve via
  config.LoadSharedConfigProfile and report SharedConfigCredentials. Update the
  expectation to match, consistent with HomeEnvTest and the fork's provider.

* fix(test): use example account/instance ids in assume_role scan fixtures

* fix(test): align app_signals service-events assertions with v0.150 log format

Two app_signals_service_events assertions broke on brittle log-string matching,
not agent behavior (export + delivery are verified by the sibling subtests):

- service_events_sent_to_otlp_monitoring_endpoint: collector-core v0.125.0
  injects otelcol.component.id/kind fields between the exporter's "Preparing to
  make HTTP request" msg and its "url", breaking the contiguous-substring
  match. Use a same-line regexp instead.
- TestAppSignalsOnPremCredentialsStartup: drop the assertion on "no EC2 IMDS
  role found" -- an aws-sdk-go-v2-specific string emitted by entitystore's
  benign best-effort IMDS probe (v1 wording differed). sigv4auth resolving from
  the credentials file is still covered by the surviving assertion + E2E
  delivery with IMDS disabled.

* fix(test): rename kubeletstats *.cpu.utilization to *.cpu.usage for v0.150

kubeletstatsreceiver deprecated the *.cpu.utilization metrics (default-off from
contrib v0.125.0) and removed k8s.node.cpu.utilization, k8s.pod.cpu.utilization
and container.cpu.utilization by v0.136.0, replaced by *.cpu.usage (unit {cpu}).
The bump to v0.150 crosses both. The deployed container-insights kubeletstats
config already emits *.cpu.usage, so update the EKS otel test expectations
(standard, attr_limit) to match, and refresh the stale entries in the
gpu/neuron/efa metric tables (those suites don't assert them today but
shouldn't carry deleted metric names).

* fix(test): gate otel/standard KSM assertions on node-metadata enrichment warmup

The kube_node_* host attributes (host.id / host.image.id / cloud.availability_zone)
are added by the agent's Lease-based enrichment, which is eventually consistent
(cold for the first few minutes after start, then warm). The suite's QueryCache
queries each metric once and caches it, so a single early cold read poisoned every
TestKSM_NodeBucket_* assertion. Add a TestMain readiness gate that polls kube_node_info
(non-cached) until all nodes are enriched before any cached query is issued. Best-effort
(logs + proceeds on timeout); assertions unchanged.

* fix(test): add validator retry loop to liscsi (match efa)

liscsi validated with a single one-shot metric fetch and no retry, so a slow
node_diskio_instance_store_* propagation window (e.g. ec2_instance_performance_exceeded_*)
caused hard failures (353 vs 1076 metrics on the same commit). Wrap go test in the same
3x/60s retry loop efa uses; required metrics unchanged.
@sky333999
fix(efa): use full IAM role names to avoid 38-char name_prefix cap (#726
71a725b 
)
@sky333999
fix(credential_chain): accept both SDK v1 and v2 provider names in Co…
5523271 
…mmonConfigTest (#728)

#719 updated the expected provider name to the aws-sdk-go-v2 value
(SharedConfigCredentials), but agent main still uses the SDK v1 chain, which
logs SharedCredentialsProvider — so credential_chain fails on every agent PR.
Accept either name so the test passes on both sides of the SDK v2 migration.
@Klarsen04
test(dbi): assert full MySQL metric set in DBI integration test
32f71ed 
Add assertions for the DBI-enabled console metrics (mysql.connection.count,
mysql.connection.errors, mysql.max_used_connections, mysql.commands,
mysql.query.count, mysql.table.size), the new InnoDB metrics
(mysql.deadlocks, mysql.history_list_length, mysql.active_transactions),
the remaining Top SQL digest sums (sum_rows_affected, sum_select_range_check,
sum_sort_merge_passes, sum_sort_range), and process.threads.
@Klarsen04
test(dbi): assert process.threads in PostgreSQL DBI integration test
fde1bd3 
process.threads is emitted by the shared DBI hostmetrics process scraper
for both engines; assert it in the postgres test for parity with mysql.
@Klarsen04
Merge remote-tracking branch 'origin/main' into feature/mysql-dbi-int…
822ed1a 
…egration-tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Klarsen04 @sky333999