[karthikeyangs9]

Summary

Adds an opt-in COST_TAG_KEYS env var (e.g. Project,Environment) that emits a new aws.tag.<key> attribute on aws.cost.unblended / aws.cost.amortized, so Last9 dashboards can pivot on cost-allocation tags — matching the AWS Cost Explorer UI's Tag group-by.

Default behaviour is unchanged: when COST_TAG_KEYS is empty (the default), no extra CE calls are made and existing region-keyed series are byte-identical.

Why a separate CE call per tag key

CE's GetCostAndUsage enforces two structural limits we already absorbed once (commit 6e9c34c):

1. GroupBy ≤ 2 dimensions per call. Region + Tag + Service in one call would be a 3-dim request — exactly the ValidationException we hit before.
2. Only one TAG entry per GroupBy. Multi-tag pivots in a single call are illegal regardless of the 2-dim cap.

So tag rows live as a parallel series: existing call returns [SERVICE, REGION] rows, new call returns [SERVICE, TAG:<key>] rows. They share aws.service, aws.account.id, cost.date and are distinguished by aws.region vs aws.tag.<key>. No 3-dim GroupBy ever happens.

Metric stream	aws.region	aws.tag.<key>
Region rows (existing)	✅	❌
Tag rows (new, opt-in)	❌	✅

Cost impact

accounts × tag_keys extra CE calls per run, $0.01 each. Default 1 account × 3 keys → $0.03/day, ~$0.90/month.

Customer prerequisite

Tags must be activated in AWS Console → Billing → Cost Allocation Tags. Activation is not retroactive; pre-activation cost has no tag attribution and surfaces as aws.tag.<key>=\"untagged\". CE indexing latency after activation is ~24h.

Tag-key sanitization

AWS tag keys allow + - = . _ : / @ space; Prometheus label names disallow most. Sanitized to lowercase + [^a-z0-9_] → _ (e.g. cost-center → cost_center, aws:Project → aws_project).

Files

File	Change
main.py	New COST_TAG_KEYS parsing, _sanitize_tag_key, fetch_tag_costs, integration in poll/lambda_handler, conditional aws.tag.<key> attribute in send_otlp_metrics
deploy.sh	Propagate COST_TAG_KEYS env (Lambda API + CloudFormation paths)
cloudformation.yaml	New CostTagKeys parameter wired to env var
README.md	Documents prerequisites, dimensions, pricing
test_parse.py	4 new tests covering _sanitize_tag_key edge cases

Test plan

• python3 test_parse.py — 11 passed
• ruff format + ruff check — clean
• Smoke test fetch_tag_costs with a fake CE client (verified: GroupBy shape, Filter on linked-account, Project$alpha → tag_value=\"alpha\", empty Project$ → untagged, zero-cost rows skipped)
• Smoke test send_otlp_metrics with mixed region+tag rows — confirmed region rows get aws.region, tag rows get aws.tag.cost_center, no cross-pollination
• Live AWS verification: deploy to test account with COST_TAG_KEYS=Project, run Lambda, query aws_cost_unblended_USD{aws_tag_project!=\"\"} in Last9

[nishantmodak]

Completed the full six-file review at 045399aa4ad40f7514565f420f73e59d69086811 against base 65b6452e1e29385900403c65af6a000c9b7dc6b0, including collector callers, deployment configuration, all dashboard queries/variables, tests and CI. Two P2 defects need correction before merge: enabling tags inflates ordinary cost totals, and the new tag panels cannot display the documented Project/Environment keys. The inlines give the required changes.

Earlier in today’s 18:00 UTC run, isolated validation on Python 3.13.15 established: all 24 existing head tests and 7 base tests pass; five additional contract probes pass, including 100 seeded value/cost cases, partial-page failure handling, Lambda/poll parity and export failure propagation. Twenty native PromQL query/month contracts and both deployment-script branches pass with synthetic boundaries. The attached total assertion repeatedly returns $20 instead of $10 on head and passes on base; two tag keys return $30. The tag-panel assertion returns no data for Project/Environment; Name is a passing control. Disabled-tag payloads are byte-identical to base. All four current-head GitHub lint/test checks pass separately.

This continuation rechecked the entire diff and relevant contracts, refreshed discussions/CI, and verified all 50 head/base source files against their Git objects. No tests rerun in this continuation; the execution results above are retained evidence from the same exact source, with runtime/dependency provenance recorded in that run.

Limits: CE and OTLP boundaries are synthetic; deployment commands are stubbed. Promtool evaluates the actual dashboard expressions against the collector's exported data with dot-to-underscore label normalization and fixture-relative sample times; this is not live Last9 ingestion or browser/deployment verification. No infrastructure, container image, live AWS calls or customer data were used. Direct dependencies match requirements.txt; this repository has no transitive lockfile.

Copyable regression tests and exact commands for both findings

Save the following as aws-cost-explorer/test_review_regression.py. It uses the repository's plain Python test-function style and imports the actual main implementation. Install the pinned requirements in an isolated Python 3.13 environment and put Prometheus promtool 3.5.0 on PATH (or set PROMTOOL to its absolute path). No server is needed.

cd aws-cost-explorer
python3.13 -m venv .venv-review
. .venv-review/bin/activate
python -m pip install -r requirements.txt
python test_review_regression.py test_positive_controls
python test_review_regression.py test_total_is_invariant_when_tags_enabled
python test_review_regression.py test_documented_project_key_is_visible

On the inspected head: controls exit 0; each of the last two commands exits 1 with its assertion shown by promtool. Synthetic scenario: one account/service/day costs $10; CE returns the same $10 decomposed by region and by each requested tag. No random ordering or timing assumptions. The broad value probe used seed 180010, independently of these deterministic regressions. The same total test passes on base; the tag feature/panels are new, so Name and the disabled/exact-region paths are positive controls rather than a claimed base implementation.

"""Save beside main.py; plain test functions match test_parse.py's framework."""
import json, os, pathlib, subprocess, tempfile
from unittest.mock import patch
os.environ.setdefault('OTEL_EXPORTER_OTLP_ENDPOINT', 'http://localhost')
import main

def check_dashboard(tag_keys, panel_name, expected, region='.*'):
    class CE:
        def get_dimension_values(self, **kw):
            return {'DimensionValues': [{'Value': '111111111111'}]}
        def get_cost_and_usage(self, **kw):
            dim = kw['GroupBy'][1]
            key = dim['Key'] + '$alpha' if dim['Type'] == 'TAG' else 'us-east-1'
            return {'ResultsByTime': [{'TimePeriod': {'Start': '2026-09-17'},
                'Groups': [{'Keys': ['AmazonEC2', key], 'Metrics': {
                    k: {'Amount': '10', 'Unit': 'USD'}
                    for k in ['UnblendedCost', 'AmortizedCost']}}]}]}
    with patch.object(main, 'COST_TAG_KEYS', tag_keys, create=True), \
         patch.object(main.requests, 'post') as post:
        post.return_value.status_code = 200
        main.poll(CE())
    payload = post.call_args.kwargs['json']
    metric = next(m for m in payload['resourceMetrics'][0]['scopeMetrics'][0]['metrics']
                  if m['name'] == 'aws.cost.unblended')
    inputs = []
    for dp in metric['gauge']['dataPoints']:
        labels = {a['key'].replace('.', '_'): a['value']['stringValue']
                  for a in dp['attributes']}
        series = 'aws_cost_unblended_USD{' + ','.join(
            k + '=' + json.dumps(v) for k, v in sorted(labels.items())) + '}'
        inputs.append({'series': series, 'values': str(dp['asDouble'])})
    dashboard = json.loads(pathlib.Path('last9-dashboard.json').read_text())['dashboard']
    expr = next(p['queries'][0]['expr'] for p in dashboard['panels'] if p['name'] == panel_name)
    for var in ['account', 'service', 'tag']:
        expr = expr.replace('$' + var, '.*')
    expr = expr.replace('$region', region)
    if panel_name == 'Top 10 Tag Values (Loaded)':
        expr = 'sum(' + expr + ')'  # Assert the amount independently of the tag label's name.
    case = {'rule_files': [], 'evaluation_interval': '1m', 'tests': [{
        'interval': '1m', 'input_series': inputs, 'promql_expr_test': [{
            'expr': expr, 'eval_time': '1m', 'exp_samples': expected}]}]}
    with tempfile.TemporaryDirectory() as tmp:
        path = pathlib.Path(tmp) / 'case.json'
        path.write_text(json.dumps(case))  # JSON is valid YAML for promtool.
        result = subprocess.run([os.environ.get('PROMTOOL', 'promtool'), 'test', 'rules', str(path)],
                                capture_output=True, text=True, timeout=20)
    assert result.returncode == 0, result.stdout + result.stderr

def test_total_is_invariant_when_tags_enabled():
    panel = 'Total Spend (Loaded)' if hasattr(main, 'fetch_tag_costs') else 'Total Spend (last 7 days)'
    check_dashboard(['Project'], panel, [{'labels': '{}', 'value': 10}])

def test_documented_project_key_is_visible():
    check_dashboard(['Project'], 'Top 10 Tag Values (Loaded)',
                    [{'labels': '{}', 'value': 10}])

def test_positive_controls():
    panel = 'Total Spend (Loaded)' if hasattr(main, 'fetch_tag_costs') else 'Total Spend (last 7 days)'
    check_dashboard([], panel, [{'labels': '{}', 'value': 10}])
    check_dashboard(['Project'], panel, [{'labels': '{}', 'value': 10}], 'us-east-1')
    if hasattr(main, 'fetch_tag_costs'):
        check_dashboard(['Name'], 'Top 10 Tag Values (Loaded)',
                        [{'labels': '{}', 'value': 10}])

if __name__ == '__main__':
    import sys
    globals()[sys.argv[1]]()

[nishantmodak]

[P2] Keep tag decompositions out of existing cost totals

With COST_TAG_KEYS=Project, this combines a complete region decomposition and a complete tag decomposition under the same cost metrics. The shipped dashboard defaults $region to .*, which also matches series with no aws_region. Thus a $10 account/service/day displays $20 in Total Spend; Project,Environment displays $30. Service/account totals and averages inherit the same inflation. Removing aws.region from tag rows does not isolate them from those queries.

Keep ordinary totals on exactly one decomposition: give tag costs a separate metric family, or add a reliable stream discriminator and update all affected dashboard consumers. Preserve legitimate regionless baseline costs when excluding tag rows. Validate that enabling one or multiple tag keys leaves ordinary spend unchanged.

Reproduced on 045399aa4ad40f7514565f420f73e59d69086811 with actual poll/OTLP encoding and native promtool 3.5.0 executing the dashboard query. The consolidated review includes the complete runnable aws-cost-explorer/test_review_regression.py and setup. Run python test_review_regression.py test_total_is_invariant_when_tags_enabled: expected 10, actual 20; base 65b6452e1e29385900403c65af6a000c9b7dc6b0 passes. The disabled-tag and exact-region controls pass. Two-key probe returns 30. Synthetic CE/HTTP only; no live AWS/Last9 call.

Execution provenance: reproduced in today’s 18:00 UTC run; this continuation verified identical source and reused those recorded results, without rerunning tests.

[nishantmodak]

[P2] Bind tag panels to the configured tag key

The documented COST_TAG_KEYS=Project,Environment emits aws.tag.project and aws.tag.environment, which become aws_tag_project/aws_tag_environment. These new panels and the Tag variable instead require aws_tag_name, produced only when the AWS key is Name (or sanitizes to name). Normal documented configuration therefore gives an empty tag selector and empty tag charts; Untagged Spend % falls back to a misleading zero.

Make the selected tag key drive the label used by the tag variable, all three queries and legends, or use a consistent explicit key/value metric contract across the producer and dashboard. Include Project and Environment in the dashboard integration tests.

Reproduced on 045399aa4ad40f7514565f420f73e59d69086811: the consolidated review contains the full test/setup. Run python test_review_regression.py test_documented_project_key_is_visible from aws-cost-explorer: expected $10 from the actual exported Project=alpha row, actual empty vector. Environment fails equivalently; Name passes. This is new functionality absent from base; the Name control establishes the query/engine fixture works. No live backend or browser verification claimed.

Execution provenance: reproduced in today’s 18:00 UTC run; this continuation verified identical source and reused those recorded results, without rerunning tests.