🤖 AI SRE Agent on GKE

AI-assisted Kubernetes incident investigation and remediation platform built on Google Kubernetes Engine (GKE) using FastAPI, Prometheus, Grafana, Google IAP, Kubernetes APIs, and policy-controlled remediation.

The project is designed to reduce repetitive Kubernetes troubleshooting work by collecting operational evidence, analyzing workload health, surfacing likely root causes, and recommending controlled remediation actions.

🎥 Demo

The demo covers the end-to-end flow:

Google IAP Authentication
        ↓
AI SRE Dashboard
        ↓
Platform Health
        ↓
Monitoring
        ↓
Incident Investigation
        ↓
Namespace / Deployment Selection
        ↓
Root Cause Analysis
        ↓
Recommended Remediation

🚀 What This Project Does

A normal Kubernetes incident often means switching between several tools:

Alert
  ↓
kubectl get pods
  ↓
kubectl describe pod
  ↓
kubectl logs
  ↓
kubectl get events
  ↓
Prometheus
  ↓
Grafana
  ↓
Manual Correlation
  ↓
Root Cause Analysis

This project brings those signals into one investigation workflow.

Example questions:

Why is payment-api crashing?
Show unhealthy pods
Check recent errors
Which containers are restarting frequently?

The platform gathers Kubernetes context and presents a structured investigation result.

✨ Features

Kubernetes workload investigation
Dynamic namespace discovery
Dynamic deployment discovery
Pod health analysis
Container restart analysis
Kubernetes event inspection
Prometheus integration
Grafana integration
Platform health dashboard
Incident investigation interface
Root-cause oriented output
Google Identity-Aware Proxy protection
Google Cloud IAM authorization
GKE Ingress
Google Managed HTTPS
Kubernetes RBAC
Human-in-the-loop remediation
Policy-controlled auto-healing
Sample failure workloads for testing

🏗️ Architecture

                           User
                            │
                            ▼
                   https://<YOUR_DOMAIN>
                            │
                            ▼
                Google HTTPS Load Balancer
                            │
                            ▼
                 Identity-Aware Proxy
                            │
                            ▼
                    IAM Authorization
                            │
                            ▼
                       GKE Ingress
                            │
                            ▼
                  ai-sre-agent Service
                            │
                            ▼
                     FastAPI Backend
                            │
              ┌─────────────┼─────────────┐
              │             │             │
              ▼             ▼             ▼
         Kubernetes      Prometheus      Grafana
              │             │             │
              └─────────────┼─────────────┘
                            │
                            ▼
                    Incident Analysis
                            │
                            ▼
                   Root Cause Analysis
                            │
                            ▼
                Recommended Remediation
                            │
                            ▼
                Policy-Controlled Action

🧰 Technology Stack

Component	Technology
Backend	FastAPI
Frontend	HTML, CSS, JavaScript
Kubernetes	Google Kubernetes Engine
Container Runtime	Docker
Monitoring	Prometheus
Visualization	Grafana
Authentication	Google Identity-Aware Proxy
Authorization	Google Cloud IAM
HTTPS	Google Managed Certificate
Load Balancing	GKE Ingress
Registry	Google Artifact Registry
Kubernetes Integration	Kubernetes Python Client
Cluster Security	Kubernetes RBAC

📁 Repository Structure

ai-sre-agent-gke/
├── app/
│   ├── __init__.py
│   ├── agent.py
│   ├── main.py
│   ├── remediation/
│   │   ├── __init__.py
│   │   ├── executor.py
│   │   ├── planner.py
│   │   ├── policies.py
│   │   └── verifier.py
│   ├── static/
│   │   ├── app.js
│   │   └── style.css
│   ├── templates/
│   │   ├── index.html
│   │   └── login.html
│   └── tools/
│       ├── __init__.py
│       └── kubernetes_tools.py
│
├── docs/
│   └── demo/
│       ├── ai-sre-demo.gif
│       ├── iap.png
│       ├── investigate.png
│       ├── login.png
│       ├── monitoring.png
│       ├── overview.png
│       ├── result-1.png
│       └── result-2.png
│
├── incident-app/
│   ├── crashloop-demo.yaml
│   ├── imagepull-demo.yaml
│   ├── payment-api-crash.yaml
│   └── pending-demo.yaml
│
├── k8s/
│   ├── ai-sre-agent-deployment.yaml
│   ├── ai-sre-agent-service.yaml
│   ├── ai-sre-backendconfig.yaml
│   ├── ai-sre-ingress.yaml
│   ├── managed-certificate.yaml
│   ├── rbac.yaml
│   └── serviceaccount.yaml
│
├── monitoring/
│   └── prometheus-values.yaml
│
├── .dockerignore
├── .gitignore
├── Dockerfile
├── LICENSE
├── README.md
└── requirements.txt

✅ Prerequisites

You will need:

Google Cloud project
GKE cluster
Artifact Registry repository
gcloud
kubectl
Docker
Helm
Python 3
DNS hostname
Google OAuth client for IAP
Prometheus and Grafana

Authenticate:

gcloud auth login

Set your project:

gcloud config set project <PROJECT_ID>

Verify cluster access:

kubectl get nodes

🐳 Build the Container Image

If you are building from an Apple Silicon Mac and your GKE nodes use AMD64:

docker buildx build \
  --platform linux/amd64 \
  -t <REGION>-docker.pkg.dev/<PROJECT_ID>/<REPOSITORY>/ai-sre-agent:v1 \
  --push .

☸️ Deploy to GKE

Apply the ServiceAccount:

kubectl apply -f k8s/serviceaccount.yaml

Apply RBAC:

kubectl apply -f k8s/rbac.yaml

Deploy the application:

kubectl apply -f k8s/ai-sre-agent-deployment.yaml

Apply the Service:

kubectl apply -f k8s/ai-sre-agent-service.yaml

Check status:

kubectl get pods -n ai-sre
kubectl get svc -n ai-sre

Restart when required:

kubectl rollout restart deployment ai-sre-agent -n ai-sre
kubectl rollout status deployment ai-sre-agent -n ai-sre

🔐 Kubernetes RBAC

The AI SRE Agent uses a dedicated Kubernetes ServiceAccount.

Verify the ServiceAccount used by the deployment:

kubectl get deployment ai-sre-agent \
  -n ai-sre \
  -o jsonpath='{.spec.template.spec.serviceAccountName}'
echo

Verify in-cluster Kubernetes access:

kubectl exec -n ai-sre deployment/ai-sre-agent -- \
  python -c '
from kubernetes import client, config
config.load_incluster_config()
v1 = client.CoreV1Api()
print([n.metadata.name for n in v1.list_namespace().items])
'

Typical read permissions include:

namespaces
pods
pods/log
events
services
endpoints
deployments
replicasets

Write permissions should be limited to explicitly approved remediation actions.

📊 Prometheus and Grafana

This project uses the open-source kube-prometheus-stack.

Add the Helm repository:

helm repo add prometheus-community \
  https://prometheus-community.github.io/helm-charts

Update:

helm repo update

Create the monitoring namespace:

kubectl create namespace monitoring

Install:

helm install monitoring \
  prometheus-community/kube-prometheus-stack \
  -n monitoring \
  -f monitoring/prometheus-values.yaml

Check services:

kubectl get svc -n monitoring

Example internal Prometheus endpoint:

http://monitoring-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090

🔎 Incident Investigation

The investigation flow allows an engineer to:

Select a namespace
Select a deployment
Enter an investigation question
Collect Kubernetes evidence
Review the generated analysis
Review remediation recommendations

Example questions:

Why is payment-api crashing?
Show unhealthy pods
Check recent errors

The workflow can inspect:

Pod state
Container state
Restart count
Readiness
Kubernetes events
Deployment state

🧪 Sample Incidents

Sample workloads are included under:

incident-app/

Examples:

crashloop-demo.yaml
imagepull-demo.yaml
payment-api-crash.yaml
pending-demo.yaml

Deploy one:

kubectl apply -f incident-app/payment-api-crash.yaml

Check:

kubectl get pods -n ai-sre

These manifests help reproduce common failure scenarios such as:

CrashLoopBackOff
ImagePullBackOff
Pending

🌐 GKE Ingress

Apply the Ingress:

kubectl apply -f k8s/ai-sre-ingress.yaml

Check:

kubectl describe ingress ai-sre-ingress -n ai-sre

Traffic path:

Internet
  ↓
Google HTTPS Load Balancer
  ↓
GKE Ingress
  ↓
ClusterIP Service
  ↓
FastAPI Pod

📍 Global Static IP

Create:

gcloud compute addresses create ai-sre-static-ip \
  --global

Retrieve:

gcloud compute addresses describe ai-sre-static-ip \
  --global \
  --format="get(address)"

Point your DNS record to this IP.

🔒 HTTPS with Google Managed Certificate

Example:

apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata:
  name: ai-sre-cert
  namespace: ai-sre
spec:
  domains:
    - <YOUR_DOMAIN>

Apply:

kubectl apply -f k8s/managed-certificate.yaml

Check:

kubectl describe managedcertificate ai-sre-cert -n ai-sre

Wait for:

CertificateStatus: Active

🛡️ Google Identity-Aware Proxy

The portal is protected using Google Cloud IAP.

Internet
  ↓
HTTPS Load Balancer
  ↓
Google IAP
  ↓
IAM Authorization
  ↓
GKE Backend

Example BackendConfig:

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: ai-sre-backendconfig
  namespace: ai-sre
spec:
  iap:
    enabled: true
    oauthclientCredentials:
      secretName: ai-sre-auth

Apply:

kubectl apply -f k8s/ai-sre-backendconfig.yaml

List GKE-created backend services:

gcloud compute backend-services list --global

🔑 OAuth Credentials

Do not commit OAuth credentials to GitHub.

Create or patch the Kubernetes Secret at deployment time:

kubectl patch secret ai-sre-auth \
  -n ai-sre \
  --type merge \
  -p='{
    "stringData": {
      "client_id": "<GOOGLE_CLIENT_ID>",
      "client_secret": "<GOOGLE_CLIENT_SECRET>"
    }
  }'

For IAP using a custom OAuth client, configure this callback under Authorized redirect URIs:

https://iap.googleapis.com/v1/oauth/clientIds/<CLIENT_ID>:handleRedirect

👥 Grant IAP Access

Grant access only to required users:

gcloud projects add-iam-policy-binding \
  <PROJECT_ID> \
  --member="user:<AUTHORIZED_USER>" \
  --role="roles/iap.httpsResourceAccessor"

🧠 Investigation Workflow

Detect
  ↓
Collect
  ↓
Correlate
  ↓
Reason
  ↓
Recommend
  ↓
Remediate
  ↓
Verify

Evidence can come from:

Kubernetes API
Prometheus
Grafana
Events
Logs
Container Status
Deployment State

🛠️ Controlled Auto-Healing

The project follows a human-in-the-loop remediation model.

Incident Detected
       ↓
Evidence Collected
       ↓
Root Cause Identified
       ↓
Risk Classification
       ↓
Remediation Proposed
       ↓
Human Approval
       ↓
Execute Action
       ↓
Verify Recovery

A low-risk action might be:

Rollout Restart

A configuration issue such as:

DATABASE_URL missing

should not be blindly restarted.

Instead, it should be classified as:

Configuration Fix Required
Auto-Heal Eligible: NO

🔐 Security

Never commit:

OAuth client secrets
API keys
DNS provider tokens
service-account keys
kubeconfig files
session secrets
private certificates
personal credentials

Prefer:

Kubernetes Secrets
Google Secret Manager
Workload Identity
Environment Variables

🧭 Roadmap

💡 Design Philosophy

The goal is not to give an AI unrestricted Kubernetes access.

The approach is:

AI for investigation
        +
Policies for control
        +
Humans for approval
        +
Automation for execution
        +
Observability for verification

The engineer remains in control.

⚠️ Disclaimer

This project is intended for learning, experimentation, and controlled environments.

Do not enable automated remediation in production without:

proper RBAC
policy controls
approval mechanisms
audit logging
rollback procedures
validation
testing

⭐ Support

If you find the project useful, consider starring the repository.

Navigation Menu

Repository navigation

About

Repository files navigation

🤖 AI SRE Agent on GKE

🎥 Demo

🚀 What This Project Does

✨ Features

🏗️ Architecture

🧰 Technology Stack

📁 Repository Structure

✅ Prerequisites

🐳 Build the Container Image

☸️ Deploy to GKE

🔐 Kubernetes RBAC

📊 Prometheus and Grafana

🔎 Incident Investigation

🧪 Sample Incidents

🌐 GKE Ingress

📍 Global Static IP

🔒 HTTPS with Google Managed Certificate

🛡️ Google Identity-Aware Proxy

🔑 OAuth Credentials

👥 Grant IAP Access

🧠 Investigation Workflow

🛠️ Controlled Auto-Healing

🔐 Security

🧭 Roadmap

💡 Design Philosophy

⚠️ Disclaimer

⭐ Support

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
app		app
docs/demo		docs/demo
incident-app		incident-app
k8s		k8s
monitoring		monitoring
.dockerignore		.dockerignore
.gitignore		.gitignore
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
requirements.txt		requirements.txt

Latest commit

History

Folders and files

Repository files navigation

🤖 AI SRE Agent on GKE

🎥 Demo

🚀 What This Project Does

✨ Features

🏗️ Architecture

🧰 Technology Stack

📁 Repository Structure

✅ Prerequisites

🐳 Build the Container Image

☸️ Deploy to GKE

🔐 Kubernetes RBAC

📊 Prometheus and Grafana

🔎 Incident Investigation

🧪 Sample Incidents

🌐 GKE Ingress

📍 Global Static IP

🔒 HTTPS with Google Managed Certificate

🛡️ Google Identity-Aware Proxy

🔑 OAuth Credentials

👥 Grant IAP Access

🧠 Investigation Workflow

🛠️ Controlled Auto-Healing

🔐 Security

🧭 Roadmap

💡 Design Philosophy

⚠️ Disclaimer

⭐ Support

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages