Exercises

Exercise 1: Horizontal Pod Autoscaler

In this exercise, you will create and configure a Horizontal Pod Autoscaler (HPA) to automatically scale a deployment based on CPU utilization.

Step 1: Create a Deployment

First, create a deployment that will be scaled by the HPA:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: web
        image: nginx:latest
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 256Mi

Apply the deployment:

oc apply -f web-deployment.yaml
oc get deployment web-deployment

Step 2: Create an HPA

Create a Horizontal Pod Autoscaler that scales the deployment based on CPU utilization:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-deployment
  minReplicas: 2
  maxReplicas: 4
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

Apply the HPA:

oc apply -f web-hpa.yaml
oc get hpa web-hpa

Step 3: Monitor HPA Status

Monitor the HPA to see its current status and scaling decisions:

# View HPA status
oc get hpa web-hpa

# Get detailed HPA information
oc describe hpa web-hpa

# Watch HPA and deployment replicas
oc get hpa web-hpa -w
oc get deployment web-deployment -w

Observe the current replica count and target metrics.

Step 4: Generate Load

Generate CPU load on the pods to trigger scaling. In a separate terminal, run:

# Get a pod name
POD_NAME=$(oc get pods -l app=web -o jsonpath='{.items[0].metadata.name}')

# Generate CPU load (this will run in the background)
oc exec -it $POD_NAME -- sh -c "while true; do dd if=/dev/zero of=/dev/null; done" &

Monitor the HPA and deployment to observe scaling behavior:

# Watch HPA scaling
oc get hpa web-hpa -w

# Check deployment replica count
oc get deployment web-deployment

Step 5: Observe Scaling Behavior

After a few minutes, observe how the HPA responds:

# View HPA events
oc describe hpa web-hpa

# Check current replica count
oc get pods -l app=web

# View HPA metrics
oc get hpa web-hpa -o yaml

Note how the replica count changes based on CPU utilization.

Step 6: Stop Load and Observe Scale-Down

Stop the load generation and observe the scale-down behavior:

# Stop the load generation process
oc exec -it $POD_NAME -- pkill dd

# Monitor scale-down
oc get hpa web-hpa -w
oc get deployment web-deployment -w

Observe the stabilization window and scale-down policies in action.

Step 7: Clean Up

Remove the test resources:

oc delete hpa web-hpa
oc delete deployment web-deployment

Exercise 2: Vertical Pod Autoscaler

In this exercise, you will create and configure a Vertical Pod Autoscaler (VPA) to automatically adjust CPU and memory requests and limits for containers based on historical usage.

Step 1: Create a Deployment

Create a deployment that will be managed by the VPA:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vpa-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: vpa-app
  template:
    metadata:
      labels:
        app: vpa-app
    spec:
      containers:
      - name: app
        image: nginx:latest
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 256Mi

Apply the deployment:

oc apply -f vpa-deployment.yaml
oc get deployment vpa-deployment
oc get pods -l app=vpa-app

Step 2: Create a VPA in Recommendation Mode

Creating a VPA requires administrator privileges because VPA is a cluster-scoped resource, not namespace-scoped. Please ask your instructor to create the VPA for you.

Create a VPA in "Off" mode to view recommendations without automatic updates:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: vpa-recommender
  do-not-deploy: "true"
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vpa-deployment
  updatePolicy:
    updateMode: "Off"
  resourcePolicy:
    containerPolicies:
    - containerName: app
      minAllowed:
        cpu: 50m
        memory: 64Mi
      maxAllowed:
        cpu: 2
        memory: 4Gi
      controlledResources: ["cpu", "memory"]

The instructor will apply the VPA. After it’s created, you can view it:

# View the VPA (instructor will create it)
oc get vpa vpa-recommender

Step 3: View VPA Recommendations

After the VPA has collected some usage data, view its recommendations:

# View VPA status
oc get vpa vpa-recommender -o yaml

# Get detailed VPA information
oc describe vpa vpa-recommender

The VPA will show recommendations including:
* Target: Recommended values for optimal performance
* Lower Bound: Minimum values to maintain performance
* Upper Bound: Maximum values observed

Step 4: Generate Load and Monitor

Generate some load on the pods to provide usage data for VPA analysis:

# Get a pod name
POD_NAME=$(oc get pods -l app=vpa-app -o jsonpath='{.items[0].metadata.name}')

# Generate some CPU load
oc exec -it $POD_NAME -- sh -c "for i in {1..100}; do dd if=/dev/zero of=/dev/null bs=1M count=10; done"

Wait a few minutes, then check the VPA recommendations again:

oc describe vpa vpa-recommender

Observe how the recommendations may change based on actual usage patterns.

Step 5: Create VPA in Initial Mode

This step also requires administrator privileges. Please ask your instructor to create this VPA.

Create a new VPA in "Initial" mode that sets resource requests only when pods are first created:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: vpa-initial
  do-not-deploy: "true"
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vpa-deployment
  updatePolicy:
    updateMode: "Initial"
  resourcePolicy:
    containerPolicies:
    - containerName: app
      minAllowed:
        cpu: 50m
        memory: 64Mi
      maxAllowed:
        cpu: 2
        memory: 4Gi
      controlledResources: ["cpu", "memory"]

# The instructor will delete the previous VPA and create the new one
# You can delete and recreate the deployment to see Initial mode in action
oc delete deployment vpa-deployment
oc apply -f vpa-deployment.yaml

Check the new pods to see if VPA has set resource requests:

oc get pods -l app=vpa-app -o yaml | grep -A 10 resources
oc describe pod -l app=vpa-app

Step 6: Compare Resource Settings

Compare the original resource settings with VPA recommendations:

# View VPA recommendations
oc get vpa vpa-initial -o yaml | grep -A 20 recommendation

# View actual pod resource settings
oc get pod -l app=vpa-app -o jsonpath='{.items[0].spec.containers[0].resources}'

Step 7: Clean Up

Remove the test resources. Note that VPA deletion requires administrator privileges:

# Ask instructor to delete the VPA
# oc delete vpa vpa-initial

# You can delete the deployment
oc delete deployment vpa-deployment

Exercise 3: Descheduler Operator

In this exercise, you will examine the Descheduler Operator configuration and understand how descheduling profiles work. Note that creating or modifying the Descheduler requires administrator privileges.

Step 1: View Descheduler Configuration

Check if the Descheduler Operator is installed and view its configuration:

# View Descheduler custom resources (requires admin privileges to create, but you can view)
oc get kubedescheduler -n openshift-kube-descheduler-operator

# View the cluster Descheduler configuration
oc get kubedescheduler cluster -n openshift-kube-descheduler-operator -o yaml

If the Descheduler is configured, you’ll see the active profiles and settings.

Step 2: Examine Descheduler Profiles

View the profiles that are enabled in the Descheduler configuration:

# View enabled profiles
oc get kubedescheduler cluster -n openshift-kube-descheduler-operator -o jsonpath='{.spec.profiles}'

# View full Descheduler spec
oc get kubedescheduler cluster -n openshift-kube-descheduler-operator -o jsonpath='{.spec}' | jq .

Note which profiles are active. The available profiles include:
* AffinityAndTaints
* TopologyAndDuplicates
* SoftTopologyAndDuplicates
* LifecycleAndUtilization
* LongLifecycle
* CompactAndScale
* KubeVirtRelieveAndMigrate
* EvictPodsWithPVC
* EvictPodsWithLocalStorage

Step 3: View Descheduler Settings

Examine the Descheduler configuration settings:

# View descheduling interval
oc get kubedescheduler cluster -n openshift-kube-descheduler-operator -o jsonpath='{.spec.deschedulingIntervalSeconds}'

# View descheduler mode (Predictive or Automatic)
oc get kubedescheduler cluster -n openshift-kube-descheduler-operator -o jsonpath='{.spec.mode}'

# View eviction limits
oc get kubedescheduler cluster -n openshift-kube-descheduler-operator -o jsonpath='{.spec.evictionLimits}'

Step 4: View Descheduler Operator Status

Check the status of the Descheduler Operator:

# View Descheduler Operator deployment
oc get deployment -n openshift-kube-descheduler-operator

# View Descheduler pods
oc get pods -n openshift-kube-descheduler-operator

# View Descheduler Operator logs (if accessible)
oc logs -n openshift-kube-descheduler-operator -l app=descheduler-operator --tail=50

Step 5: Understand Descheduler Behavior

Create a deployment to observe how descheduling might affect it:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: descheduler-test
spec:
  replicas: 3
  selector:
    matchLabels:
      app: descheduler-test
  template:
    metadata:
      labels:
        app: descheduler-test
    spec:
      containers:
      - name: app
        image: nginx:latest
        resources:
          requests:
            cpu: 100m
            memory: 128Mi

oc apply -f descheduler-test.yaml
oc get pods -l app=descheduler-test -o wide

Monitor the pods to see their distribution across nodes. If the Descheduler is active with certain profiles, it may move pods to optimize cluster balance.

Step 6: Check for Descheduling Events

Look for descheduling-related events:

# View events related to descheduling
oc get events --all-namespaces --field-selector reason=Descheduled

# View recent events for your test deployment
oc get events --field-selector involvedObject.name=descheduler-test

# Watch pod movements
oc get pods -l app=descheduler-test -o wide -w

Note: If the Descheduler is in Predictive mode, it will simulate evictions but not actually move pods. Only Automatic mode performs actual evictions.

Step 7: Example Descheduler Configuration

Creating or modifying the Descheduler requires administrator privileges. This example shows what a Descheduler configuration might look like:

apiVersion: operator.openshift.io/v1
kind: KubeDescheduler
metadata:
  name: cluster
  namespace: openshift-kube-descheduler-operator
  do-not-deploy: "true"
spec:
  managementState: Managed
  deschedulingIntervalSeconds: 3600
  profiles:
  - AffinityAndTaints
  - TopologyAndDuplicates
  mode: Automatic
  evictionLimits:
    total: 10
    node: 5

This configuration would:
* Run descheduling every hour (3600 seconds)
* Use the AffinityAndTaints and TopologyAndDuplicates profiles
* Actually evict pods (Automatic mode)
* Limit evictions to 10 total and 5 per node per run

Step 8: Clean Up

Remove the test deployment:

oc delete deployment descheduler-test