Document status	35 - Reviewed

Password Rotation

This document describes the manual password rotation pipelines used to maintain security compliance across all environments.

Overview

The solution implements a manual password rotation mechanism to ensure passwords can be rotated as needed for security and compliance requirements. The pipelines perform password rotation for both virtual machines and databases, updating all dependent systems automatically.

Execution Model

Password rotation is triggered manually through Azure DevOps during scheduled service windows.

Service Window Schedule

Password rotation is performed during regular service windows to minimize operational risk and ensure proper oversight. This schedule keeps passwords well within typical 90-day compliance requirements.

The service will be down for a short period (5 minutes) during password rotation as applications are restarted to pick up new credentials, hence it needs to be performed during a service window.

Pipeline Structure

Pipeline Name	Link
password-rotation-non-production.yml	Password Rotation Non Production
password-rotation-production.yml	Password Rotation Production

Production environments are in a separate pipeline to ensure explicit manual control and prevent accidental rotation.

Pipeline Stages

The password rotation pipeline executes the following stages in sequence for each environment:

1. Password Age Check (Informational)

The first stage checks the current age of all password secrets stored in Key Vault. This is informational only - rotation proceeds regardless of password age. The check examines the timestamps of the following secrets:

Secret Name	Purpose
vm-psql-adminpassword	PostgreSQL VM administrator password
vm-psql-db-adminpassword	PostgreSQL database administrator password
vm-psql-db-ecc-writerpassword	ECC database writer password
vm-psql-db-dmc-writerpassword	DMC database writer password
vm-psql-db-migration-writerpassword	Migration database writer password
vm-psql-db-readerpassword	Database reader password
vm-dataintegration-adminpassword	Data Integration VM administrator password

The keyvault secrets containing the connection strings that reference these passwords are also automatically updated.

Secret Name	Purpose
vm-psql-db-admin-connectionstring	PostgreSQL admin connection string
vm-psql-db-ecc-writer-connectionstring	ECC application connection string
vm-psql-db-dmc-writer-connectionstring	DMC application connection string
vm-psql-db-migration-writer-connectionstring	Migration application connection string

The oldest password timestamp is used to report the current password age. This information helps operations teams track when passwords were last rotated.

Pipeline Output Variables:

• daysSinceRotation: number of days since last rotation

2. VM Password Rotation

This stage generates new secure passwords for virtual machine administrator accounts and updates them in:

1. Azure Key Vault - The new passwords are stored as secrets
2. Virtual Machines - The VM admin passwords are updated via Azure CLI

This stage processes the following VMs:

• PSQL VM
• Data Integration VM (optional, based on configuration)
• Agent VM uses SSH keys and is not included in password rotation

3. Database Password Rotation

After VM passwords are rotated, database passwords are updated. This stage:

1. Generates new secure passwords for database users
2. Updates the passwords in PostgreSQL via SSH
3. Stores the new passwords in Key Vault

Note: This stage runs on a self-hosted agent that has network access to the database VM.

4. Update Connection Strings

Once passwords are rotated, this stage updates all database connection strings with the new credentials. The updated connection strings are stored in Key Vault and referenced by applications.

5. Restart Container Apps

This stage restarts the container apps to ensure the services are picking up the new credentials

1. The restart causes a brief downtime (approximately 5 minutes) as the applications restart and establish new connections to the database with updated credentials.

Important: This is why the password rotation must be performed during a scheduled service window to minimize impact on users.

6. Server Validation

The final stage validates that the PSQL server is healthy and accepting connections with the new credentials. The validation check will also use the new credentials in order to login to the server, ensuring the rotation is successful.

Pipeline Parameters

The following parameters are available when running the pipeline:

Parameter	Description	Default
rotateVmPasswords	Include VM passwords in rotation	true
rotateDbPasswords	Include database passwords in rotation	true

When to Rotate Passwords

• Security incident response - If credentials may have been compromised
• Before major releases - To ensure fresh credentials before deployment
• Periodic maintenance - Based on organizational security policy (e.g., every 90 days)
• Staff changes - When team members with password access leave the organization

Performing Password Rotations

Follow these steps to perform a password rotation during a service window.

Step 1: Pre-Rotation Checks

1. Verify you are within the scheduled service window
2. Notify stakeholders that password rotation is starting
3. Ensure you have access to Azure DevOps and the relevant pipelines

Step 2: Run Non-Production Pipeline

1. Navigate to Password Rotation Non Production
2. Click Run pipeline
3. Select the target branch (typically release/*)
4. Review parameters:
   • rotateVmPasswords: Leave as true (default)
   • rotateDbPasswords: Leave as true (default)
5. Click Run
6. Monitor pipeline execution and verify all stages complete successfully:
   • Check Rotation Status (informational)
   • Rotate VM Passwords
   • Rotate DB Passwords
   • Update Connection Strings
   • Post-Rotation (Container App restart)
   • Server Validation

Step 3: Validate Non-Production Environments

1. After the pipeline completes, verify applications are functioning:
   • The application will be down for a short period of time during the restart. Access the EnergyConnect application for each environment after restarts and confirm it is working as expected.

Step 4: Run Production Pipeline

1. Navigate to Password Rotation Production
2. Click Run pipeline
3. Select the target branch (typically release/*)
4. Review parameters (same as non-production)
5. Click Run
6. Monitor pipeline execution and verify all stages complete successfully

Step 5: Post-Rotation Validation

1. After the pipeline completes, verify applications are functioning:
   • The application will be down for a short period of time during the restart. Access the EnergyConnect application for each environment after restarts and confirm it is working as expected.
2. Server Validation stage will confirm the PSQL server is healthy and accepting connections with the new credentials.

Troubleshooting

If password rotation fails:

1. Verify VM is running

   • Check Azure Portal to ensure the VM is running and accessible (VMs can be stopped due to auto shutdown in lower level environments)

2. VM Password Rotation Failed

   • Check Azure CLI permissions on the service connection
   • Verify VM is running and accessible
   • Review pipeline logs for specific error messages

3. Database Password Rotation Failed

   • Ensure self-hosted agent has network access to PostgreSQL VM
   • Verify SSH connectivity from agent to database VM
   • Check that sshpass is installed on the agent

4. Connection String Update Failed

   • Verify Key Vault access permissions
   • Check that all required secrets exist

5. Container Apps Not Picking Up New Credentials

   • Manually restart Container Apps via Azure Portal
   • Verify Key Vault references in Container App configuration

Compliance

The bi-monthly service window schedule (approximately every 60 days) ensures passwords are rotated well within the common 90-day compliance requirement, providing a 30-day buffer.

Important: If a service window is missed, ensure password rotation is performed before passwords reach 90 days of age.