# WHP CLAUDE DEVELOPMENT DOCUMENTATION

## ⚠️ CRITICAL COMPLIANCE REQUIREMENTS - READ FIRST ⚠️

```
████████████████████████████████████████████████████████████████████
█                                                                  █
█  MANDATORY WORKFLOWS - FAILURE TO FOLLOW WILL BREAK THE SYSTEM  █
█                                                                  █
████████████████████████████████████████████████████████████████████
```

### 🚨 ABSOLUTE REQUIREMENTS BEFORE ANY CODE COMMIT 🚨

**YOU MUST FOLLOW THIS EXACT SEQUENCE FOR EVERY CODE CHANGE:**

1. **ALWAYS** pull latest changes first: `git pull`
2. **FOR UI/UX CHANGES**: Ask @ui-ux-design-specialist to review ALL interface modifications
3. **FOR ALL CHANGES**: Ask @agent-security-auditor to audit ONLY the edited files
4. **FOR ALL CHANGES**: Ask @agent-software-documentation-expert to review documentation needs
5. **ONLY AFTER** all relevant agents approve: commit and push changes

**VIOLATION OF THIS WORKFLOW IS PROHIBITED**

### 🚨 CRITICAL LOCAL DEVELOPMENT ENVIRONMENT LIMITATIONS 🚨

```
████████████████████████████████████████████████████████████████████
█                                                                  █
█  WARNING: LOCAL DEVELOPMENT SYSTEM HAS NO SERVICES INSTALLED    █
█                                                                  █
█  FOR ALL VALIDATION AND TESTING: SSH INTO TESTING SERVER        █
█                                                                  █
█  THIS IS A MANDATORY REQUIREMENT - NO EXCEPTIONS                █
█                                                                  █
████████████████████████████████████████████████████████████████████
```

**CRITICAL TESTING AND VALIDATION REQUIREMENTS:**

- **LOCAL SYSTEM LIMITATION**: The local development environment `/home/jknapp/code/whp/` does NOT have any services installed
- **MANDATORY TESTING METHOD**: ALL validation and testing MUST be performed by SSH'ing into the testing server
- **DATABASE ACCESS**: The testing server has a `.my.cnf` file that defines the host, user, and password for database access
- **NO LOCAL EXECUTION**: NEVER attempt to run services, databases, or validation scripts locally
- **TESTING SERVER REQUIRED**: ALL functional testing, database operations, and service validation must occur on the testing server

**WHEN TO USE TESTING SERVER:**
- Database connectivity testing
- Service validation and health checks
- WordPress installations and management
- Container operations and Docker commands
- Email system testing
- DNS operations and validation
- SSL certificate operations
- Backup and restore testing
- Any operation requiring active services

**THIS IS A MANDATORY REQUIREMENT THAT CANNOT BE BYPASSED**

### 🚨 MANDATORY STAGING/RELEASE WORKFLOW 🚨

**ALL DEVELOPMENT MUST USE STAGING DIRECTORIES:**

- **SQL Migrations**: MUST be placed in `sql/migrations/staging/` during development
- **Shell Scripts**: MUST be placed in `scripts/updates/staging/` during development
- **Release Process**: ONLY `build-release.sh` moves files from staging to versioned directories
- **NEVER** directly create files in versioned directories (e.g., `2025.08.1/`)

**BYPASSING STAGING IS STRICTLY FORBIDDEN**

### 🚨 SHELL SCRIPT SECURITY REQUIREMENTS 🚨

**ALL SHELL SCRIPTS MUST:**

- Use EXACT shebang: `#!/usr/bin/env bash`
- Undergo security validation before execution
- Include proper error handling and exit codes
- Use absolute paths only
- Be placed in `scripts/updates/staging/` during development

**SCRIPTS NOT MEETING THESE REQUIREMENTS WILL BE REJECTED**

---

## 📋 QUICK REFERENCE - MANDATORY WORKFLOWS

### 🚨 TESTING ENVIRONMENT CHECKLIST

```
□ NEVER run services locally - local system has NO services installed
□ SSH into testing server for ALL validation and testing
□ Use testing server's .my.cnf for database access
□ ALL functional testing MUST occur on testing server
□ NO LOCAL EXECUTION of services or validation scripts
```

### Pre-Commit Checklist

```
□ git pull completed
□ @ui-ux-design-specialist reviewed (if UI changes)
□ @agent-security-auditor audited edited files
□ @agent-software-documentation-expert reviewed documentation
□ All agents approved changes
□ Only then: git commit and push
```

### Development File Placement

```
□ SQL migrations → sql/migrations/staging/
□ Shell scripts → scripts/updates/staging/
□ Web files → web-files/ (direct editing allowed)
□ Never create versioned directories manually
```

### Shell Script Validation

```
□ Shebang: #!/usr/bin/env bash
□ Error handling implemented
□ Absolute paths used
□ Security requirements met
□ Placed in staging directory
```

---

## 📝 SETUP.SH VERIFICATION HISTORY

### October 21, 2025 - Comprehensive 75-Commit Review
**Reviewer:** Claude AI Agent
**Commits Reviewed:** Last 75 commits (b888fb0 to 06d40e8)
**Status:** ✅ VERIFIED READY FOR PRODUCTION

**Review Scope:**
- Commits 1-25: Recent features (health check API, SSL refresh, backup upload worker)
- Commits 26-50: Backup system architecture (multipart uploads, decoupled processing)
- Commits 51-75: Critical fixes (database security, concurrency control, status fixes)

**Changes Made to setup.sh:**
1. Added SSL certificate refresh cron job (daily at 3 AM)
   - Runs `/root/whp/scripts/refresh-system-ssl.php`
   - Keeps httpd/proftpd certificates synchronized with HAProxy
2. Updated final instructions with SSL refresh and health check documentation

**Verification Results:**
- ✅ All 7 cron jobs properly configured
- ✅ All 58 database tables present in complete-database-setup.sql
- ✅ Database security: MySQL/PostgreSQL bound to 127.0.0.1 only
- ✅ Backup system complete (4-worker architecture with multipart uploads)
- ✅ Health check API configured with configurable thresholds
- ✅ SSL certificate management automated
- ✅ WordPress management system complete (including cloning)
- ✅ Terminal access system configured
- ✅ Dual database support (MySQL + PostgreSQL)
- ✅ Security monitoring and IP blocking operational
- ✅ API system comprehensive with rate limiting

**Critical Features Verified:**
- Decoupled backup architecture (creation + upload workers)
- S3 multipart upload support for large files (>100MB)
- Backup worker concurrency control (PID-based locking)
- WordPress update queue system
- Delegated user access system
- Terminal access logging and preferences
- Health check monitoring with HAProxy container metrics

**Database Migrations Verified:**
All migrations from 2025.07.1 through 2025.10.1 verified present in complete-database-setup.sql, including:
- multipart_uploads, backup_upload_queue, backup_local_storage
- wp_update_queue, delegated_users
- terminal_access_log, terminal_preferences
- All backup_history enhancements (claimed status, upload tracking)

**Conclusion:**
setup.sh is production-ready for fresh installations. All recent features, security fixes, and database migrations properly integrated. No additional changes required.

---

## PROJECT OVERVIEW

WHP is a modern, containerized web hosting control panel designed for hosting providers and enterprises. It provides comprehensive hosting management capabilities including:

- Multi-user web hosting with Docker containerization
- WordPress management with auto-updates and security hardening
- Database management (MySQL/MariaDB and PostgreSQL)
- DNS management with PowerDNS integration
- Email management with Poste.io integration
- Backup and restore functionality
- Terminal access and file management
- cPanel import capabilities
- API system for external integrations (WHMCS, etc.)

## PROJECT STRUCTURE

### Root Directory Layout

```
/home/jknapp/code/whp/
├── CLAUDE.md                 # This documentation file
├── setup.sh                  # Initial installation script
├── update.sh                 # Update system script
├── web-files/                # Main web application (DIRECT EDITING ALLOWED)
├── scripts/                  # System management scripts
│   ├── apply-scripts.sh      # Shell script execution system
│   └── updates/              # Update shell scripts
│       ├── staging/          # ⚠️ REQUIRED: Development scripts HERE
│       └── [version]/        # ⚠️ FORBIDDEN: Never create manually
├── sql/                      # Database schemas and migrations
│   └── migrations/
│       ├── staging/          # ⚠️ REQUIRED: Development migrations HERE
│       └── [date]/           # ⚠️ FORBIDDEN: Never create manually
├── configs/                  # System configuration templates
├── docs/                     # Technical documentation
├── Notes/                    # Development notes and guides
├── terminal-server/          # WebSocket terminal server
├── release-server/           # Update distribution system
└── whmcs-module/             # WHMCS integration module
```

### Web Application Structure (`web-files/`)

```
web-files/                    # ✅ DIRECT EDITING ALLOWED
├── index.php                 # Main application entry point
├── login.php                 # Authentication system
├── auto-prepend.php          # Global includes and security
├── pages/                    # Application pages
├── api/                      # API endpoints
├── libs/                     # Core PHP libraries
├── css/, js/, font/, images/ # Frontend assets
├── includes/                 # Reusable components
├── phpMyAdmin/               # Database management
├── filemanager/              # File management interface
└── configs/config.php        # Application configuration
```

## 🚨 MANDATORY AI AGENT WORKFLOW 🚨

### Required Agent Consultations

**THESE CONSULTATIONS ARE NOT OPTIONAL - THEY ARE REQUIRED**

#### @agent-security-auditor - REQUIRED FOR ALL CHANGES
- **MUST** audit code changes for security vulnerabilities
- **MUST** review authentication and authorization implementations
- **MUST** validate input sanitization and SQL injection prevention
- **MUST** check for CSRF protection and session management issues
- **REQUIRED** before ANY commit

#### @agent-software-documentation-expert - REQUIRED FOR ALL CHANGES
- **MUST** review code changes to identify documentation updates needed
- **MUST** ensure documentation accuracy and completeness
- **MUST** update function signatures and API endpoint documentation
- **MUST** create or update all documentation files as recommended
- **IMPLEMENTATION REQUIRED**: All documentation recommendations MUST be implemented immediately
- **REQUIRED** before ANY commit

#### @ui-ux-design-specialist - REQUIRED FOR UI/UX CHANGES
- **MUST** review interface modifications and ensure DESIGN_PRACTICES.md compliance
- **MUST** maintain UI/UX consistency across the WHP control panel
- **MUST** validate responsive design and cross-browser compatibility
- **REQUIRED** before ANY UI/UX commit

#### @project-development-planner - REQUIRED FOR NEW FEATURES
- **MUST** analyze proposed enhancements and requirements
- **MUST** break down complex features into actionable steps
- **MUST** create detailed implementation plans
- **REQUIRED** before starting major feature development

### MANDATORY Agent Workflow Sequence

```
1. Planning Phase (if new feature):
   └── @project-development-planner creates implementation plan

2. Development Phase:
   └── Code implementation following established patterns

3. Pre-Commit Review (REQUIRED FOR ALL CHANGES):
   ├── IF UI/UX changes: @ui-ux-design-specialist review FIRST
   ├── @agent-security-auditor audits edited files
   └── @agent-software-documentation-expert reviews documentation

4. Commit Authorization:
   └── ONLY AFTER all relevant agents approve
```

**SKIPPING ANY REQUIRED AGENT REVIEW IS PROHIBITED**

## 🚨 STAGING TO RELEASE WORKFLOW 🚨

### REQUIRED Development Process

**DEVELOPMENT PHASE (MANDATORY STAGING USAGE):**
1. **SQL Migrations**: MUST be created in `sql/migrations/staging/`
2. **Shell Scripts**: MUST be created in `scripts/updates/staging/`
3. **Testing**: Thoroughly test all staged components
4. **Security Review**: Agent validation REQUIRED

**RELEASE PHASE (AUTOMATED ONLY):**
1. **ONLY** `build-release.sh` may move files from staging
2. Files moved to versioned directories (e.g., `sql/migrations/2025.08.1/`)
3. `UPDATE_HISTORY.json` created with version tracking
4. Historical record maintained

**MANUAL CREATION OF VERSIONED DIRECTORIES IS STRICTLY FORBIDDEN**

### Shell Script Security Validation

**REQUIRED STANDARDS FOR ALL SHELL SCRIPTS:**

```bash
#!/usr/bin/env bash
# ⚠️ EXACT SHEBANG REQUIRED - NO VARIATIONS ALLOWED

# REQUIRED: Proper error handling
set -euo pipefail

# REQUIRED: Absolute paths only
SCRIPT_DIR="/absolute/path/to/directory"

# REQUIRED: Input validation
if [[ -z "${1:-}" ]]; then
    echo "Error: Missing required parameter" >&2
    exit 1
fi

# REQUIRED: Exit codes
exit 0  # Success
exit 1  # Failure
```

**SECURITY VALIDATION CHECKS:**
- Path validation (prevent directory traversal)
- Shebang enforcement (`#!/usr/bin/env bash` only)
- Dangerous pattern detection
- File size limits (<1MB)
- Permission validation

## CORE COMPONENTS AND LIBRARIES

### Authentication and Security (`libs/`)

- `security.php` - Core security functions and session management
- `security_manager.php` - Advanced security features and monitoring
- `permission_manager.php` - Role-based access control system
- `permission_checker.php` - Permission validation helpers
- `api_auth.php` - API authentication middleware
- `ApiKeyManager.php` - API key generation and validation
- `password_manager.php` - Password hashing and validation

### User and Resource Management

- `usermgmt.php` - User account management
- `mysqlmgmt.php` - MySQL/MariaDB database management
- `pgsqlmgmt.php` - PostgreSQL database management
- `emailmgmt.php` - Email account management
- `haproxy_manager.php` - Load balancer configuration

### Infrastructure Management

- `docker-api.php` - Docker container management
- `network.php` - Network configuration utilities
- `server-info.php` - System information gathering
- `ssl_manager.php` - SSL certificate management

### DNS and Domain Management

- `pdns-api.php` - PowerDNS API integration
- `pdns-functions.php` - DNS management functions
- `pdns-integration.php` - DNS system integration
- `dns_validator.php` - DNS validation utilities

### Backup and WordPress

- `BackupEngine.php` - Core backup functionality
- `BackupJob.php`, `BackupSchedule.php`, `BackupTarget.php` - Backup system components
- `WordPressManager.php` - WordPress installation and management
- `CpanelBackupImporter.php` - cPanel migration tools

### Communication and Integration

- `mailserver-client.php` - Mail server integration
- `announcement_manager.php` - System announcements

## API SYSTEM

### External API Endpoints (`api/external/`)

The API system provides external integration capabilities:

- `users.php` - User management operations (CRUD)
- `sso.php` - Single sign-on token generation
- `system.php` - System information and health checks

### Internal API Endpoints (`api/`)

- `user-management.php` - Internal user operations
- `wordpress-ajax.php` - WordPress management operations
- `backup-actions.php` - Backup system operations
- `docker-management.php` - Container management
- `email-management.php` - Email system operations
- `terminal.php` - Web terminal functionality

### Unauthenticated API Endpoints (`api/`)

- `health.php` - Unauthenticated health check for monitoring systems
  - **No authentication required** (designed for external monitoring tools)
  - Rate limited to 10 requests per minute per IP address
  - Returns status-only by default, detailed metrics with `detailed=1` parameter
  - Supports CPU, connections, disk, and Docker health checks
  - Bypasses authentication via `auto-prepend.php` exclusion
  - **Documentation**: `/docs/api/health-check-api.md`
  - **Integration Guide**: `/docs/integration/health-check-monitoring.md`

### API Authentication

- API keys are managed through `ApiKeyManager.php`
- Rate limiting and usage tracking in `api_usage_logs` table
- Support for both API key and session-based authentication
- IP whitelisting and comprehensive audit logging
- **Exception**: Health check endpoint (`health.php`) requires no authentication

## DATABASE ARCHITECTURE

### Primary Database (MySQL/MariaDB)

Key tables include:
- `users` - User accounts and authentication
- `client_allowances` - Resource limits and permissions
- `domains` - Domain management
- `containers` - Docker container tracking
- `databases` - Database instance management
- `dns_records` - DNS zone data
- `backup_schedules` - Backup automation
- `api_keys` - API authentication
- `wordpress_sites` - WordPress installation tracking
- `announcements` - System messaging

### REQUIRED Migration System Workflow

**DEVELOPMENT (STAGING REQUIRED):**
- Create migrations in `sql/migrations/staging/` ONLY
- Date-based naming: `YYYY-MM-DD_descriptive-name.sql`
- Include rollback capabilities for critical changes
- Test thoroughly before release

**RELEASE (AUTOMATED ONLY):**
- `build-release.sh` moves staging migrations to dated directories
- Automatic migration detection and application
- Schema versioning and integrity checks
- Historical tracking maintained

## SHELL SCRIPT EXECUTION SYSTEM

### REQUIRED Script Organization

**DEVELOPMENT PHASE (MANDATORY):**
- Scripts MUST be placed in `scripts/updates/staging/` during development
- Similar organization to SQL migrations in `sql/migrations/staging/`
- Scripts MUST be thoroughly tested before release
- Security validation REQUIRED

**RELEASE PHASE (AUTOMATED ONLY):**
- `build-release.sh` automatically moves scripts from staging to versioned directories
- Scripts organized by version (e.g., `scripts/updates/2025.08.1/`)
- Maintains historical record of all executed scripts
- Creates `UPDATE_HISTORY.json` with comprehensive version tracking

### MANDATORY Execution Workflow

The shell script system integrates into the update process with this REQUIRED sequence:

1. **Database Migrations**: SQL migrations applied first via `scripts/apply-migrations.sh`
2. **Shell Script Execution**: Scripts executed via `scripts/apply-scripts.sh`
3. **File Synchronization**: Web files synchronized via rsync
4. **Service Management**: Services restarted as needed

### REQUIRED Script Standards

**SECURITY STANDARDS (MANDATORY):**
- **Shebang**: ALL scripts MUST use `#!/usr/bin/env bash` - NO EXCEPTIONS
- **Validation**: Scripts undergo security validation before execution
- **Permissions**: Proper file permissions and ownership validation REQUIRED
- **Input Sanitization**: ALL user inputs MUST be properly sanitized

**ERROR HANDLING (MANDATORY):**
- Scripts MUST implement proper error handling and exit codes
- Failed scripts WILL halt the update process to prevent system corruption
- Comprehensive logging of all script operations REQUIRED
- Rollback capabilities for critical operations REQUIRED

**BEST PRACTICES (REQUIRED):**
- Use absolute paths for ALL file operations
- Implement idempotent operations (safe to run multiple times)
- Include descriptive comments explaining script purpose and operations
- Test scripts thoroughly in development environment
- Follow established coding standards for shell scripting

### Script Execution Engine

The `scripts/apply-scripts.sh` system provides:

**EXECUTION FEATURES:**
- **Sequential Processing**: Scripts executed in order based on filename within each version
- **Version-Based Processing**: Processes versions in chronological order (YYYY.MM.N format)
- **Error Recovery**: Stops execution on first failure to prevent cascade issues
- **Timeout Protection**: 5-minute timeout per script to prevent hanging operations
- **Logging**: Comprehensive logs of all script operations and outputs

**SECURITY VALIDATION (MANDATORY):**
- **Path Validation**: Prevents directory traversal attacks with canonical path checking
- **Shebang Enforcement**: Requires exact `#!/usr/bin/env bash` shebang format
- **Dangerous Pattern Detection**: Scans for potentially harmful command patterns
- **Permission Validation**: Ensures proper file ownership and permissions
- **File Size Limits**: Prevents execution of suspiciously large files (>1MB)
- **Environment Sanitization**: Scripts run in controlled environment with limited variables

### Script Execution API

#### apply-scripts.sh Function Signature

```bash
scripts/apply-scripts.sh <current_version> <new_version> <scripts_dir>
```

**REQUIRED Parameters:**
- `current_version`: Current system version in YYYY.MM.N format (e.g., "2025.07.1")
- `new_version`: Target version to update to in YYYY.MM.N format (e.g., "2025.08.1")
- `scripts_dir`: Absolute path to scripts directory (MUST end with "/scripts/updates")

**Return Codes:**
- `0`: All scripts executed successfully
- `1`: Validation error, script execution failure, or security violation

**Examples:**
```bash
# Update from version 2025.07.1 to 2025.08.1
./scripts/apply-scripts.sh "2025.07.1" "2025.08.1" "/root/whp/scripts/updates"

# No scripts to execute (same version)
./scripts/apply-scripts.sh "2025.08.1" "2025.08.1" "/root/whp/scripts/updates"
```

### MANDATORY Development Workflow

1. **Create Script**: Develop script in `scripts/updates/staging/` ONLY
2. **Security Review**: Ensure script follows ALL security requirements
3. **Testing**: Test script thoroughly in development environment
4. **Agent Review**: Security auditor validation REQUIRED
5. **Release**: Script moved to versioned directory during build process ONLY
6. **Execution**: Script runs automatically during system updates

### Update History Tracking

The system creates comprehensive tracking files during the build process:

#### UPDATE_HISTORY.json Format

```json
{
  "current_version": "2025.08.1",
  "oldest_supported": "2025.07.1",
  "migration_count": 5,
  "script_count": 3,
  "total_versions": 2,
  "build_date": "2025-08-01T12:00:00Z",
  "build_host": "build-server"
}
```

**REQUIRED Fields:**
- `current_version`: Version being built/installed
- `oldest_supported`: Earliest version this release can update from
- `migration_count`: Number of SQL migration directories included
- `script_count`: Number of shell script directories included  
- `total_versions`: Unique versions covered by migrations and scripts
- `build_date`: ISO 8601 timestamp of build
- `build_host`: Hostname where build was performed

This file is used by the update system to validate compatibility and provide detailed update information.

## KEY WORKFLOWS

### User Account Creation

1. Create user in `users` table
2. Set up `client_allowances` with resource limits
3. Create system user account (PAM integration)
4. Set up directory structure and permissions
5. Configure initial quotas and limits

### Site Deployment

1. Domain registration in `domains` table
2. Container type selection and resource allocation
3. Docker container creation and configuration
4. HAProxy configuration for load balancing
5. DNS record creation via PowerDNS
6. SSL certificate provisioning (Let's Encrypt)

### WordPress Installation

1. Site container preparation
2. WordPress core download and extraction
3. Database creation and configuration
4. wp-config.php generation with security keys
5. Security hardening (8+ security features)
6. Plugin installation (auto-login, security, etc.)
7. Auto-update configuration

### Backup Process

1. Backup job scheduling via `backup_schedules`
2. File system snapshot creation
3. Database export (mysqldump/pg_dump)
4. Compression and encryption
5. Storage to local/remote destinations
6. Cleanup of old backups based on retention policy

## PERMISSION SYSTEM

### Role-Based Access Control

- **Root Users**: Full system access and management
- **Regular Users**: Access to own resources only
- Domain ownership validation for all operations
- API-level permission checking

### Permission Matrix

| Resource | Root | Regular User |
|----------|------|--------------|
| User Management | Full | None |
| Security Settings | Full | None |
| System Configuration | Full | None |
| Own Domains | Full | Full |
| Own Databases | Full | Full |
| Own Email Accounts | Full | Full |

### Access Logging

All access attempts logged to `permissions.db` SQLite database:
- User identification and IP tracking
- Timestamp and resource accessed
- Success/failure status and reason codes
- Automatic cleanup after 90 days

## MANDATORY DEVELOPMENT GUIDELINES

### REQUIRED File Organization

- Place reusable code in `libs/` directory
- API endpoints in appropriate `api/` subdirectories
- Page components in `pages/` directory
- Database changes REQUIRE migration files in `sql/migrations/staging/`
- Shell scripts for updates in `scripts/updates/staging/` during development

### MANDATORY Security Requirements

- ALL database queries MUST use prepared statements
- Input validation on ALL user inputs REQUIRED
- CSRF protection via nonce tokens REQUIRED
- Session management with IP validation REQUIRED
- Permission checking on ALL operations REQUIRED
- Shell scripts MUST use `#!/usr/bin/env bash` shebang
- ALL shell scripts undergo security validation before execution
- Script execution with proper error handling and comprehensive logging REQUIRED

### REQUIRED Coding Standards

- PHP 8.2+ compatibility REQUIRED
- Error handling with appropriate user feedback REQUIRED
- Logging for debugging and audit purposes REQUIRED
- Comments for complex business logic REQUIRED
- Database transactions for multi-table operations REQUIRED

## MANDATORY VERSION CONTROL REQUIREMENTS

### ABSOLUTE PRE-COMMIT REQUIREMENTS

**THIS SEQUENCE IS MANDATORY FOR EVERY CODE CHANGE:**

```
1. git pull (ALWAYS first)
2. IF UI/UX changes: @ui-ux-design-specialist review REQUIRED
3. @agent-security-auditor audit REQUIRED for ALL changes
4. @agent-software-documentation-expert review REQUIRED for ALL changes
5. ONLY after agent approvals: git commit and push
```

**ADDITIONAL REQUIREMENTS:**
- Create meaningful commit messages describing the change purpose
- Use feature branches for significant changes
- Test changes in development environment before production
- NEVER bypass agent reviews

## DEPLOYMENT AND UPDATES

### Production Deployment

- Use the release server system for updates
- Apply database migrations via `scripts/apply-migrations.sh`
- Execute shell scripts via `scripts/apply-scripts.sh`
- Sync web files via rsync to `/docker/whp/web/`
- Verify all services after deployment

### MANDATORY Update Process

1. **Download Updates**: Retrieve update package via `scripts/download-update.sh`
2. **Apply SQL Migrations**: Execute database changes via `scripts/apply-migrations.sh`
3. **Execute Shell Scripts**: Run system configuration updates via `scripts/apply-scripts.sh`
4. **Sync Web Files**: Update application files via rsync to `/docker/whp/web/`
5. **Restart Services**: Restart necessary system services as required
6. **Verify System**: Confirm all components are functioning correctly

The update process ensures shell scripts run after database migrations but before file synchronization, allowing scripts to work with updated schemas while preparing the system for new application files.

## SYSTEM INTEGRATION POINTS

### Docker Integration

- Container lifecycle management via Docker API
- Resource allocation and monitoring
- Network isolation and security
- Volume management for persistent data

### HAProxy Load Balancing

- Automatic configuration updates
- SSL termination and certificate management
- Health checking and failover
- Performance monitoring

### PowerDNS Integration

- Zone management and record updates
- DNSSEC support and validation
- API-based configuration changes
- Real-time DNS propagation

### Mail Server (Poste.io)

- Domain and user synchronization
- Quota management and monitoring
- DKIM configuration and validation
- Spam protection and filtering

## BACKUP AND RECOVERY

### Backup Strategies

- File-level backups with rsync/tar
- Database dumps with compression
- Incremental backup support
- Remote storage integration (S3, FTP, etc.)

### Recovery Procedures

- Point-in-time restoration capabilities
- Selective file/database recovery
- Container rebuild from backups
- Disaster recovery documentation

## MONITORING AND MAINTENANCE

### System Health Monitoring

- Container resource usage tracking
- Database performance monitoring
- Disk space and quota management
- Service availability checking

### Automated Maintenance

- Log rotation and cleanup
- Backup cleanup based on retention
- SSL certificate renewal
- WordPress core/plugin updates
- Security patch application

## EXTERNAL INTEGRATIONS

### WHMCS Module

- Located in `whmcs-module/` directory
- Provides hosting automation capabilities
- API-based account provisioning
- SSO integration for seamless user experience

### API Integrations

- RESTful API with comprehensive endpoints
- Rate limiting and usage tracking
- Webhook support for real-time events
- Authentication via API keys or sessions

## TROUBLESHOOTING

### Common Issues

- **Permission denied errors**: Check user roles and domain ownership
- **Container startup failures**: Verify Docker service and resource limits
- **Database connection issues**: Check MySQL/PostgreSQL service status
- **SSL certificate problems**: Verify Let's Encrypt integration
- **Backup failures**: Check storage permissions and space availability

### Log Locations

- Application logs: `/docker/whp/logs/`
- Apache logs: Container-specific log directories
- System logs: `/var/log/` on host system
- Permission logs: `/docker/whp/sql/permissions.db`

### Debug Tools

- Use Security Management page for access statistics
- Check container logs via Docker commands
- Monitor system resources via server info page
- Review API usage logs for integration issues

---

## 🚨 FINAL REMINDERS - NON-NEGOTIABLE REQUIREMENTS 🚨

```
████████████████████████████████████████████████████████████████████
█                                                                  █
█  1. ALWAYS git pull before making changes                       █
█  2. ALWAYS use staging directories for migrations and scripts    █
█  3. ALWAYS get agent approvals before committing                 █
█  4. NEVER create versioned directories manually                  █
█  5. NEVER bypass security validation requirements                █
█  6. NEVER run services locally - SSH to testing server          █
█  7. NEVER attempt local validation - use testing server         █
█                                                                  █
█  COMPLIANCE WITH THESE REQUIREMENTS IS MANDATORY                 █
█                                                                  █
████████████████████████████████████████████████████████████████████
```

**VIOLATION OF ANY REQUIREMENT LISTED IN THIS DOCUMENT IS STRICTLY PROHIBITED**
- ALWAYS run @agent-sql-migration-auditor after creating an SQL Migration file to check the file and update the complete-database-setup.sql