Synchronizing your data across various cloud platforms like Google Cloud, AWS, Azure, or private colocation setups can be a tedious task. Automating this securely is essential. I personally use this script heavily to synchronize a production folder from Google Cloud Platform (GCP) or AWS back to my onsite servers—it has saved me and others countless times. Here’s the perfect Python solution: syncing remote folders using Rsync with SSH key authentication.

Advantages of Rsync and SSH Key Authentication

  • Enhanced Security: SSH keys provide stronger authentication without passwords.
  • High Efficiency: Rsync sends only file changes, significantly speeding up transfers.
  • Fully Automated: Python automation integrated with cron simplifies your workflow.

Step-by-Step Guide for Ubuntu

Step 1: Install Python and Necessary Tools

Ensure your Ubuntu VM has Python and required tools installed:

sudo apt update
sudo apt install -y python3 python3-pip rsync

Python Rsync Script with SSH Key Authentication

Use this Python script to automate synchronization:

import subprocess
from datetime import datetime

# Rsync over SSH with key authentication
def run_rsync():
    ssh_user = "user"
    ssh_host = "prod-10.domain.com"  # Hostname or IP
    ssh_path = "/zvol_1/"  # Path on remote server
    destination = "/zvol_2/prod-10/" 
    ssh_key_path = "/path/to/your/private_key"  # Update this to your private SSH key path

# Update Log File Path
    log_file = f"/path/to/logs/rsync_log_{datetime.now().strftime('%Y%m%d_%H%M%S')}.txt"
    source = f"{ssh_user}@{ssh_host}:{ssh_path}"

    rsync_command = [
        "rsync", "-avz", "--delete", "-e",
        f"ssh -i {ssh_key_path} -T -c aes128-ctr -o Compression=no -x",
        source, destination
    ]

    try:
        print("Running rsync via ssh key authentication...")
        process = subprocess.run(
            rsync_command,
            text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE
        )

        with open(log_file, "w") as log:
            log.write("RSYNC OUTPUT:\n")
            log.write(process.stdout)
            log.write("\n\nRSYNC ERRORS:\n")
            log.write(process.stderr)

        print(f"Rsync completed. Logs saved to {log_file}")

    except FileNotFoundError as e:
        print(f"Error: Required command not found: {e}")
    except Exception as e:
        print(f"An error occurred: {e}")

if __name__ == "__main__":
    run_rsync()

Download Rsync Script

Automating with Cron

Set up the cron job for hourly automated synchronization:

sudo crontab -e

Add this line to schedule your script to run hourly:

0 * * * * sudo python3 /path/to/rsync_over_ssh.py

Save and exit to activate your automated syncing process!

Alternative: Using sshpass (Home Labs Only)

For convenience in home labs or non-production environments, you can use sshpass. While convenient, this is not recommended for secure production setups.

Download Rsync Script using sshpass

Security Best Practices

Always use SSH keys instead of passwords for secure authentication in production environments. Keep your private keys secure and regularly update your software to maintain strong security.

Conclusion

Congratulations! You’ve successfully automated secure and efficient data synchronization across cloud environments using Python, Rsync, and SSH key authentication. Streamline your workflow, enhance security, and minimize manual effort!

Happy Automating!

Categorized in:

PythonUbuntu