Periodically backup your Google Photos to Google Cloud Storage
Why?
Google Cloud Storage is cheaper, and you pay only for what you use than Google One. Also, you can erase any photo, and you still have a copy of that.
Installation
Create a Compute Engine (a VM).
If you choose Ubuntu, first of all, remove snap
sudo apt autoremove --purge snapd
sudo rm -rf /var/cache/snapd/
rm -rf ~/snap
Install gcsfuse or follow the official instructions.
export GCSFUSE_REPO=gcsfuse-`lsb_release -c -s`
echo "deb http://packages.cloud.google.com/apt $GCSFUSE_REPO main" | sudo tee /etc/apt/sources.list.d/gcsfuse.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
sudo apt-get update
sudo apt-get install gcsfuse
On Google Cloud console create a bucket of the type Nearline, in my case the name of the bucket is tank1, then back to your VM and create a dir with the same name of the bucket.
mkdir name-of-your-bucket
Now install gphotos-sync.
sudo apt install -y python3-pip
pip3 install gphotos-sync
I created a small Python script to deal with multiple Google accounts. I’ll explain later how it works.
cat <<EOF > /home/ubuntu/synchronize.py
#!/usr/bin/env python3
import os
import sys
import subprocess
from pathlib import Path
import requests
home = Path(os.path.expanduser("~")) / "tank1/photos"
args = [
"--ntfs",
"--retry-download",
"--skip-albums",
"--photos-path", ".",
"--log-level", "DEBUG",
]
env = os.environ.copy()
env["LC_ALL"] = "en_US.UTF-8"
for p in home.glob("*/*"):
subprocess.run(["/home/ubuntu/.local/bin/gphotos-sync", *args, str(p.relative_to(home))], check=True, cwd=home, env=env, stdout=sys.stdout, stderr=subprocess.STDOUT)
# I use healthchecks.io to alert me if the script has stopped work
url = "https://hc-ping.com/uuid4"
response = requests.get(url, timeout=60)
response.raise_for_status()
EOF
Give execute permission.
chmod u+x synchronize.py
Now let’s create some systemd scripts.
sudo su
Let’s create a service to gcsfuse, responsible to mount the bucket locally using the FUSE.
cat <<EOF >/etc/systemd/system/gcsfuse.service
# Script stolen from https://gist.github.com/craigafinch/292f98618f8eadc33e9633e6e3b54c05
[Unit]
Description=Google Cloud Storage FUSE mounter
After=local-fs.target network-online.target google.service sys-fs-fuse-connections.mount
Before=shutdown.target
[Service]
Type=forking
User=ubuntu
ExecStart=/bin/gcsfuse tank1 /home/ubuntu/tank1
ExecStop=/bin/fusermount -u /home/ubuntu/tank1
Restart=always
[Install]
WantedBy=multi-user.target
EOF
Enable and start the service:
systemctl enable gcsfuse.service
systemctl start gcsfuse.service
cat <<EOF >/etc/systemd/system/gphotos-sync.service
[Unit]
Description=Run gphotos-sync for each account
[Service]
User=ubuntu
ExecStart=/home/ubuntu/synchronize.py
EOF
And enable the service.
systemctl enable gphotos-sync.service
Now let’s create a timer to run 1 minute after the boot the gphotos-sync.service with gcsfuse.service as dependency.
cat <<EOF >/etc/systemd/system/gphotos-sync.timer
[Unit]
Description=Run gphotos sync service weekly
Requires=gcsfuse.service
[Timer]
OnBootSec=1min
Unit=gphotos-sync.service
[Install]
WantedBy=timers.target
EOF
systemctl enable gphotos-sync.timer
systemctl start gphotos-sync.timer
exit (back to ubuntu user)
Now follow https://docs.google.com/document/d/1ck1679H8ifmZ_4eVbDeD_-jezIcZ-j6MlaNaeQiz7y0/edit to get a client_secret.json to use with gphotos-sync.
mkdir -p /home/ubuntu/.config/gphotos-sync/
# Copy the contents of the json to the file bellow
vim /home/ubuntu/.config/gphotos-sync/client_secret.json
Testing
Due to an issue with gcsfuse, I was unable to create the backup dir directly on the bucket. The workaround is to create a temp directory and start the gphotos-sync manually first.
mkdir -p ~/temp/username/0
cd ~/temp
gphotos-sync --ntfs --skip-albums --photos-path . username/0
# gphotos-sync will ask for a token, paste it and CTRL-C to stop the download of photos.
cp ~/temp/username/ ~/tank1/photos/username
Verify if it is working.
./synchronize.py
After executing the command above, the script should start the backup. You can wait until it finishes or continue to the steps below.
Schedule startup and shutdown of the VM
The content below is based on and simplified version of Scheduling compute instances with Cloud Scheduler by Google
Back to your VM and add the label runtime with the value weekly, this is needed by the function below to know which instances should be started or shutdown.
Create a new directory, in my case, I will call functions and add two files:
index.js
const Compute = require("@google-cloud/compute");
const compute = new Compute();
exports.startInstancePubSub = async (event, context, callback) => {
try {
const payload = JSON.parse(Buffer.from(event.data, "base64").toString());
const options = { filter: `labels.${payload.label}` };
const [vms] = await compute.getVMs(options);
await Promise.all(
vms.map(async (instance) => {
if (payload.zone === instance.zone.id) {
const [operation] = await compute.zone(payload.zone).vm(instance.name).start();
return operation.promise();
}
}),
);
const message = "Successfully started instance(s)";
console.log(message);
callback(null, message);
} catch (err) {
console.log(err);
callback(err);
}
};
exports.stopInstancePubSub = async (event, context, callback) => {
try {
const payload = JSON.parse(Buffer.from(event.data, "base64").toString());
const options = { filter: `labels.${payload.label}` };
const [vms] = await compute.getVMs(options);
await Promise.all(
vms.map(async (instance) => {
if (payload.zone === instance.zone.id) {
const [operation] = await compute.zone(payload.zone).vm(instance.name).stop();
return operation.promise();
} else {
return Promise.resolve();
}
}),
);
const message = "Successfully stopped instance(s)";
console.log(message);
callback(null, message);
} catch (err) {
console.log(err);
callback(err);
}
};
And
package.json
{
"main": "index.js",
"private": true,
"dependencies": {
"@google-cloud/compute": "^2.4.1"
}
}
Create a PubSub topic to start the instance.
gcloud pubsub topics create start-instance-event
Now deploy the startInstancePubSub function
gcloud functions deploy startInstancePubSub \
--trigger-topic start-instance-event \
--runtime nodejs12 \
--allow-unauthenticated
And another PubSub topic to stop the instance.
gcloud pubsub topics create stop-instance-event
And the stopInstancePubSub function
gcloud functions deploy stopInstancePubSub \
--trigger-topic stop-instance-event \
--runtime nodejs12 \
--allow-unauthenticated
And finally, let’s create two Cloud Scheduler to publish on the topics on Sunday and Monday at midnight.
gcloud beta scheduler jobs create pubsub startup-weekly-instances \
--schedule '0 0 * * SUN' \
--topic start-instance-event \
--message-body '{"zone":"us-central1-a", "label":"runtime=weekly"}' \
--time-zone 'America/Sao_Paulo'
gcloud beta scheduler jobs create pubsub shutdown-weekly-instances \
--schedule '0 0 * * MON' \
--topic stop-instance-event \
--message-body '{"zone":"us-central1-a", "label":"runtime=weekly"}' \
--time-zone 'America/Sao_Paulo'
After this setup, your VM will start every Sunday, backup all your photos of all accounts and shutdown on Monday.