Cloud sync setup

This page connects your Pis to Google Drive with rclone, so each Pi can back up its sessions and browse the other’s. What the sync does once it is running (the publish projection, the peer mirror and the read-only peer IO view) is in Saving and loading sessions.

You do not need to understand the sync daemon to follow this. You do need:

• physical or SSH access to each Pi as the patch user,
• a laptop or desktop with a web browser (the Pi is headless, so the Google sign-in step cannot be done on the Pi alone),
• a Google account for the Pis to share. A dedicated account for your rig is better than your personal one, because every Pi will hold a credential for it.

Work through the sections in order. Sections 1 to 3 are done once, not per Pi. Sections 4 to 7 are done on each Pi. Section 8 checks that the Pis can see each other.

⚠️ /etc/mseq/rclone.conf is a secret. It contains a live Google credential, and rclone obscure only disguises it: it does not encrypt it. Never commit the file, paste it into chat or logs, or copy it into the repo. If it leaks, go to Google Account → Security → Third-party access and remove mseq-cloud-sync, then repeat Sections 1 and 5.

0. Choose how the Pis sign in (read first)

There are two ways for the Pis to sign in to Drive. Pick one before touching anything.

If you trust everyone who can get their hands on the Pis, use shared-account OAuth. The rest of this page is written for that path. The service-account variant is summarised in Section 9.

1. Create your own Google OAuth client (once)

rclone ships with a shared default OAuth client ID. It works, but it is heavily rate-limited and you will hit throttling during normal sync. Create your own client. It is free and takes ~10 minutes.

Do this once, signed in to the Google account the Pis will share, at https://console.cloud.google.com.

1. Create a project (top bar → project picker → New Project). Name it e.g. mseq-cloud-sync. Select it once created.

2. Enable the Drive API: APIs & Services → Library → search “Google Drive API” → Enable.

3. Configure the OAuth consent screen: APIs & Services → OAuth consent screen.
   • User type: External.
   • Fill in the required app name (mseq-cloud-sync), the support email, and the developer email. You can leave logos/links blank.
   • Scopes: you can leave the scope list empty here; rclone requests the scope at config time (Section 5). Click through.
   • Test users: add the shared account as a test user. This is important (see the box below).

   ⚠️ Publishing status = “Testing” expires tokens after 7 days. If you leave the consent screen in Testing status, the refresh token Google issues will stop working after 7 days and sync will silently die. Two fixes: (a) add the account as a Test User and accept that you may need to re-auth periodically, or (b) click Publish App (push to Production): for an unverified app used only by its own account this is fine and gives a non-expiring refresh token. Recommended: Publish App.

4. Create the OAuth client credential: APIs & Services → Credentials → Create Credentials → OAuth client ID.
   • Application type: Desktop app.
   • Name: mseq-rclone.
   • Click Create. Google shows a Client ID and Client secret. Copy both somewhere safe (a password manager). You’ll paste them into rclone config on each Pi in Section 5.

2. (Reference) The shared Drive layout

The sync daemon expects this tree in the shared account’s Drive, with one folder per Pi named after its hostname. The examples on this page call the two Pis mseq-a and mseq-b. You don’t create the per-Pi folders by hand: the smoke test in Section 7 and the daemon’s first push create them. You only create the root in Section 3.

mseq/
  mseq-a/
    patches/        # mseq-a's published sessions (it writes; its peer reads)
    _contrib/       # reserved; nothing uses it yet
  mseq-b/
    patches/        # mseq-b's published sessions
    _contrib/

3. Create the shared root folder (once)

The mseq/ folder must exist before either Pi pushes. Easiest: in the Drive web UI, signed in as the shared account, create a top-level folder named exactly mseq. That’s all this step needs.

(You can also create it with rclone from any machine once you’ve completed a config: rclone --config <conf> mkdir gdrive:mseq.)

4. Install rclone on the Pi

On each Pi, as patch:

sudo apt-get update
sudo apt-get install -y rclone

Create the secrets directory. It must be writable by the patch group because rclone refreshes OAuth tokens by writing a temporary config file beside /etc/mseq/rclone.conf and renaming it into place:

sudo install -d -m 0770 -o root -g patch /etc/mseq

5. Run rclone config for Google Drive (on each Pi: the headless flow)

This is the step people get stuck on, because the Pi has no browser and rclone’s default flow tries to open one. Follow the wizard answers exactly.

On the Pi:

umask 077
rclone config --config /tmp/mseq-rclone.conf

Answer the wizard:

After you answer No to “Use auto config?”, rclone prints a command like:

rclone authorize "drive" "<long-base64-blob>"

Now go to your laptop/desktop that has rclone installed and a browser, and run that exact command there:

# On your desktop, NOT the Pi:
rclone authorize "drive" "<long-base64-blob>"

It opens a browser, you sign in as the shared account and approve access. rclone then prints a token blob ({"access_token":...}). Copy the entire blob, return to the Pi’s still-waiting wizard, and paste it at the config_token> prompt.

Finish the wizard:

💡 No desktop rclone? Two alternatives to the rclone authorize dance: install rclone on the desktop (it’s a single binary), or SSH into the Pi with a forwarded port: ssh -L 53682:localhost:53682 patch@<pi>.local, and then answer y to “Use auto config?”; the browser on your laptop will reach the Pi’s local callback through the tunnel.

Scope note (why drive, not drive.file)

It is tempting to narrow the token “to the mseq/ subtree”. Be aware:

• OAuth scopes are not folder scoping. drive.file only lets rclone see files it created, which breaks peer browse: the other Pi could never read your published sessions. Use the full drive scope.
• If you want to reduce blast radius, use rclone’s root_folder_id instead (Section 6). That constrains what the remote can see, but it is a convenience boundary, not a security boundary: the token still has account-wide reach.

6. (Optional) Confine the remote to the mseq/ folder

You can make the gdrive remote treat the mseq/ folder as its root, so a stray command can’t wander the whole Drive. Get the folder’s ID from its Drive URL (https://drive.google.com/drive/folders/<THIS_PART>), then, on the Pi, before installing the config:

rclone config --config /tmp/mseq-rclone.conf
# e (edit existing) → gdrive → walk to advanced config → set:
#   root_folder_id> <THIS_PART>

If you set this, the daemon has to address paths relative to mseq/ (for example gdrive:mseq-a/patches rather than gdrive:mseq/mseq-a/patches). To avoid confusion across the two Pis, it’s simplest to skip root_folder_id and keep the full gdrive:mseq/... paths that RCLONE_REMOTE in Section 7 assumes. Reduce blast radius with file permissions and physical trust instead.

7. Install the config and smoke-test (on each Pi)

Move the freshly-built config into place with permissions that let the daemon user refresh OAuth tokens. rclone may rewrite the config after a successful command, so the file itself and the parent /etc/mseq directory must be writable by patch:

sudo install -m 0660 -o patch -g patch /tmp/mseq-rclone.conf /etc/mseq/rclone.conf
rm -f /tmp/mseq-rclone.conf   # /tmp is tmpfs (RAM-backed) on Pi OS

If you previously installed the config as 0640 root:patch, fix it with:

sudo chown root:patch /etc/mseq
sudo chmod 0770 /etc/mseq
sudo chown patch:patch /etc/mseq/rclone.conf
sudo chmod 0660 /etc/mseq/rclone.conf

Without this, rclone lsd ... may still list successfully but print errors like Failed to save config ... permission denied whenever it tries to persist a refreshed token.

Smoke-test as the daemon user, always passing the explicit config path:

sudo -u patch rclone --config /etc/mseq/rclone.conf lsd gdrive:
sudo -u patch rclone --config /etc/mseq/rclone.conf lsd gdrive:mseq
sudo -u patch rclone --config /etc/mseq/rclone.conf mkdir gdrive:mseq/$(hostname)/patches
sudo -u patch rclone --config /etc/mseq/rclone.conf mkdir gdrive:mseq/$(hostname)/_contrib

The first two should list without error; the mkdirs create this Pi’s folder. ($(hostname) is the Pi’s PI_ID, for example mseq-a.)

Then create /etc/mseq/cloud-sync.env. All six fields are required: the sync daemon refuses to run without them, and omitting PATCHES_DIR gets you PATCHES_DIR: unbound variable on the first cycle rather than a helpful error:

# /etc/mseq/cloud-sync.env: this Pi's cloud-sync identity
PI_ID=mseq-a                         # this Pi's id (its hostname); names its folder on the remote
PEERS=mseq-b                         # the one peer to mirror from (exactly one, no commas)
RCLONE_REMOTE=gdrive:mseq            # the shared root folder from section 3
RCLONE_CONFIG=/etc/mseq/rclone.conf  # the config installed above
PATCHES_DIR=/home/patch/code/sequencer/patches       # PD's session catalog, the publish source
CLOUD_SYNC_DIR=/home/patch/code/sequencer/cloud-sync

install/cloud-sync.env.example is the same set with fuller comments, so copying that and editing PI_ID / PEERS is the safer route.

Install it 0640 root:patch:

sudo chown root:patch /etc/mseq/cloud-sync.env
sudo chmod 0640 /etc/mseq/cloud-sync.env

7.1 Verify the cloud-sync/publish tmpfs

The sync daemon publishes from a generated projection under cloud-sync/publish/, not directly from patches/. That publish directory has to be a tmpfs mount so the daemon’s temporary generations do not add SD-card write load.

You do not provision it here. install/bootstrap-pi.sh installs and enables a system-scope mount unit for exactly this path, and mseq-cloud-sync.service depends on it via RequiresMountsFor=. Do not also add an /etc/fstab entry. Two definitions of one mountpoint is a drift generator, and the unit is the one the service waits on.

So this step is a check. Run on each Pi:

findmnt /home/patch/code/sequencer/cloud-sync/publish
systemctl is-enabled 'home-patch-code-sequencer-cloud\x2dsync-publish.mount'

findmnt should show FSTYPE as tmpfs with options similar to size=20480k,mode=750,uid=1000,gid=1000, and the unit should report enabled. If both are as expected, move on to section 7.2.

If the mount is missing, the bootstrap step did not run (or the unit was removed). Reinstall it from the repo rather than reaching for fstab:

cd /home/patch/code/sequencer
MOUNT_UNIT='home-patch-code-sequencer-cloud\x2dsync-publish.mount'
sudo mkdir -p cloud-sync/publish
sudo chown patch:patch cloud-sync cloud-sync/publish
sudo install -m 0644 "install/systemd/system/$MOUNT_UNIT" "/etc/systemd/system/$MOUNT_UNIT"
sudo systemctl daemon-reload
sudo systemctl enable --now "$MOUNT_UNIT"
findmnt /home/patch/code/sequencer/cloud-sync/publish

If an old fstab entry is present, remove it. It duplicates the unit, and a line split across two lines makes mount fail with a parse error such as parse error at line 7:

sudo cp /etc/fstab /etc/fstab.bak.mseq-cloud-sync.$(date +%Y%m%d%H%M%S)
sudo python3 - <<'PY'
from pathlib import Path

path = Path('/etc/fstab')
lines = path.read_text().splitlines()
fixed = []
skip_orphan_zero = False
for line in lines:
    if '/home/patch/code/sequencer/cloud-sync/publish' in line:
        skip_orphan_zero = True
        continue
    if skip_orphan_zero and line.strip() == '0':
        skip_orphan_zero = False
        continue
    skip_orphan_zero = False
    fixed.append(line.rstrip())
path.write_text('\n'.join(fixed) + '\n')
PY
sudo systemctl daemon-reload
findmnt /home/patch/code/sequencer/cloud-sync/publish

7.2 Final per-Pi permission and mount check

Run this on each Pi before moving on:

stat -c 'etc_mseq=%U:%G %a' /etc/mseq
stat -c 'rclone_conf=%U:%G %a' /etc/mseq/rclone.conf
stat -c 'cloud_sync_env=%U:%G %a' /etc/mseq/cloud-sync.env
findmnt -no TARGET,FSTYPE,OPTIONS /home/patch/code/sequencer/cloud-sync/publish
sudo -u patch rclone --config /etc/mseq/rclone.conf lsd gdrive:mseq

Expected state:

etc_mseq=root:patch 770
rclone_conf=patch:patch 660
cloud_sync_env=root:patch 640
/home/patch/code/sequencer/cloud-sync/publish tmpfs ...

The final rclone command should not print Failed to save config.

8. Confirm the Pis can see each other (after both are set up)

If this is wrong, the peer IO view just looks empty, with no obvious error. From each Pi, list the other Pi’s published folder:

# On mseq-a:
sudo -u patch rclone --config /etc/mseq/rclone.conf lsd gdrive:mseq/mseq-b/patches
# On mseq-b:
sudo -u patch rclone --config /etc/mseq/rclone.conf lsd gdrive:mseq/mseq-a/patches

An empty but successful listing (the peer hasn’t pushed yet) is fine. A permission or not-found error means the sign-in model isn’t wired correctly. Fix it before expecting the peer IO view to show anything.

9. Service-account variant (only if you chose it in Section 0)

Instead of signing in through the browser:

1. In Cloud Console: IAM & Admin → Service Accounts → Create. Create one service account per Pi (or one shared: your call). For each, Keys → Add Key → JSON and download the JSON.
2. Copy each Pi’s JSON to it as /etc/mseq/google-service-account.json (0640 root:patch).
3. In rclone config, leave client_id/client_secret blank and set service_account_file> /etc/mseq/google-service-account.json.
4. Share mseq/ with each service account’s email (it looks like name@project.iam.gserviceaccount.com) as Editor in the Drive web UI. Without this, the service account cannot read or write the shared tree.
5. Run the Section 8 check.

If a service-account JSON leaks, delete that key in Cloud Console (IAM & Admin → Service Accounts → Keys) and add a new one.

If something goes wrong

Symptoms, likely causes and fixes are on Cloud sync troubleshooting.

Cross-References

• Cloud sync troubleshooting: when sync misbehaves
• Saving and loading sessions: the publish projection, the peer mirror and the peer view