Cloud sync setup

Audience: a human operator setting up a Pi by hand. It walks through getting rclone talking to the shared Google Drive so a Pi can back up its sessions and browse a peer’s. What the sync does once it is running (the publish projection, the peer mirror, and the read-only peer view) is in Saving and loading sessions.

You do not need to understand the sync daemon to follow this. You do need:

• physical/SSH access to each Pi as the patch user,
• a laptop/desktop with a web browser (the Pi is headless: the OAuth step cannot be done on the Pi alone),
• access to the shared info.malleable@gmail.com Google account.

Work through the sections in order. Sections 1–3 are done once for the whole fleet. Sections 4–7 are done on each Pi. Section 8 verifies the peers can see each other.

⚠️ /etc/mseq/rclone.conf is a secret. It contains a live Google credential. Never commit it, paste it into chat/logs, or copy it into the repo. If it leaks, revoke the token in Google immediately (Section 9).

0. Decide the auth model (read first)

There are two ways for the Pis to authenticate to Drive. Pick one before touching anything.

For the current two trusted performer Pis, use shared-account OAuth. The rest of this guide is written for that path; the service-account variant is summarised in Section 10.

1. Create your own Google OAuth client (once, fleet-wide)

rclone ships with a shared default OAuth client ID. It works, but it is heavily rate-limited and you will hit throttling during normal sync. Create your own client. It is free and takes ~10 minutes.

Do this once, signed in to the shared Google account (info.malleable@gmail.com), at https://console.cloud.google.com.

1. Create a project (top bar → project picker → New Project). Name it e.g. mseq-cloud-sync. Select it once created.

2. Enable the Drive API: APIs & Services → Library → search “Google Drive API” → Enable.

3. Configure the OAuth consent screen: APIs & Services → OAuth consent screen.
   • User type: External.
   • Fill in the required app name (mseq-cloud-sync), the support email, and the developer email. You can leave logos/links blank.
   • Scopes: you can leave the scope list empty here; rclone requests the scope at config time (Section 5). Click through.
   • Test users: add info.malleable@gmail.com as a test user. This is important (see the box below).

   ⚠️ Publishing status = “Testing” expires tokens after 7 days. If you leave the consent screen in Testing status, the refresh token Google issues will stop working after 7 days and sync will silently die. Two fixes: (a) add the account as a Test User and accept that you may need to re-auth periodically, or (b) click Publish App (push to Production): for an unverified app used only by its own account this is fine and gives a non-expiring refresh token. Recommended: Publish App.

4. Create the OAuth client credential: APIs & Services → Credentials → Create Credentials → OAuth client ID.
   • Application type: Desktop app.
   • Name: mseq-rclone.
   • Click Create. Google shows a Client ID and Client secret. Copy both somewhere safe (a password manager). You’ll paste them into rclone config on each Pi in Section 5.

2. (Reference) The shared Drive layout

The sync daemon expects this tree under the shared account’s Drive. You don’t create the per-Pi folders by hand here: the smoke test in Section 7 and the daemon’s first push create them. You only create the root in Section 3.

mseq/
  mseq-pi4/
    patches/        # mseq-pi4's published sessions (it writes; the peer reads)
    _contrib/       # reserved for a future contribution feature (unused in v1)
  studio-mseq-pi4/
    patches/        # studio-mseq-pi4's published sessions
    _contrib/

3. Create the shared root folder (once, fleet-wide)

The mseq/ folder must exist before either Pi pushes. Easiest: in the Drive web UI, signed in as info.malleable@gmail.com, create a top-level folder named exactly mseq. That’s all this step needs.

(You can also create it with rclone from any machine once you’ve completed a config: rclone --config <conf> mkdir gdrive:mseq.)

4. Install rclone on the Pi

On each Pi, as patch:

sudo apt-get update
sudo apt-get install -y rclone

Create the secrets directory. It must be writable by the patch group because rclone refreshes OAuth tokens by writing a temporary config file beside /etc/mseq/rclone.conf and renaming it into place:

sudo install -d -m 0770 -o root -g patch /etc/mseq

5. Run rclone config for Google Drive (on each Pi: the headless flow)

This is the step people get stuck on, because the Pi has no browser and rclone’s default flow tries to open one. Follow the wizard answers exactly.

On the Pi:

umask 077
rclone config --config /tmp/mseq-rclone.conf

Answer the wizard:

After you answer No to “Use auto config?”, rclone prints a command like:

rclone authorize "drive" "<long-base64-blob>"

Now go to your laptop/desktop that has rclone installed and a browser, and run that exact command there:

# On your desktop, NOT the Pi:
rclone authorize "drive" "<long-base64-blob>"

It opens a browser, you sign in as info.malleable@gmail.com and approve access. rclone then prints a token blob ({"access_token":...}). Copy the entire blob, return to the Pi’s still-waiting wizard, and paste it at the config_token> prompt.

Finish the wizard:

💡 No desktop rclone? Two alternatives to the rclone authorize dance: install rclone on the desktop (it’s a single binary), or SSH into the Pi with a forwarded port: ssh -L 53682:localhost:53682 patch@<pi>.local, and then answer y to “Use auto config?”; the browser on your laptop will reach the Pi’s local callback through the tunnel.

Scope note (why drive, not drive.file)

It is tempting to narrow the token “to the mseq/ subtree”. Be aware:

• OAuth scopes are not folder scoping. drive.file only lets rclone see files it created, which breaks peer browse: the other Pi could never read your published sessions. Use the full drive scope.
• If you want to reduce blast radius, use rclone’s root_folder_id instead (Section 6). That constrains what the remote can see, but it is a convenience boundary, not a security boundary: the token still has account-wide reach.

6. (Optional) Confine the remote to the mseq/ folder

You can make the gdrive remote treat the mseq/ folder as its root, so a stray command can’t wander the whole Drive. Get the folder’s ID from its Drive URL (https://drive.google.com/drive/folders/<THIS_PART>), then, on the Pi, before installing the config:

rclone config --config /tmp/mseq-rclone.conf
# e (edit existing) → gdrive → walk to advanced config → set:
#   root_folder_id> <THIS_PART>

If you set this, the daemon should address paths relative to mseq/ (e.g. gdrive:mseq-pi4/patches rather than gdrive:mseq/mseq-pi4/patches). To avoid confusion across the two Pis, it’s simplest to skip root_folder_id for v1 and keep the full gdrive:mseq/... paths the daemon config in section 3.1 assumes. Reduce blast radius via file permissions + physical trust instead.

7. Install the config and smoke-test (on each Pi)

Move the freshly-built config into place with permissions that let the daemon user refresh OAuth tokens. rclone may rewrite the config after a successful command, so the file itself and the parent /etc/mseq directory must be writable by patch:

sudo install -m 0660 -o patch -g patch /tmp/mseq-rclone.conf /etc/mseq/rclone.conf
rm -f /tmp/mseq-rclone.conf   # /tmp is tmpfs (RAM-backed) on Pi OS

If you previously installed the config as 0640 root:patch, fix it with:

sudo chown root:patch /etc/mseq
sudo chmod 0770 /etc/mseq
sudo chown patch:patch /etc/mseq/rclone.conf
sudo chmod 0660 /etc/mseq/rclone.conf

Without this, rclone lsd ... may still list successfully but print errors like Failed to save config ... permission denied whenever it tries to persist a refreshed token.

Smoke-test as the daemon user, always passing the explicit config path:

sudo -u patch rclone --config /etc/mseq/rclone.conf lsd gdrive:
sudo -u patch rclone --config /etc/mseq/rclone.conf lsd gdrive:mseq
sudo -u patch rclone --config /etc/mseq/rclone.conf mkdir gdrive:mseq/$(hostname)/patches
sudo -u patch rclone --config /etc/mseq/rclone.conf mkdir gdrive:mseq/$(hostname)/_contrib

The first two should list without error; the mkdirs create this Pi’s namespace. ($(hostname) is the Pi’s pi_id, e.g. mseq-pi4.)

Then create /etc/mseq/cloud-sync.env. All six fields are required: the sync daemon refuses to run without them, and omitting PATCHES_DIR gets you PATCHES_DIR: unbound variable on the first cycle rather than a helpful error:

# /etc/mseq/cloud-sync.env — this Pi's cloud-sync identity
PI_ID=mseq-pi4                       # this Pi's id (its hostname); names its folder on the remote
PEERS=mseq-pi3                       # the one peer to mirror from (v1 allows exactly one, no commas)
RCLONE_REMOTE=gdrive:mseq            # the shared root folder from section 3
RCLONE_CONFIG=/etc/mseq/rclone.conf  # the config installed above
PATCHES_DIR=/home/patch/code/sequencer/patches       # PD's session catalog, the publish source
CLOUD_SYNC_DIR=/home/patch/code/sequencer/cloud-sync

install/cloud-sync.env.example is the same set with fuller comments, so copying that and editing PI_ID / PEERS is the safer route.

Install it 0640 root:patch:

sudo chown root:patch /etc/mseq/cloud-sync.env
sudo chmod 0640 /etc/mseq/cloud-sync.env

7.1 Verify the cloud-sync/publish tmpfs

The sync daemon publishes from a generated projection under cloud-sync/publish/, not directly from patches/. That publish directory has to be a tmpfs mount so the daemon’s temporary generations do not add SD-card write load.

You do not provision it here. install/bootstrap-pi.sh installs and enables a system-scope mount unit for exactly this path, and mseq-cloud-sync.service depends on it via RequiresMountsFor=. Do not also add an /etc/fstab entry. Two definitions of one mountpoint is a drift generator, and the unit is the one the service waits on.

So this step is a check. Run on each Pi:

findmnt /home/patch/code/sequencer/cloud-sync/publish
systemctl is-enabled 'home-patch-code-sequencer-cloud\x2dsync-publish.mount'

findmnt should show FSTYPE as tmpfs with options similar to size=20480k,mode=750,uid=1000,gid=1000, and the unit should report enabled. If both are as expected, move on to section 7.2.

If the mount is missing, the bootstrap step did not run (or the unit was removed). Reinstall it from the repo rather than reaching for fstab:

cd /home/patch/code/sequencer
MOUNT_UNIT='home-patch-code-sequencer-cloud\x2dsync-publish.mount'
sudo mkdir -p cloud-sync/publish
sudo chown patch:patch cloud-sync cloud-sync/publish
sudo install -m 0644 "install/systemd/system/$MOUNT_UNIT" "/etc/systemd/system/$MOUNT_UNIT"
sudo systemctl daemon-reload
sudo systemctl enable --now "$MOUNT_UNIT"
findmnt /home/patch/code/sequencer/cloud-sync/publish

If an old fstab entry is present, remove it. It duplicates the unit, and a line split across two lines makes mount fail with a parse error such as parse error at line 7:

sudo cp /etc/fstab /etc/fstab.bak.mseq-cloud-sync.$(date +%Y%m%d%H%M%S)
sudo python3 - <<'PY'
from pathlib import Path

path = Path('/etc/fstab')
lines = path.read_text().splitlines()
fixed = []
skip_orphan_zero = False
for line in lines:
    if '/home/patch/code/sequencer/cloud-sync/publish' in line:
        skip_orphan_zero = True
        continue
    if skip_orphan_zero and line.strip() == '0':
        skip_orphan_zero = False
        continue
    skip_orphan_zero = False
    fixed.append(line.rstrip())
path.write_text('\n'.join(fixed) + '\n')
PY
sudo systemctl daemon-reload
findmnt /home/patch/code/sequencer/cloud-sync/publish

7.2 Final per-Pi permission and mount check

Run this on each Pi before moving on:

stat -c 'etc_mseq=%U:%G %a' /etc/mseq
stat -c 'rclone_conf=%U:%G %a' /etc/mseq/rclone.conf
stat -c 'cloud_sync_env=%U:%G %a' /etc/mseq/cloud-sync.env
findmnt -no TARGET,FSTYPE,OPTIONS /home/patch/code/sequencer/cloud-sync/publish
sudo -u patch rclone --config /etc/mseq/rclone.conf lsd gdrive:mseq

Expected state:

etc_mseq=root:patch 770
rclone_conf=patch:patch 660
cloud_sync_env=root:patch 640
/home/patch/code/sequencer/cloud-sync/publish tmpfs ...

The final rclone command should not print Failed to save config.

8. Confirm the peers can see each other (after both Pis are set up)

This is the pull-browse precondition. If it’s wrong, the Phase 2 PEER view will just look empty with no obvious error. From each Pi, list the other Pi’s published tree:

# On studio-mseq-pi4:
sudo -u patch rclone --config /etc/mseq/rclone.conf lsd gdrive:mseq/mseq-pi4/patches
# On mseq-pi4:
sudo -u patch rclone --config /etc/mseq/rclone.conf lsd gdrive:mseq/studio-mseq-pi4/patches

An empty but successful listing (the peer hasn’t pushed yet) is fine. A permission or not-found error means the auth model isn’t wired correctly. Fix it before expecting the PEER view to show anything.

If something goes wrong

Symptoms, likely causes and fixes are on Cloud sync troubleshooting, along with what to do if a credential leaks and the service-account variant of the auth model.

Cross-References

• Cloud sync troubleshooting: when sync misbehaves, and credential recovery
• Saving and loading sessions: the publish projection, the peer mirror and the peer view