Storage reorganisation · running order · merged plan

Same-Bucket Running Order

Separate every stored file into a folder of its own account — in the bucket we already have, under two top-level names it has never used. Nothing is deleted and nothing is moved: every file is copied, and every original stays exactly where it is. Six steps, three gates, each reversible on its own.

The running order at a glance

Six steps and three gates. Steps 1 to 3 are invisible to users; behaviour changes at step 4; nothing is deleted or moved in any of them.

  1. Step 1Store the full file pathWiden the existing columns and fill in where every file is now. Add the migration table. No file moves.
  2. Step 2One rule finds every fileEvery place that glues a location together by hand calls one shared rule instead — API, website, mobile service. Still no file moves.
  3. Gate 1Both steps live, every repo green, older records still displaying, and zero files moved.
  4. Step 3Thumbnails learn both shapesThe live service watches the new folders too and files each thumbnail beside its picture. Must be deployed before step 4.
  5. Step 4New uploads change shapeFlip one setting: new files land in the account’s folder, staged under temp/ until the record is saved. Existing files untouched.
  6. Gate 2An upload from web and phone lands correctly and gets a thumbnail; five abandoned uploads never leave temp/.
  7. Step 5Copy the existing filesAccount by account, smallest first, thumbnail before picture. Copy, verify, update the record, leave the original.
  8. Gate 3One account copied, verified, and rolled back once for real before anything runs at scale.
  9. Step 6Keep and watch the old foldersStop writing to them, log which app versions still read them, and let evidence — not a date — decide when they can go.

Why the order matters more than the work

Files are kept in one shared set of folders today, with the folder name added by the software at the moment it is needed. We are separating them by account.

The risky part is not the copying. It is that customers are running app versions we cannot update, and those apps ask for the old locations. So the sequence is arranged to keep the old locations working for as long as anything asks for them, and every step is reversible on its own.

The old folders do not go away.

They stay exactly as they are, readable, and nothing is ever written to them again or removed from them. Phone apps already installed on customers’ devices will keep asking for the old locations for as long as those versions are in use, and keeping the originals is what lets them keep working. Every rollback in this plan is a rollback onto those originals.

How the two plans reconcile

This document is the running order — reorganise the bucket we have — carrying the folder structure, the staging rules and the migration table designed for the new-bucket plan. Nine questions were answered differently in the two. Here is each one, and how it is settled.

Question Running order said New-bucket plan said Settled as
Where do the new folders live? The same bucket, re-organised in place A second bucket, filled by copying The same bucket. Nothing to create, no region or CORS settings to match, no irreversible permission decision, and no new address for the apps to learn.
How does a read find a file? The record holds the complete path Work it out from the account, with a lookup table for the exceptions The record holds the complete path. Existing records are backfilled with where their file is now, and the migration table covers the gap while the copying runs. This is the running order’s answer, taken whole.
What tracks the copying? A one-shot script, one account at a time One new table, drained by a scheduled job The table and the job, still one account at a time. A script that dies halfway leaves nothing behind to resume from; a table survives the restart and can say what it skipped and why.
Do uploads wait for a save? No — an upload lands in its folder Yes — staged under temp/ until the record is saved Staged. Five pictures picked and then abandoned must not reach an account’s folder, because no record will ever reference them.
Who makes the thumbnails? Teach the live service both shapes Write a second service for the new bucket Teach the live one. With one bucket there is no second service to write — it already watches these folders and will fire on them whether it has been taught them or not. This is the one part of the plan that got harder.
Old thumbnails: copy or remake? Open question Copy them — thumbnail before picture Copy, thumbnail first. A copy counts as a file arriving, so the service wakes for every one. Copy the thumbnail first and it finds one already there when the picture follows, and does nothing.
Exports and imports? Per-account folder, self-deleting after ninety days Never copied, kept indefinitely, no lifecycle rule at all Per-account folder, old ones never copied, and the ninety-day rule scoped to the export prefixes only. A bucket-wide rule in a shared bucket would reach the photos.
How long do the old folders stay? Indefinitely; a report decides when A year minimum, retired on version numbers Both. A year is the floor, the report is the evidence, and clearing them out stays a separate decision this plan does not take.
How many steps? Six, three gates Nine, three gates Six, three gates. The three that go are the bucket you no longer create, the address you no longer serve, and the second thumbnail service you no longer write.

The shape of it

Two folder shapes live in one bucket for the whole migration. Files are copied from the flat shape into the account shape, and the flat original is never touched — so the legacy key stays a working rollback target right to the end.

Why this is safe in one namespace: the code writes to 70 folder prefixes today, across 43 distinct top-level names — inventory/, document-manager/, categories/, profile/, assets/QR and the rest. Not one of them is img or doc. The new shape occupies two top-level names the bucket has never used, so the two shapes cannot collide and either can be listed on its own. Confirm it once against the live bucket with gsutil ls gs://<bucket>/ before step 1 signs off — the mobile API writes here too.

Legacy shape · keep readable

70 flat prefixes

  • Everything from every account in one folder per kind
  • Untouched by the migration — no deletes, no renames
  • Stops being written to at the last step, not before
copy in place
account by account

Account shape · target

img/ · doc/

  • One folder per account, keyed on the CAP‑number
  • Pictures and documents kept apart
  • Library images shared, outside any account

Both boxes are the same bucket. That is the whole idea: a copy is a same-bucket rewrite, so it needs no network transfer and cannot land in the wrong region — the one class of mistake the other plan had to guard against with a settings checklist.

Where files go

Two new top-level areas, beside the ones already there. The account number does the separating; everything else is there so a path can be read at a glance.

capexpert-prod-bucket/
│
├── img/                                          ← new: pictures
│   ├── CAP-12345678/
│   │   ├── assets/
│   │   │   ├── image1_sticker_uuid.jpg
│   │   │   └── thumb/
│   │   │       └── image1_sticker_uuid.jpg   ← made by the thumbnail service
│   │   └── user/profile/                     ← the person is named in the table, not the path
│   │       ├── me.jpg
│   │       └── thumb/
│   │           └── me.jpg
│   ├── model-library/                        ← no account: shared catalogue
│   │   ├── ct-scanner.jpg
│   │   └── thumb/
│   │       └── ct-scanner.jpg
│   ├── category/                             ← no account: shared catalogue
│   │   ├── imaging.png
│   │   └── thumb/
│   │       └── imaging.png
│   ├── qr/                                   ← belongs to nobody: printed labels
│   │   └── 000123.png
│   └── temp/                                 ← staging, before the user saves
│       └── assets/
│           ├── image1_sticker_uuid.jpg
│           └── thumb/
│               └── image1_sticker_uuid.jpg
│
├── doc/                                          ← new: documents
│   ├── CAP-12345678/
│   │   ├── assets/warranty/warrantyX.pdf
│   │   ├── assets/service-contract/sc-2291.pdf
│   │   ├── ezrfp-quote/quote-4471.pdf
│   │   ├── cad/floor-3-layout.dwg
│   │   ├── w9/vendor-w9.pdf
│   │   ├── purchasing-formulary/formulary-2026.xlsx
│   │   └── user/documents/                   ← the documents table, which has no account column
│   │       └── signed-nda.pdf
│   ├── model-library/
│   │   ├── spec-files/ct-scanner-spec.pdf
│   │   └── mfr-files/siemens-reference.pdf
│   ├── miscellaneous/                        ← the same three kinds, but shared: no account
│   │   ├── cad/standard-room-template.dwg
│   │   ├── w9/capexpert-w9.pdf
│   │   └── purchasing-formulary/baseline-formulary.xlsx
│   ├── global/                               ← belongs to nobody: exports, reports, samples
│   │   └── capture/export/capture-2026-08-31.csv
│   └── temp/
│       └── cad/floor-3-layout.dwg
│
├── inventory/images/                             ← legacy, untouched, still read
├── document-manager/
├── categories/  profile/  assets/QR
└── … 43 legacy top-level names in all

Deciding where a file goes

Work down this list and stop at the first one that fits. The order matters — the earlier questions catch the files that must not be filed under an account.

  1. Is it a report, an export, or an uploaded spreadsheet? It goes under doc/global/ when it belongs to nobody, or the account’s own export folder where it has one. These are always regenerable, so old ones are never copied — and the ninety-day deletion rule applies here and nowhere else.
  2. Does it belong to the shared equipment library rather than a customer? Reference pictures and manuals for a make and model are shared by every account, so no single account owns them. Pictures go under img/model-library/ and img/category/, documents under doc/model-library/spec-files/ and mfr-files/, with no account in the path.
  3. Does it belong to a person rather than an account? Profile pictures and personal documents follow the individual, and one person can act across several accounts. They sit inside the owning account as <tree>/CAP-12345678/user/…, with the person recorded in the table rather than the path. The one feature with no account at all — chatbot uploads — gets a per-user folder at the root of its tree instead: doc/chatbot-uploads/<user>/.
  4. Is it a picture or a document? Pictures go under img/, documents under doc/. This is the only part of the path that is purely about file type.
  5. Which kind of thing is it? The last folder names what the file is for — inventory, warranty, service contract, quote, manual, and so on. This comes from one agreed list, and the list is now settled: 45 upload kinds, each naming exactly one destination, in one file. A new kind that names no destination fails the build rather than picking a folder at random.

The five owners, and which of them get copied

Most files belong to an account. Four kinds do not, and each gets its own shape in the owner slot. This is one setting per upload kind, and it decides the whole path.

OwnerWhere it goesWhat lives thereCopied across?
An account <tree>/CAP-12345678/… Inventory pictures, warranties, service contracts, quotes — the bulk of everything. Yes, account by account.
The shared catalogue img/model-library/…
img/category/…
doc/model-library/…
Equipment models and categories. One copy, seen by every account. Yes, in a single pass — there is no account to loop over.
A person, inside an account <tree>/CAP-12345678/user/… Profile photos, and the documents table, which has no account column at all — it reaches one through the person who created the row. Yes, in the same single pass.
A person, with no account doc/chatbot-uploads/<user>/ Chatbot attachments. Sent to the model the moment they are uploaded, so there is no save for them to wait for. Yes — the only kind that keeps a person in the path.
Nobody doc/global/…
img/qr/…
Data exports, reports, the EULA, sample files, printed QR labels. No — see below.

Why a profile photo sits under the account but is tagged to the person. The path puts it inside the owning account, so deleting that account’s folder takes its people’s files with it. The person is recorded in the table instead of the path, which keeps “delete everything belonging to this person” an indexed lookup rather than a search through path text.

One consequence to expect at cutover: the upload function’s folder setting defaults to the profile folder. Any caller that forgets to say which folder it wants will now fail loudly for want of a person, instead of quietly filing an equipment picture among the profile photos. That is the better failure, but it will expose callers that were leaning on the default.

Exports and imports are the cheapest part of this work. Ten of the seventy folder prefixes are exports or reports, and four more are one-shot import files — a fifth of the list. Every one can be regenerated from live data, and nobody re-downloads last quarter’s spreadsheet.

So new ones write to the new shape from cutover onward, old ones are never copied, and they expire on the ninety-day rule the earlier running order asked for. That rule must be attached to the export prefixes and nothing else: this is a shared bucket now, and a rule written at the bucket level would start deleting the originals that older phone apps depend on. Everything else in this plan can be reversed. That cannot.

How a read finds a file

Three tiers, tried in order. The first covers nearly everything, the second covers the cases the first gets wrong, and the third is what makes the whole migration invisible while it runs.

Today each part of the system adds the folder name itself, at the moment it needs it, in dozens of separate places — around 40 in the API, roughly 170 in the website, and about 70 in the mobile app. That is what has to stop: one shared rule replaces all of it, and once it exists, moving a file becomes a data change rather than a code change.

The rule tolerates both shapes at once. That is what makes step 5 routine instead of frightening: there is no moment where a file exists somewhere the code will not look.

And because installed phone apps display whatever location the mobile service sends them, fixing the mobile service reaches every phone already in customers’ hands, with no app-store release.

  1. Use the path the record holds

    img/CAP-12345678/assets/image1.jpg

    Anything containing a slash is already a complete location — use it as it stands. This is the tier that grows as records are saved, and the one the optional backfill would fill in wholesale.

  2. Look it up in the migration table

    final_path ?? file_path

    For a bare file name whose owner the reader cannot work out: a transferred item, a marketplace listing, the documents table, a shared chat room, an emailed link. One lookup on the name, and the row answers where the file is, not only where it is headed.

  3. Fall back to the original folder

    inventory/images/image1.jpg

    Every older record, and everything not yet copied. Same bucket, legacy key. Removing this fallback is the very last thing anyone does, and only once no app version still needs it.

The one new table

Additive, holds no primary data, and can be emptied and rebuilt at any time. With the backfill in, it is no longer what reads depend on — but it is still needed, for three jobs nothing else can do.

  1. It holds the staged file

    img/temp/assets/pump.jpg

    An upload happens before the record that will reference it exists, so at that moment there is nowhere else to write anything down. The row is handed an id when the upload link is issued, and the save hands that id back — that is the only link between a staged file and the record that ends up pointing at it.

  2. It is the copy queue

    is_committed = true AND is_copied = false

    One row per file still owed a copy, with the reason it was skipped if it was. A one-shot script that dies halfway leaves nothing to resume from; this survives the restart, and it can answer “how much is left, for whom” at any moment.

  3. It keeps abandoned uploads out of account folders

    is_committed

    Five pictures picked and then never saved must never reach an account’s folder — no record will ever reference them. One flag cannot tell that case apart from a real save whose copy failed. Two can.

Two unique guards, and neither is on the account plus the file name. Anything uploaded before its owning account is known has no destination to key on, and the database treats two empty values as different from each other — so a single guard on the destination would have quietly permitted unlimited duplicates of precisely the files staging exists for.

So it is split in two: one row per destination for every row that has one, and one row per staging address while it does not. Between them every row is covered at every moment of its life. A transferred file still gets two rows — one source, two destinations — which is correct, and which a guard on the file name alone would have blocked.

Seventeen columns

Every one answers a question nothing else in the table can, and three of them exist because of failure cases rather than features.

ColumnWhat it holds
idThe row’s own number. Handed back when an upload link is issued, and given back on save so the right row can be marked.
file_nameThe bare name, exactly as the existing tables store it.
file_pathWhere the file actually is, as uploaded. inventory/images/1.jpg for anything taken before the switch, a staged or account-shaped key after it. Written once and never rewritten: this is what the copier reads from, and without it the copier has to guess, because a finished path like img/CAP-1/assets/a.jpg has lost which original folder it came from — every picture folder collapses into assets.
source_pathWhere the file belongs — img/CAP-12345678/assets/1.jpg, img/model-library/1.png. Always account-shaped and never a staging path, because staging is where a file sits, not where it belongs. Carries {SCOPE} in the account slot until the account is settled, and the save fills that in.
final_pathWhere the file actually got to, written the moment the copy completes. Empty until then — so this being empty and “not copied yet” are two ways of saying one thing, rather than two facts to keep in step.
account_idThe owning account, as the ordinary numeric id every other table already uses. Empty for three of the five owner kinds.
account_capex_idThe account’s CAP‑number, kept beside the numeric id purely so the save can replace {SCOPE} without looking the account up. One column that removes a join from the busiest write in the migration.
user_idThe other owner. Set for profile photos, personal documents and chatbot uploads, empty otherwise. This is the column that answers delete everything belonging to this person — the account column cannot, and searching the path text is not an answer.
owner_scopeOne of exactly five words — account, library, user, user-root, global — naming which owner kind this row has. Never empty, which is what makes “how many catalogue files are still pending” a question you can actually ask.
module_nameWhich part of the app the file came from. Recorded when the upload link is handed out.
is_committedThe record referencing this file was saved. The flag that makes staging work — see below.
is_copiedThe file has reached its final place. This and is_committed are the two flags the scheduled job reads — one file still owed a copy is is_committed = true AND is_copied = false.
skipped_reasonWhy a file could not be copied. Clearing it puts the file back in the queue.
created_byWho asked for the upload link. Distinct from user_id, which is the person the file belongs to and is empty unless the file is personal — one records the actor, the other the owner, and they are the same person only by coincidence.
created_at · updated_at · deleted_atThe usual three.

Three path columns, not two nullable ones taking turns. An earlier shape had a temp path and a destination that swapped roles depending on which was empty — and the live table proved the cost: two columns held the identical value on every row, so the table could not say where an object was without also reading a flag. Now the source is the source, the destination is the destination, and a destination that is empty is the staging state. The whole read rule is is_copied ? final_path : file_path.

Four uploads, four different paths

Which shape is live, and whether the upload is itself the save, decide everything. This is the table to check an implementation against.

Shape liveUpload is the save?file_pathsource_pathfinal_path
legacynoinventory/images/1.jpgimg/{SCOPE}/assets/1.jpg
legacyyesinventory/images/1.jpgimg/CAP-12345678/assets/1.jpg
accountnoimg/temp/assets/1.jpgimg/{SCOPE}/assets/1.jpg
accountyesimg/CAP-12345678/assets/1.jpgimg/CAP-12345678/assets/1.jpgimg/CAP-12345678/assets/1.jpg

While the legacy shape is live, an upload keeps its flat key whatever the caller asked for — the object has to be where today’s reads look for it, and an account-shaped key written before reads tolerate it is a file nothing can find. Only the destination column looks ahead. And no row needs to be told which shape it is in: file_path says so on its face, because the two shapes share no top-level name.

What a row actually looks like

One of each owner kind, plus one still in staging. final_path is left out because it is empty on every one of them — nothing has been copied yet, and it records where a file got to, not where it is headed.

owner_scopefile_namefile_pathsource_pathaccount_capex_iduser_id
accountimg1.jpginventory/images/img1.jpgimg/CAP-12345678/assets/img1.jpgCAP-12345678
account · stagedpump.jpgimg/temp/assets/pump.jpgimg/{SCOPE}/assets/pump.jpg
libraryct-scanner.jpginventory/images/ct-scanner.jpgimg/model-library/ct-scanner.jpg
userme.jpgprofile/me.jpgimg/CAP-12345678/user/profile/me.jpgCAP-123456784471
user-rootscan.pdfchatbot-uploads/4471/scan.pdfdoc/chatbot-uploads/4471/scan.pdf4471

Read the first column and you know how to read the rest of the row. That is the whole point of it: the account column is filled in on two rows out of five, so grouping or filtering by it alone silently omits the other three — and only user_id can find one person’s files, because the path does not name them.

Settle these before step 1

Four things had to be agreed before any of this could start. Three are now answered; one is still a number nobody has.

The six steps

Steps 1 to 3 change nothing a user can see and can ship on ordinary release days. Behaviour changes at step 4. Nothing is deleted or moved at any point, in any step.

  1. 1

    Database: store the complete file path

    Change the database so each record holds a file’s complete location rather than just its name, and fill in the existing records with the location each file is at right now — not where it is going. Add the migration table in the same step. Nothing moves.

    • Inventory photos and sticker images — the two largest sets by far
    • Warranties, service contracts, quotes, manuals and the rest
    • Skip anything already holding a complete location; a good number already do
    • Leave anything you are unsure about untouched, and write down that you deferred it
    • The migration table: one migration, seventeen columns, two partial unique guards. Reverse it once before relying on it — a migration you have never reversed is a migration you cannot reverse

    Why first. Once the database knows exactly where each file lives, moving one becomes a checkable operation instead of a guess. Recording the current location, separately from changing it later, is the single most important decision in the plan.

    Three things to watch. Photo order carries meaning — the first is the preview shown on thousands of records, so any update that reshuffles them changes what customers see. Two columns that look identical are stored differently, one a proper list and one a single line of text with commas, and an update written for one corrupts the other. And a few columns may hold web addresses pointing at other companies’ sites: prefixing one of those destroys it with no way back. The rule that protects all three is leave anything that already contains a slash alone.

    Done when The update can be run twice with no ill effect, the order of each account’s photos is unchanged, and reversing it returns the data exactly to its previous state — verified by running the reversal, not by reading it.

    • dbMigrations
    • backendApi
  2. 2

    Code: every repo uses the complete file path

    Today each part of the system adds the folder name itself, at the moment it needs it, in dozens of separate places. Replace all of it with one shared rule — the three tiers above — and have every one of those places call it instead.

    • Build the shared rule once, and cover both cases with tests
    • Web service — around 40 places build a location by hand. Count them first, confirm zero at the end
    • Mobile service — three live versions run side by side and each carries its own copy, so all three need it
    • Website — roughly 170 places build locations in the browser
    • This is also where the module name starts being recorded when an upload link is handed out, and where the account is threaded through the upload endpoint, which today does not receive it at all

    Two things come out of this. Once the shared rule exists, changing where files live becomes a data change rather than a code change — which is what makes step 5 routine instead of frightening. And because installed phone apps display whatever location the mobile service sends them, fixing the mobile service reaches every phone already in customers’ hands, with no app-store release.

    43 hand-built public URLs, in 21 files, paste the bucket name and a folder together to make a public address. With one bucket every one of them still points at the right bucket — but they still build a legacy key, so each has to go through the shared rule or it will 404 on anything uploaded after step 4.

    Done when Nowhere in any repo builds a file location by hand, all builds and checks pass, pictures and documents still display everywhere they did before, and older records holding only a file name still work in the live system.

    • backendApi
    • capExpertApp
    • mobileApi
    • mobileApp
  3. Gate 1 — stop here

    Steps 1 and 2 must be finished and live before anything else. This is the point of no ambiguity: the system now knows where every file lives, and not one file has moved. Nowhere in any repo builds a location by hand; all builds, checks and tests pass across the web service, mobile service and website; older records holding only a file name still display correctly in the live system; nothing the website depends on has been lost from the published interface; the step-1 reversal has been performed once; and zero files have moved.

  4. 3

    Thumbnail service: accept the new structure

    Update the live service to watch the new folders as well as the old ones, and to place each thumbnail beside the picture it came from rather than in one shared pile. Deploy it, then confirm an upload to an old folder still gets a thumbnail exactly as before. It must be live before step 4 — the service only reacts to files arriving, and a picture that lands before it is ready gets no thumbnail and no second chance.

    • One rule covers every picture branch: watch img/ and insert thumb/ before the file name, so account pictures, catalogue pictures, category pictures and profile photos are all handled without a case for each
    • Skip a thumbnail that already exists — the single most valuable line in it, and what makes step 5 free of image processing
    • Match the width and quality the current service uses. Different-sized thumbnails change the appearance of every list view the moment copying starts, and that gets reported as a bug

    This service does two jobs and only one is obvious. It makes thumbnails, and it is also what makes pictures publicly viewable at all. If uploads start going somewhere it is not watching, they lose their thumbnail and stop being viewable on the website.

    Two ways this bites. It currently throws away everything but the bare file name when deciding where to put a thumbnail — left alone, every account’s thumbnails land in one folder and quietly overwrite each other wherever two accounts share a file name. It would pass a quick test and surface months later. And it is triggered by every file arriving and filters in its own code, so the only thing stopping it making a thumbnail of a thumbnail, forever, is that check. Test it against a thumbnail path before deploying; getting it wrong bills per invocation.

    Done when The updated service is deployed and running — not merely written — a test upload to an old folder still produces a thumbnail, and a thumbnail dropped into a new folder produces nothing at all.

    • Cloud Function — outside the repos
  5. 4

    New uploads go to the new structure

    Flip the shape setting and redeploy. From this moment anything new lands in the account’s own folder; every existing file is still exactly where it was, and both are readable.

    • Uploads land under img/temp/ or doc/temp/ first and only move to the account folder when the record is saved. Each of the 24 places that hands out an upload link has to hand the id back on save
    • Phone uploads — the mobile service already knows which account is uploading; it simply is not using that when choosing the folder
    • Reports, exports and uploaded spreadsheets — new ones go to per-account folders with the ninety-day rule attached to those prefixes only. Old ones are not copied; they expire where they are
    • Fix the extension bug while you are here: the API strips punctuation before splitting off the extension, so report.zip is stored as reportzi. The mobile API already fixed this — port that fix rather than writing a second one

    The upload step already works out the complete location and hands it back — the rest of the system just discards it. That discarding is the original cause of this whole project.

    Two things to watch. Phones store photos while offline and send them later, sometimes weeks later, and those carry only a file name — so the system has to keep accepting bare file names indefinitely, with no end date. And the ninety-day deletion must be aimed only at reports and spreadsheets: in a shared bucket a rule written one level too high reaches the photo and document folders and starts deleting originals that older phone apps depend on. Everything else in this plan can be reversed. That cannot.

    Done when A test upload from both the website and a phone lands in the correct account folder and gets a matching thumbnail. The thumbnail is the proof that step 3 was deployed properly, and an empty temp/ prefix afterwards is the proof that the save wiring works.

    • backendApi
    • capExpertApp
    • mobileApi
  6. Gate 2 — stop here

    Confirm new uploads work end to end before touching anything old. New uploads from both the website and a phone land in the right account folder; each gets a thumbnail and appears correctly to the customer; a photo queued offline on a phone before the change still syncs; upload five pictures and abandon them, and confirm none of them ever leaves temp/; the deletion rule is confirmed to touch reports and spreadsheets only; and old files are all still exactly where they were.

  7. 5

    Copy existing files into the new structure

    One account at a time, smallest first. For each account, read the records that reference files, queue them, and let the scheduled job work through them: confirm the file is there and note whether it is public or private, copy it to the new folder keeping that same setting, confirm the copy is identical, update the record to point at the new location, and leave the original exactly where it is.

    • Copy the thumbnail before its picture. A copy counts as a file arriving, so every one wakes the thumbnail service. Thumbnail first and it finds one already there when the picture follows, and does nothing. The other way round and it remakes every thumbnail in the system
    • Two passes per batch, not one. The copier runs several files at a time, so a thumbnail and its own picture could otherwise be in flight together and land in the wrong order. All the thumbnails, then all the pictures
    • A missing thumbnail fixes itself: mark that row skipped, copy the picture, and the service makes a fresh one because it finds none
    • Watch the skipped count, not just the copied count. A rising skipped count means the source paths are wrong and every further account will hit the same fault. Stop and read the reasons
    • The scheduled work is a job class triggered over HTTP, not a decorator — this codebase has no self-scheduling jobs, and there is a working example to copy

    Copy before updating the record, never the other way round. If the job stops halfway, every record still points at a file that exists. And one account at a time means a problem affects one customer and can be undone for that customer alone.

    Expect shared files. Where two accounts have ended up sharing one file — usually because equipment was transferred between them — it is copied into both accounts’ folders. Moving it would take it away from one of them. This is also why a guard on the file name alone would have been wrong: one source, two destinations, two rows.

    Done when One small account is fully copied, the copies are confirmed identical to the originals, every original is still present, and the customer sees no difference at all.

    • backendApi
    • Cloud Scheduler
  8. Gate 3 — stop here

    Prove it on one account, including undoing it. Do not begin copying at scale until the reversal has actually been performed once, not merely described.

    Real storage numbers recorded, and time and cost estimated from them; one account fully copied and verified with all originals still present; that account’s pictures displaying correctly on both the website and a phone; new uploads for that account still getting thumbnails; and the reversal rehearsed — that account pointed back at the old locations and everything confirmed working again.

  9. 6

    Keep the old folders, and watch them

    Stop writing to the legacy folders, and start recording who still reads them. Log which app version made each request for a picture, and build a report answering one question: which app versions have asked for an old location in the last thirty days?

    • There is no way to make half a bucket read-only with one setting, so “frozen” here is a code rule — nothing writes a legacy key any more — optionally backed by a permission condition on those prefixes, and evidenced by the report rather than by the bucket refusing writes
    • Set the retirement date from version numbers, not the calendar. Pick a date and you will cut off real users
    • A year is the floor. Only once those builds have drained does anyone remove the third read tier, retire the old setting, or consider deleting anything

    The old folders can only ever be cleared out once no app still asks for them. There is no date at which that becomes true — only evidence. Without this measurement they must be kept indefinitely by default, which is a perfectly acceptable outcome and costs only storage.

    Done when The report exists and the agreed condition is written down: thirty consecutive days with no app version requesting an old location. Clearing the folders out remains a separate decision that this plan does not take.

    • backendApi
    • mobileApi

Where the code stands against this plan

Some of this is already built on the branch, and one part of it was built for two buckets. This is the difference, file by file.

What the code does todayWhat this plan needs
Two bucket names and a useOldBucket flag, read from the environment One bucket name and a shape flag. The second bucket key goes away, and the flag is honestly named for what it now switches. For a trial before the rename, pointing the second key at the same bucket makes the existing code behave exactly as this plan describes.
An upload keeps its flat key whenever the old bucket is live Unchanged in intent — the condition just becomes “the legacy shape is live”. This is the line that keeps a pre-cutover upload where today’s reads look for it.
Each new row records is_new_bucket Column dropped; nothing to record. The shape is readable from file_path.
The copier picks a source bucket from that flag and always writes to the second bucket One bucket for both ends. The “delete the staged object afterwards” test becomes “was the source under <tree>/temp/”.
A liveBucketName getter switches between the two Nothing to switch. QR paths and view links read the one bucket.
The registry column is called new_bucket_source_path Renamed to source_path. The migration is edited in place and the table holds no primary data, so this is one line plus a re-run — the entity and its four call sites follow.
Existing tables hold bare file names Step 1, and the largest piece of new work in this plan. Nothing on the branch backfills complete paths today — the branch was built on the premise that no existing column would change.

Do not do these

Each one breaks something you cannot get back.

Seven ways to lose something permanently

  • Do not delete from the old folders while copying. Copy only. Those originals staying byte-identical is what makes every rollback a setting change instead of a restore.
  • Do not write a lifecycle rule above the export folders. One bucket now holds both shapes, so a rule one level too high starts deleting photos that installed phone apps still read.
  • Do not change what is public and what is private. A copy does not carry permissions across, so a public file has to be made public again explicitly — but the status itself does not change.
  • Do not move uploads before reads tolerate both shapes. That single ordering mistake creates files nothing can find.
  • Do not add a folder prefix to anything that already contains a slash. The document manager already stores complete paths, and a few columns hold addresses on other companies’ sites. Prefixing either produces a path that points nowhere.
  • Do not let the thumbnail service keep only the bare file name. Every account’s thumbnails would land in one folder and overwrite each other wherever two accounts share a name.
  • Do not reshuffle a photo array while backfilling it. The first entry is the preview customers see on thousands of records.

What does not change

Worth stating plainly, because the list of things being left alone is the reason this plan is small.

If something goes wrong

Every step before the last is one setting away from where it started.

What you seeWhat you do
New uploads landing in the wrong placeUnset the shape flag and redeploy. Uploads go back to the legacy folders; whatever reached the new ones just sits there harmlessly.
Pictures not loadingThe third read tier already covers every file not yet copied. Check the shared rule is reaching it, and that older records still resolve.
Thumbnails appearing in the wrong folder, or multiplyingRoll the thumbnail service back to its previous revision — it is deployed separately from everything else, which is exactly why step 3 stands alone.
The copier behaving badlyPause the scheduled job. The originals are untouched, so nothing has been lost — only postponed.
Wrong rows in the migration tableClear the skipped reason to re-queue, or empty the table and rebuild it. It holds no primary data.
The step-1 backfill wrote something wrongRun its reversal — which is why the gate requires it to have been run once for real, on a copy, before anything else proceeds.

What this plan deliberately does not do

Keeping files separated by account was partly about restricting who can see what. This plan does not achieve that, and it is worth being clear about why.

Pictures are public today — anyone with the address can open one — and separating them into folders does not change that. The thumbnail service makes every picture public as it is created, and folder-level permissions cannot restrict a file that is already public to everyone.

What this plan does is make that restriction possible. It is three further pieces of work after this one, each with its own reversal.

Not in this plan

  • Nothing is deleted from the old folders, at any point
  • Pictures are not switched from public to private
  • Folder-level permissions are not applied
  • The thumbnail service is not stopped from making pictures public