Storage reorganisation · running order

Migration Running Order

What happens, in what order, and how you know each step finished. The whole plan is built so that nothing is ever deleted or moved — every file is copied, and every original stays exactly where it is.

Why the order matters more than the work

Files are currently kept in one shared set of folders, with the folder name added by the software at the moment it is needed. We are separating them by account. The risky part is not the copying — it is that customers are running app versions we cannot update, and those apps expect the old locations. So the sequence is arranged to keep the old locations working forever, and every step is reversible on its own.

  • 1 Database
  • 2 Code, all repos
  • 3 Thumbnail service
  • 4 New uploads
  • 5 Copy existing files
  • 6 Keep and watch the old folders

How the new structure works

Every location is built from the same four parts, in the same order. Once you can read one, you can read all of them.

imgpictures or documents
/
CAP-123456which account owns it
/
inventorywhat kind of thing it is
/
image1.jpgthe file itself
  • img/CAP-123456/inventory/image1.jpg
  • img/CAP-123456/inventory/thumbnail/image1.jpg  ← thumbnails sit beside the picture they came from
  • doc/CAP-123456/warranty/warrantyX.pdf
  • doc/CAP-123456/service_contracts/contractY.pdf

Deciding where a file goes

Work down this list and stop at the first one that fits. The order matters — the earlier questions catch the files that must not be filed under an account.

  1. Is it a report, an export, or an uploaded spreadsheet?It goes to exports or imports under the account, and deletes itself after ninety days. These are always regenerable, so they are never copied or kept.
  2. Does it belong to the shared equipment library rather than a customer?Reference pictures and manuals for a make and model are shared by every account, so no single account owns them. They go under models, with no account in the path.
  3. Does it belong to a person rather than an account?Profile pictures and feedback attachments follow the individual, and one person can act across several accounts. They go under users, keyed to the person.
  4. Is it a picture or a document?Pictures go under img, documents under doc. This is the only part of the path that is purely about file type.
  5. Which kind of thing is it?The last folder names what the file is for — inventory, warranty, service contracts, quotes, manuals, and so on. This comes from one agreed list, settled before work starts.
The old folders do not go away. They stay exactly as they are, readable forever, and nothing is ever written to them again or removed from them. Phone apps already installed on customers' devices will keep asking for the old locations for as long as those versions are in use, and keeping the originals is what lets them keep working.

Settle these four things first

Each one changes the work rather than following from it, so guessing costs more than asking.

  1. Measure the real storageHow many files there are, how much space they take, the longest file name, and how often one file is shared by two accounts. The test copy of the database cannot answer this — it holds a small handful of files repeated many thousands of times, so its totals are meaningless. Every estimate of time and cost depends on the real numbers.
  2. The folder naming listAgree the exact name for each type of file — one settled list, before anyone starts. Renaming a folder later means copying everything twice.
  3. Who owns the thumbnail serviceThumbnails are produced by a small separate service that nobody on the main team currently maintains. Find its owner and confirm who can deploy it.
  4. Whether old thumbnails are copied or simply remadeThe service can rebuild any thumbnail from the original picture. Remaking them is probably cheaper and simpler than copying hundreds of thousands of small files.
Step 1

Database: store the complete file path

No files move

Record where every file already lives

Change the database so that each record holds a file’s complete location rather than just its name. Fill in the existing records with the location each file is at right now — not where it is going. Nothing moves in this step.

  • Inventory photos and sticker images — the two largest sets by far
  • Warranties, service contracts, quotes, manuals and the rest
  • Skip anything already holding a complete location; a good number already do
  • Leave anything you are unsure about untouched and write down that you deferred it

This is the step that makes everything afterwards safe. Once the database knows exactly where each file lives, moving one becomes a simple checkable operation instead of a guess. Recording the current location, separately from changing it later, is the single most important decision in the plan.

Done whenThe update can be run twice with no ill effect, the order of each account’s photos is unchanged, and reversing it returns the data exactly to its previous state.
Three things to watch. The order photos are stored in matters — the first is the one shown as the preview, so any update that reshuffles them changes what customers see on thousands of records. Two places that look identical are stored differently, one as a proper list and one as a single line of text with commas, and an update written for one corrupts the other. And a few places may hold web addresses pointing at other companies’ sites rather than our own files — adding our folder name to one of those destroys it with no way back.
Step 2

Code: every repo uses the complete file path

Still no files move

Stop the software working out locations for itself

Today each part of the system adds the folder name itself, at the moment it needs it, in dozens of separate places. Replace all of that with one shared rule: if the database holds a complete location, use it as it is; if it holds only a file name, treat it as an older record and point it at the original shared folder.

  • Build the shared rule once — and cover it with tests for both cases
  • Web service — replace every place it builds a location by hand; count them first, confirm zero at the end
  • Mobile service — three live versions run side by side and each carries its own copy, so all three need it
  • Website — roughly a hundred and seventy places build locations in the browser

Two things come out of this step. Once the shared rule exists, changing where files live becomes a data change rather than a code change — which is what makes step 5 routine instead of frightening. And because installed phone apps display whatever location the mobile service sends them, fixing the mobile service reaches every phone already in customers’ hands, with no app store release.

Done whenNowhere in any repo builds a file location by hand, all builds and checks pass, pictures and documents still display everywhere they did before, and older records holding only a file name still work in the live system.

Gate 1 — stop here

Steps 1 and 2 must be finished and live before anything else

This is the point of no ambiguity: the system now knows where every file lives, and not one file has moved.

Step 3

Thumbnail service: accept the new structure

Must be live before step 4

Teach it both the old and the new locations

Update the thumbnail service to watch the new folders as well as the old ones, and to place each thumbnail beside the picture it came from rather than in one shared pile. Deploy it, then confirm an upload to an old folder still gets a thumbnail exactly as before.

This service does two jobs and only one is obvious. It makes thumbnails, and it is also what makes pictures publicly viewable at all. If uploads start going somewhere it is not watching, they lose their thumbnail and stop being viewable on the website.

Done whenThe updated service is deployed and running — not merely written — and a test upload to an old folder still produces a thumbnail.
Watch for this. The service currently throws away everything but the bare file name when deciding where to put a thumbnail. Left as it is, every account’s thumbnails end up in one folder and quietly overwrite each other wherever two accounts happen to share a file name. It would pass a quick test and surface months later.
Step 4

New uploads go to the new structure

Existing files untouched

Point every way of adding a file at the account’s own folder

From here on, anything new lands in the right place. Existing files are still exactly where they were.

  • Website and web service uploads — place the file in that account’s folder, and make sure the system records the full location it was given rather than throwing it away and keeping only the name
  • Phone uploads — the mobile service already knows which account is uploading; it simply is not using that when choosing the folder
  • Reports, exports and uploaded spreadsheets — send new ones to per-account folders and set them to delete themselves after ninety days. Do not copy the old ones; let them expire

The upload step already works out the complete location and hands it back — the rest of the system just discards it. That discarding is the original cause of this whole project. The reports and spreadsheets are the cheapest part of the work: they can always be regenerated, nobody looks at an old one, and they very likely account for a large share of the storage bill.

Done whenA test upload from both the website and a phone lands in the correct account folder and gets a matching thumbnail. The thumbnail is the proof that step 3 was deployed properly.
Two things to watch. Phones store photos while offline and send them later, sometimes weeks later, and those carry only a file name — so the system has to keep accepting bare file names indefinitely, with no end date. And the automatic ninety-day deletion must be aimed only at reports and spreadsheets: if it reaches the photo or document folders it will start deleting originals that older phone apps still depend on. Everything else in this plan can be reversed; that cannot.

Gate 2 — stop here

Confirm new uploads work end to end before touching anything old

Step 5

Copy existing files into the new structure

Copy only — originals stay

One account at a time

For each account, work through its files one by one: check the file is there and note whether it is public or private, copy it to the new folder keeping that same setting, confirm the copy is identical, update the record to point at the new location, and leave the original exactly where it is.

Copying before updating the record, never the other way round, means that if the job stops halfway every record still points at a file that exists. Running one account at a time means a problem affects one customer and can be undone for that customer alone.

Done whenOne small account is fully copied, the copies are confirmed identical to the originals, the originals are still present, and the customer sees no difference at all.
Expect shared files. Where two accounts have ended up sharing one file — usually because equipment was transferred between them — it must be copied into both accounts’ folders. Moving it instead would take it away from one of them.

Gate 3 — stop here

Prove it on one account, including undoing it

Do not begin copying at scale until the reversal has actually been performed once, not merely described.

Step 6

Keep the old folders, and watch them

Nothing is removed

Start recording which app versions still use the old locations

Log which app version made each request for a picture, and build a simple report answering one question: which app versions have asked for an old location in the last thirty days?

The old folders can only ever be cleared out once no app still asks for them. There is no date at which that becomes true — only evidence. Without this measurement the old folders must be kept forever by default, which is a perfectly acceptable outcome and costs only storage.

Done whenThe report exists and the agreed condition is written down: thirty consecutive days with no app version requesting an old location. Clearing the folders out remains a separate decision that this plan does not take.

What this plan deliberately does not do

Keeping files separated by account was partly about restricting who can see what. This plan does not achieve that, and it is worth being clear about why. Pictures are currently public — anyone with the web address can open one — and separating them into folders does not change that. The thumbnail service makes every picture public as it is created. Folder-level permissions cannot restrict a file that is already public to everyone.

What this plan does is make that restriction possible. It is three further pieces of work after this, each with its own reversal.