CAP-2030-new-bucket-migration · vs origin/development

CAP-2030 Change Ledger

Every file this branch touches, in three repositories, with what to look at in each. The storage work records every upload in a registry table and files it into the media bucket when the record referencing it is saved — with no change to any existing table or column beyond the one new table.

What the change does

Three ideas, and everything else follows from them.

A registry row per upload. Existing tables keep storing bare file names. The new presigned_url_links row records where the object actually is, where it belongs in the media bucket, and where it eventually got to. That row is both the copy queue and the lookup for the cases where a path cannot be derived from the requesting account.

The destination comes from the module, not the folder. The front end already sent a bucket folder; it now also sends a moduleName, and one map turns that into a path. The same folder can mean an account’s asset or a shared catalogue item, which is why the folder alone was never enough.

Nothing is filed until the record is saved. A save calls commitUploads, which puts each file where it belongs. A move inside the media bucket is awaited, because reads resolve to the final path the moment the record exists. A copy out of the old bucket is queued and runs behind the save, because reads still fall back to the old bucket until it lands.

Uploads nobody ever saves stay in staging permanently. That is the point of is_committed being a separate flag from is_copied.

The registry table

Eighteen columns. These five carry the design; the rest are ownership, flags and timestamps.

ColumnHoldsEmpty?
file_pathWhere the object is, as uploaded. The only column that ever holds a temp path, because staging is where a file sits, not where it belongs. Written once.never
new_bucket_source_pathWhere it belongs — always scope-shaped, never a temp path. Carries {SCOPE} in the owner slot until the account is settled.never
final_pathWhere it got to, written when the copy completes.until then
is_new_bucketWhether there is a cross-bucket copy to make at all.never
account_capex_idThe CAP-number, so the save substitutes {SCOPE} with no join.until known

Where an upload lands, in all four combinations the table to validate against

Which bucket is live, and whether the upload is itself the save, decide everything.

BucketUpload is the savefile_pathnew_bucket_source_pathfinal_path
oldnocategories/x.pngimg/{SCOPE}/assets/x.png
oldyescategories/x.pngimg/CAP-12345678/assets/x.png
newnoimg/temp/assets/x.pngimg/{SCOPE}/assets/x.png
newyesimg/CAP-12345678/assets/x.pngimg/CAP-12345678/assets/x.pngsame

Five owner scopes, not four

The scope decides the shape of the path, and it is also what owner_scope groups by, which is why a fifth was worth adding rather than borrowing one.

ScopeShapeWho has it
account<tree>/<capexId>/<domain>/Most things. Assets, quotes, document manager.
libraryimg/<branch>/   doc/<branch>/<domain>/Shared catalogue: equipment models, categories, spec files.
user<tree>/<capexId>/user/<domain>/Profile photos and the account-less documents table.
user-root<tree>/<branch>/<userId>/Captain L2 attachments, which have no account above them.
global<tree>/<branch>/Belongs to nobody. QR codes, at img/qr/000123.png.

QR codes are the case that needed it. They are minted as an unassigned pool, so there is no account when the object is written; their name is derived from the qr id alone on both the read and the write side; and they are printed on physical labels, so the object can never move. Folding them under library would give the same path but count them alongside the catalogue, which is the one question owner_scope exists to answer.

While the old bucket is live, uploads keep its flat layout — rows one and two are identical in file_path. They have to be: today’s reads build categories/<name> from a folder constant, so an object written anywhere else is an object nothing can find. What skipping staging buys before the cutover is the destination, not the location.

dbMigrations

Branch: CAP-2030-new-bucket-migration. Committed, with one comment change pending a re-run.

The table 1 file

One migration, applied and re-runnable.

Path+What to look at
database/migrations/20260831110327-create-presigned-url-links.js195The whole table: 18 columns, two partial unique guards, six indexes, FKs to accounts and users. The owner_scope comment lists all five scopes, so it needs one more re-run. Edited in place across the session, so re-run it rather than stacking ALTERs.

backendApi

Branch: CAP-2030-new-bucket-migration. Two commits plus five files of follow-on work. Of the 5136 added lines, 4,157 are the runbook and 195 the migration copy inside it — the code itself is about 700 lines.

The mechanism 5 files

Where an upload is decided, recorded and filed. Read these four first.

Path+What to look at
src/shared/entity/presigned-url-links.entity.ts173NEW. The registry model, column-for-column with the migration. No DI token; callers use the model directly so the export worker threads can too.
src/shared/modules/misc/misc.dto.ts262The presign request gains moduleName (required), accountId and skipTemp.
src/shared/modules/storage/storage.service.ts31813The heart of it. getPreSignedUrlForUpload records the row; mediaAddresses derives the upload key and destination together; commitUploads is the one call every save makes; commitUpload moves or copies one file; capexIdFor caches account to CAP-number. qrObjectPath and liveBucketName are the QR read/write pair, and getPreSignedUrlForView takes an optional bucket so QR can move without touching its other callers.
src/shared/modules/storage/upload-targets.ts208NEW. The destination map: 46 modules to a tree/scope/branch/domain, plus finalPrefixFor, tempPrefixFor, the {SCOPE} placeholder and withScope/awaitingScope. Five owner scopes, global among them, which is what puts QR codes at img/qr rather than under an account. A new UploadModule with no entry fails the build.
src/shared/utils/enum/index.ts698The UploadModule enum (46 values, the value is the module name) and BucketFolder.ChatbotUploads.

Configuration and registration 4 files

The bucket becomes a setting, and the model is registered.

Path+What to look at
src/shared/core/config/configuration.ts4gcp.mediaBucketNew and gcp.useOldBucket. No process.env fallback: the bucket comes from config or the API does not boot.
src/shared/core/config/validation.ts2STORAGE_MEDIA_BUCKET_NEW required, STORAGE_USE_OLD_BUCKET defaulting to true.
src/shared/database/database.providers.ts2PresignedUrlLinks added to Models so sequelize.addModels registers it.
src/shared/entity/index.ts1Barrel export for the new entity.

Save paths 11 files

Each of these puts a file where it belongs once the record referencing it exists.

Path+What to look at
src/import/modules/qr-codes/qr-codes.service.ts138The QR generator now takes both its bucket and its object path from StorageService, so the writer and the reader cannot drift. Also drops a gcp.bucketName / process.env / hardcoded-default fallback chain.
src/primary/modules/chat/services/message/chat-message.service.ts9commitUploads when a non-text message is stored: the message is the record that references the attachment.
src/primary/modules/chatbot/chatbot.service.ts62Captain L2 attachments use UploadModule.CaptainL2Attachment with skipTemp, keeping doc/chatbot-uploads/<userId>/.
src/primary/modules/content/equipment-categories/equipment-categories.controller.ts8Presign removed from update; the response no longer carries signedUrl. The front end owns the upload now.
src/primary/modules/content/equipment-categories/equipment-categories.service.ts1921commitUploads after create and update. Category images are library data, so no account is resolved.
src/primary/modules/equipment-model/equipment-model.service.ts4commitUploads after create and update, taking the comma-joined images and files columns straight in.
src/primary/modules/inventory-sticker/inventory-sticker.service.ts116commitUploads on updateSticker and updateStickerWithInventoryId; the account comes from the inventory row, so accountId was added to that include.
src/primary/modules/misc/misc.controller.ts1212Swagger for the new presign fields.
src/primary/modules/misc/misc.service.ts214getSignedUrl resolves capexId through storageService.capexIdFor and threads accountId, userId and skipTemp into the context. Its own duplicate lookup and its Redis dependency are gone.
src/primary/modules/profile/profile.service.ts118commitUploads after the user row is updated, with the profile photo and the user account.
src/shared/modules/document-manager/document-manager.service.ts481Four writers: create, documentManagerServiceHistory, fileImportsGeneric, facilityDataImport. Each commits after its transaction and writes the returned final path back into file_path, which holds a complete path.

Module registration 5 files

StorageModule added wherever a consumer is declared. Nest resolves a provider in its declaring module, so every declaring module needs it.

Path+What to look at
src/import/modules/chat-message/chat-message.module.ts21The import role declares ChatMessageService as well.
src/primary/modules/chat/chat.module.ts54
src/primary/modules/inventory-sticker/inventory-sticker.module.ts21
src/primary/modules/notifications/notifications.module.ts21ChatMessageService is declared here too. Missing this is what broke boot: a provider resolves in its declaring module.
src/primary/modules/profile/profile.module.ts21

Documentation 1 file

Path+What to look at
docs/superpowers/plans/2026-08-31-new-bucket-migration.md4157NEW. The runbook: column contract, the four bucket/skipTemp combinations, and eleven tasks. Its migration and entity code blocks are the real files verbatim.

Not part of this feature 3 files

Present on the branch but unrelated to the storage work. Worth deciding whether they belong in this PR.

Path+What to look at
package.json11Removes the six inline env vars from start:dev:import. Worth a look: backendApi/CLAUDE.md documents them as deliberately inline there.
src/shared/entity/equipment-categories.entity.ts35documentManagerId relaxed from allowNull false to true, plus an import reorder. Not part of this feature.
tsconfig.json513Adds watchOptions.excludeDirectories and reflows include/exclude. Dev-watcher only.

capExpertApp

Branch: CAP-2030-new-bucket-migration. Not committed. Most of the 98 files are one or two lines: the two shared uploaders now require folder and uploadModule, so every parent had to name both.

Shared uploaders and the path map 4 files

The two shared components, the module map, and the new path builder.

Path+What to look at
src/app/common/components/file/input-file/input-file.component.ts64folder and uploadModule are now required inputs, and both are sent on every presign. Removing the folder default is what surfaced the parents that were silently using inventory/images.
src/app/common/components/file/upload-multiple-photos/upload-multiple-photos.component.ts54Same two required inputs, same presign change.
src/app/core/utils/constant.ts481UploadModuleEnum, UploadModuleType and FOLDER_UPLOAD_MODULE, the default module per bucket folder (37 entries).
src/app/core/utils/upload-targets.ts164NEW. The front-end mirror of the API destination map, keyed by wire value so the compiler enforces all 46, global scope included. Exports mediaFilePath, mediaThumbPath and storedFilePath, plus the single USE_MEDIA_BUCKET switch.

Retargets and fixes 5 files

Uploads pointed at the right destination, plus two save bugs.

Path+What to look at
src/app/modules/content/equipment-category/list/equipment-category-list.component.ts5048The dialog now presigns and uploads on save and stores the API-generated name, since the API no longer returns a signed URL. A failed upload aborts the save.
src/app/modules/content/equipment-model/add-edit/add-edit.component.html3Images and spec files retargeted from AssetImage/AssetUserManual to ModelImage/ModelSpecFile, so they land under model-library rather than an account.
src/app/modules/content/equipment-model/add-edit/add-edit.component.ts226Two fixes: removing the last spec file now clears the column instead of saving the old list back, and sticker photos seeded for reference are excluded from what is saved to the model.
src/app/modules/inventory/match-equipment-model-under-review/match-equipment-model-under-review.component.ts102Presign switched to ModelImage.
src/app/modules/inventory/match-equipment-model/match-equipment-model.component.ts102Presign switched to ModelImage.

Parent bindings 82 files

Every parent of the two shared uploaders now passes folder and uploadModule, and where the account is known it passes accountId and skipTemp. Mechanical, one to two lines each.

Path+What to look at
src/app/common/components/email-compose/email-compose.component.html1
src/app/common/components/email-compose/email-compose.component.ts31
src/app/common/components/facility/facility-list.component.html2
src/app/common/components/facility/facility-list.component.ts92
src/app/common/components/file-upload-panel/file-upload-panel.component.ts21
src/app/common/components/import-modal/import-modal.component.html1
src/app/common/components/import-modal/import-modal.component.ts81
src/app/common/components/modal/attachment/attachment-modal.component.html1
src/app/common/components/modal/attachment/attachment-modal.component.ts31
src/app/common/components/offline-marketplace/offline-marketplace.component.html2
src/app/common/components/offline-marketplace/offline-marketplace.component.ts31
src/app/common/components/update-sticker-image/update-sticker-image.component.ts42
src/app/common/modules/chat/chat-widget/chat-widget.component.ts42
src/app/common/modules/file-upload-panel/file-upload-panel.component.ts21
src/app/common/modules/inventory/inventory.component.html1
src/app/common/modules/inventory/inventory.component.ts63
src/app/layout/header/panels/upload-docs-panel/upload-docs-panel.component.html21Binds uploadModule. Note this panel uses InventoryUnclassified, which has no module entry - see the open items.
src/app/layout/header/panels/upload-docs-panel/upload-docs-panel.component.ts81
src/app/modules/cart/cart.component.html1
src/app/modules/cart/cart.component.ts41
src/app/modules/cart/shipping-price/shipping-price.component.html1
src/app/modules/cart/shipping-price/shipping-price.component.ts31
src/app/modules/copilot/bid-submit-collective/bid-submit-collective.component.html8
src/app/modules/copilot/bid-submit-collective/bid-submit-collective.component.ts63
src/app/modules/copilot/bid-submit-collective/table/bid-collective-table.component.html1
src/app/modules/copilot/bid-submit-collective/table/bid-collective-table.component.ts31
src/app/modules/copilot/bid-submit-tabbing/bid-submit.component.html8
src/app/modules/copilot/bid-submit-tabbing/bid-submit.component.ts63
src/app/modules/copilot/copilot-action-logs/copilot-action-logs.component.html1
src/app/modules/copilot/copilot-action-logs/copilot-action-logs.component.ts31
src/app/modules/copilot/copilot-forms/copilot-request-wizard/copilot-request-wizard.component.html2
src/app/modules/copilot/copilot-forms/copilot-request-wizard/copilot-request-wizard.component.ts3
src/app/modules/copilot/copilot-forms/copilot-request-wizard/wizard-equipment-builder/wizard-equipment-builder.component.html4
src/app/modules/copilot/copilot-forms/copilot-request-wizard/wizard-equipment-builder/wizard-equipment-builder.component.ts31
src/app/modules/copilot/copilot-forms/equipment-assist/equipment-assist.component.html2
src/app/modules/copilot/copilot-forms/equipment-assist/equipment-assist.component.ts85
src/app/modules/copilot/copilot-forms/equipment-row-editor/equipment-row-editor.component.html2
src/app/modules/copilot/copilot-forms/equipment-row-editor/equipment-row-editor.component.ts31
src/app/modules/copilot/launch-rfp/launch-rfp.component.html2
src/app/modules/copilot/launch-rfp/launch-rfp.component.ts3
src/app/modules/denovo/add/components/questions-input/question-input.component.html1
src/app/modules/denovo/add/components/questions-input/question-input.component.ts31
src/app/modules/denovo/edit/checkout/po-attachment-modal/po-attachment-modal.component.html1
src/app/modules/denovo/edit/checkout/po-attachment-modal/po-attachment-modal.component.ts31
src/app/modules/denovo/edit/edit-project-info/edit-project-info.component.html4
src/app/modules/denovo/edit/edit-project-info/edit-project-info.component.ts31
src/app/modules/denovo/edit/room-view/components/inventory/edit-denovo-inventory.component.html2
src/app/modules/denovo/edit/room-view/components/inventory/edit-denovo-inventory.component.ts52
src/app/modules/denovo/edit/room-view/room-view.component.html1
src/app/modules/denovo/edit/room-view/room-view.component.ts31
src/app/modules/denovo/external/external.component.html3
src/app/modules/denovo/launch-copilot-view/launch-copilot-view.component.html1
src/app/modules/denovo/launch-copilot-view/launch-copilot-view.component.ts31
src/app/modules/developer/copilot-quote/list/copilot-quote.component.html2
src/app/modules/developer/copilot-quote/list/copilot-quote.component.ts51
src/app/modules/feedback/add/add.component.html1
src/app/modules/feedback/add/add.component.ts31
src/app/modules/inventory/add-edit/add-edit.component.html52
src/app/modules/inventory/add-edit/add-edit.component.ts62
src/app/modules/inventory/popup/transfer-popup.component.html3
src/app/modules/inventory/popup/transfer-popup.component.ts31
src/app/modules/inventory/service-contracts-new/list.component.ts102
src/app/modules/listing/sell-market-place/sell-market-place.component.html1
src/app/modules/listing/sell-market-place/sell-market-place.component.ts96
src/app/modules/manage-inventory/components/divest-check-import-modal/divest-check-import-modal.component.ts91
src/app/modules/manage-inventory/manage-inventory.component.ts102
src/app/modules/manage-users/accounts/accounts.component.html2
src/app/modules/manage-users/accounts/accounts.component.ts93
src/app/modules/manage-users/accounts/consumables-catalog-import-modal/consumables-catalog-import-modal.component.ts9
src/app/modules/manage-users/accounts/facility-data-import-modal/facility-data-import-modal.component.ts91
src/app/modules/manage-users/add-account/add-account.component.html3
src/app/modules/manage-users/add-account/add-account.component.ts174
src/app/modules/manage-users/channel-partner/list.component.ts185
src/app/modules/notification/feedback/feedback.component.html1
src/app/modules/notification/feedback/feedback.component.ts31
src/app/modules/offers/facility-vendor-offer/upload-pa-popup/upload-pa-popup.component.html1
src/app/modules/offers/facility-vendor-offer/upload-pa-popup/upload-pa-popup.component.ts31
src/app/modules/profile/basic-info/basic-info.component.ts62
src/app/modules/system-setting/shipping/shipping.component.ts103
src/app/modules/transaction/details/details.component.html1
src/app/widgets/divestiture-my-to-dos/divestiture-my-to-dos.component.html2
src/app/widgets/divestiture-my-to-dos/divestiture-my-to-dos.component.ts5038Presign gains moduleName, accountId and skipTemp; the call was reindented into a multi-line form.

Incidental 3 files

Import reflow and small model additions the sweep touched.

Path+What to look at
src/app/modules/denovo/external/external.component.ts312Import reflow plus the uploadModuleEnum member the template needs.
src/app/modules/manage-inventory/models/inventory-item.model.ts1accountId added to the row model so a parent can pass it to the presign.
src/app/modules/transaction/details/details.component.ts39Import reflow only; the detailExpand animation block was removed by the sweep and restored.

Removed 4 files

Dead module, confirmed unreachable before deleting.

Path+What to look at
src/app/modules/inventory-library-logs/inventory-library-logs-routing.module.ts33
src/app/modules/inventory-library-logs/inventory-library-logs.module.ts33
src/app/modules/inventory-library-logs/log/log.component.html171
src/app/modules/inventory-library-logs/log/log.component.ts196

How this was verified

What was actually run, so you know what the gates do and do not cover.

Open items

Known and deliberate. None of it blocks the old-bucket path, which is what is live today.

Before the bucket is flipped

  • No front-end parent sends accountId except four. Account-scoped uploads therefore record img/{SCOPE}/…. On the new bucket, one with skipTemp now throws at presign rather than writing a broken path — correct, but it will surface as soon as the flag flips.
  • The account decides whose folder a file lands in, and is unchecked. A client-supplied accountId needs an authorization check before it can be trusted.
  • Reads still build paths from a folder constant, except QR codes. storedFilePath exists and USE_MEDIA_BUCKET is the single switch, but no front-end call site uses it yet. Until they do, flipping the bucket breaks image rendering.
  • QR codes are the exception, and are done. Both sides read qrObjectPath, both use liveBucketName, and the URL cache is keyed by bucket so a cutover re-signs rather than serving the old object for six days. The front end never builds a QR path — it receives a signed URL.
  • The copier job is not implemented. Designed as Task 11; deferred copies rely on it to retry.

Smaller, and worth a decision

  • upload-docs-panel uses InventoryUnclassified, which has no module entry, so its derived module falls back to document-manager-file and records the wrong destination.
  • document_manager.create with a caller-supplied transaction commits nothing — the record is not durable yet, so the caller must commit its own uploads.
  • The account logo and asset/biomed tags are not wired. createAccount and updateAccount both branch on caller-supplied transactions with several update calls each; guessing the post-commit point there was the wrong trade.
  • Three files on the branch are unrelated to this work — listed above under Not part of this feature. The package.json one changes local dev for the import role.