CAP-2030-new-bucket-migration · vs origin/development
Every file this branch touches, in three repositories, with what to look at in each. The storage work records every upload in a registry table and files it into the media bucket when the record referencing it is saved — with no change to any existing table or column beyond the one new table.
dbMigrations
1 file · +195 −0
committed · bf57565, 700fa24 + 1 uncommitted
backendApi
29 files · +5136 −120
committed · 53f69c7f8, 7f6edb592 + 5 uncommitted
capExpertApp
98 files · +708 −638
not committed yet
Three ideas, and everything else follows from them.
A registry row per upload. Existing tables keep storing bare file names. The new presigned_url_links row records where the object actually is, where it belongs in the media bucket, and where it eventually got to. That row is both the copy queue and the lookup for the cases where a path cannot be derived from the requesting account.
The destination comes from the module, not the folder. The front end already sent a bucket folder; it now also sends a moduleName, and one map turns that into a path. The same folder can mean an account’s asset or a shared catalogue item, which is why the folder alone was never enough.
Nothing is filed until the record is saved. A save calls commitUploads, which puts each file where it belongs. A move inside the media bucket is awaited, because reads resolve to the final path the moment the record exists. A copy out of the old bucket is queued and runs behind the save, because reads still fall back to the old bucket until it lands.
Uploads nobody ever saves stay in staging permanently. That is the point of is_committed being a separate flag from is_copied.
Eighteen columns. These five carry the design; the rest are ownership, flags and timestamps.
| Column | Holds | Empty? |
|---|---|---|
| file_path | Where the object is, as uploaded. The only column that ever holds a temp path, because staging is where a file sits, not where it belongs. Written once. | never |
| new_bucket_source_path | Where it belongs — always scope-shaped, never a temp path. Carries {SCOPE} in the owner slot until the account is settled. | never |
| final_path | Where it got to, written when the copy completes. | until then |
| is_new_bucket | Whether there is a cross-bucket copy to make at all. | never |
| account_capex_id | The CAP-number, so the save substitutes {SCOPE} with no join. | until known |
Which bucket is live, and whether the upload is itself the save, decide everything.
| Bucket | Upload is the save | file_path | new_bucket_source_path | final_path |
|---|---|---|---|---|
| old | no | categories/x.png | img/{SCOPE}/assets/x.png | — |
| old | yes | categories/x.png | img/CAP-12345678/assets/x.png | — |
| new | no | img/temp/assets/x.png | img/{SCOPE}/assets/x.png | — |
| new | yes | img/CAP-12345678/assets/x.png | img/CAP-12345678/assets/x.png | same |
The scope decides the shape of the path, and it is also what owner_scope groups by, which is why a fifth was worth adding rather than borrowing one.
| Scope | Shape | Who has it |
|---|---|---|
| account | <tree>/<capexId>/<domain>/ | Most things. Assets, quotes, document manager. |
| library | img/<branch>/ doc/<branch>/<domain>/ | Shared catalogue: equipment models, categories, spec files. |
| user | <tree>/<capexId>/user/<domain>/ | Profile photos and the account-less documents table. |
| user-root | <tree>/<branch>/<userId>/ | Captain L2 attachments, which have no account above them. |
| global | <tree>/<branch>/ | Belongs to nobody. QR codes, at img/qr/000123.png. |
QR codes are the case that needed it. They are minted as an unassigned pool, so there is no account when the object is written; their name is derived from the qr id alone on both the read and the write side; and they are printed on physical labels, so the object can never move. Folding them under library would give the same path but count them alongside the catalogue, which is the one question owner_scope exists to answer.
While the old bucket is live, uploads keep its flat layout — rows one and two are identical in file_path. They have to be: today’s reads build categories/<name> from a folder constant, so an object written anywhere else is an object nothing can find. What skipping staging buys before the cutover is the destination, not the location.
Branch: CAP-2030-new-bucket-migration. Committed, with one comment change pending a re-run.
One migration, applied and re-runnable.
| Path | + | − | What to look at |
|---|---|---|---|
| database/migrations/20260831110327-create-presigned-url-links.js | 195 | The whole table: 18 columns, two partial unique guards, six indexes, FKs to accounts and users. The owner_scope comment lists all five scopes, so it needs one more re-run. Edited in place across the session, so re-run it rather than stacking ALTERs. |
Branch: CAP-2030-new-bucket-migration. Two commits plus five files of follow-on work. Of the 5136 added lines, 4,157 are the runbook and 195 the migration copy inside it — the code itself is about 700 lines.
Where an upload is decided, recorded and filed. Read these four first.
| Path | + | − | What to look at |
|---|---|---|---|
| src/shared/entity/presigned-url-links.entity.ts | 173 | NEW. The registry model, column-for-column with the migration. No DI token; callers use the model directly so the export worker threads can too. | |
| src/shared/modules/misc/misc.dto.ts | 26 | 2 | The presign request gains moduleName (required), accountId and skipTemp. |
| src/shared/modules/storage/storage.service.ts | 318 | 13 | The heart of it. getPreSignedUrlForUpload records the row; mediaAddresses derives the upload key and destination together; commitUploads is the one call every save makes; commitUpload moves or copies one file; capexIdFor caches account to CAP-number. qrObjectPath and liveBucketName are the QR read/write pair, and getPreSignedUrlForView takes an optional bucket so QR can move without touching its other callers. |
| src/shared/modules/storage/upload-targets.ts | 208 | NEW. The destination map: 46 modules to a tree/scope/branch/domain, plus finalPrefixFor, tempPrefixFor, the {SCOPE} placeholder and withScope/awaitingScope. Five owner scopes, global among them, which is what puts QR codes at img/qr rather than under an account. A new UploadModule with no entry fails the build. | |
| src/shared/utils/enum/index.ts | 69 | 8 | The UploadModule enum (46 values, the value is the module name) and BucketFolder.ChatbotUploads. |
The bucket becomes a setting, and the model is registered.
| Path | + | − | What to look at |
|---|---|---|---|
| src/shared/core/config/configuration.ts | 4 | gcp.mediaBucketNew and gcp.useOldBucket. No process.env fallback: the bucket comes from config or the API does not boot. | |
| src/shared/core/config/validation.ts | 2 | STORAGE_MEDIA_BUCKET_NEW required, STORAGE_USE_OLD_BUCKET defaulting to true. | |
| src/shared/database/database.providers.ts | 2 | PresignedUrlLinks added to Models so sequelize.addModels registers it. | |
| src/shared/entity/index.ts | 1 | Barrel export for the new entity. |
Each of these puts a file where it belongs once the record referencing it exists.
| Path | + | − | What to look at |
|---|---|---|---|
| src/import/modules/qr-codes/qr-codes.service.ts | 13 | 8 | The QR generator now takes both its bucket and its object path from StorageService, so the writer and the reader cannot drift. Also drops a gcp.bucketName / process.env / hardcoded-default fallback chain. |
| src/primary/modules/chat/services/message/chat-message.service.ts | 9 | commitUploads when a non-text message is stored: the message is the record that references the attachment. | |
| src/primary/modules/chatbot/chatbot.service.ts | 6 | 2 | Captain L2 attachments use UploadModule.CaptainL2Attachment with skipTemp, keeping doc/chatbot-uploads/<userId>/. |
| src/primary/modules/content/equipment-categories/equipment-categories.controller.ts | 8 | Presign removed from update; the response no longer carries signedUrl. The front end owns the upload now. | |
| src/primary/modules/content/equipment-categories/equipment-categories.service.ts | 19 | 21 | commitUploads after create and update. Category images are library data, so no account is resolved. |
| src/primary/modules/equipment-model/equipment-model.service.ts | 4 | commitUploads after create and update, taking the comma-joined images and files columns straight in. | |
| src/primary/modules/inventory-sticker/inventory-sticker.service.ts | 11 | 6 | commitUploads on updateSticker and updateStickerWithInventoryId; the account comes from the inventory row, so accountId was added to that include. |
| src/primary/modules/misc/misc.controller.ts | 12 | 12 | Swagger for the new presign fields. |
| src/primary/modules/misc/misc.service.ts | 21 | 4 | getSignedUrl resolves capexId through storageService.capexIdFor and threads accountId, userId and skipTemp into the context. Its own duplicate lookup and its Redis dependency are gone. |
| src/primary/modules/profile/profile.service.ts | 11 | 8 | commitUploads after the user row is updated, with the profile photo and the user account. |
| src/shared/modules/document-manager/document-manager.service.ts | 48 | 1 | Four writers: create, documentManagerServiceHistory, fileImportsGeneric, facilityDataImport. Each commits after its transaction and writes the returned final path back into file_path, which holds a complete path. |
StorageModule added wherever a consumer is declared. Nest resolves a provider in its declaring module, so every declaring module needs it.
| Path | + | − | What to look at |
|---|---|---|---|
| src/import/modules/chat-message/chat-message.module.ts | 2 | 1 | The import role declares ChatMessageService as well. |
| src/primary/modules/chat/chat.module.ts | 5 | 4 | — |
| src/primary/modules/inventory-sticker/inventory-sticker.module.ts | 2 | 1 | — |
| src/primary/modules/notifications/notifications.module.ts | 2 | 1 | ChatMessageService is declared here too. Missing this is what broke boot: a provider resolves in its declaring module. |
| src/primary/modules/profile/profile.module.ts | 2 | 1 | — |
| Path | + | − | What to look at |
|---|---|---|---|
| docs/superpowers/plans/2026-08-31-new-bucket-migration.md | 4157 | NEW. The runbook: column contract, the four bucket/skipTemp combinations, and eleven tasks. Its migration and entity code blocks are the real files verbatim. |
Present on the branch but unrelated to the storage work. Worth deciding whether they belong in this PR.
| Path | + | − | What to look at |
|---|---|---|---|
| package.json | 1 | 1 | Removes the six inline env vars from start:dev:import. Worth a look: backendApi/CLAUDE.md documents them as deliberately inline there. |
| src/shared/entity/equipment-categories.entity.ts | 3 | 5 | documentManagerId relaxed from allowNull false to true, plus an import reorder. Not part of this feature. |
| tsconfig.json | 5 | 13 | Adds watchOptions.excludeDirectories and reflows include/exclude. Dev-watcher only. |
Branch: CAP-2030-new-bucket-migration. Not committed. Most of the 98 files are one or two lines: the two shared uploaders now require folder and uploadModule, so every parent had to name both.
The two shared components, the module map, and the new path builder.
| Path | + | − | What to look at |
|---|---|---|---|
| src/app/common/components/file/input-file/input-file.component.ts | 6 | 4 | folder and uploadModule are now required inputs, and both are sent on every presign. Removing the folder default is what surfaced the parents that were silently using inventory/images. |
| src/app/common/components/file/upload-multiple-photos/upload-multiple-photos.component.ts | 5 | 4 | Same two required inputs, same presign change. |
| src/app/core/utils/constant.ts | 48 | 1 | UploadModuleEnum, UploadModuleType and FOLDER_UPLOAD_MODULE, the default module per bucket folder (37 entries). |
| src/app/core/utils/upload-targets.ts | 164 | NEW. The front-end mirror of the API destination map, keyed by wire value so the compiler enforces all 46, global scope included. Exports mediaFilePath, mediaThumbPath and storedFilePath, plus the single USE_MEDIA_BUCKET switch. |
Uploads pointed at the right destination, plus two save bugs.
| Path | + | − | What to look at |
|---|---|---|---|
| src/app/modules/content/equipment-category/list/equipment-category-list.component.ts | 50 | 48 | The dialog now presigns and uploads on save and stores the API-generated name, since the API no longer returns a signed URL. A failed upload aborts the save. |
| src/app/modules/content/equipment-model/add-edit/add-edit.component.html | 3 | Images and spec files retargeted from AssetImage/AssetUserManual to ModelImage/ModelSpecFile, so they land under model-library rather than an account. | |
| src/app/modules/content/equipment-model/add-edit/add-edit.component.ts | 22 | 6 | Two fixes: removing the last spec file now clears the column instead of saving the old list back, and sticker photos seeded for reference are excluded from what is saved to the model. |
| src/app/modules/inventory/match-equipment-model-under-review/match-equipment-model-under-review.component.ts | 10 | 2 | Presign switched to ModelImage. |
| src/app/modules/inventory/match-equipment-model/match-equipment-model.component.ts | 10 | 2 | Presign switched to ModelImage. |
Every parent of the two shared uploaders now passes folder and uploadModule, and where the account is known it passes accountId and skipTemp. Mechanical, one to two lines each.
| Path | + | − | What to look at |
|---|---|---|---|
| src/app/common/components/email-compose/email-compose.component.html | 1 | — | |
| src/app/common/components/email-compose/email-compose.component.ts | 3 | 1 | — |
| src/app/common/components/facility/facility-list.component.html | 2 | — | |
| src/app/common/components/facility/facility-list.component.ts | 9 | 2 | — |
| src/app/common/components/file-upload-panel/file-upload-panel.component.ts | 2 | 1 | — |
| src/app/common/components/import-modal/import-modal.component.html | 1 | — | |
| src/app/common/components/import-modal/import-modal.component.ts | 8 | 1 | — |
| src/app/common/components/modal/attachment/attachment-modal.component.html | 1 | — | |
| src/app/common/components/modal/attachment/attachment-modal.component.ts | 3 | 1 | — |
| src/app/common/components/offline-marketplace/offline-marketplace.component.html | 2 | — | |
| src/app/common/components/offline-marketplace/offline-marketplace.component.ts | 3 | 1 | — |
| src/app/common/components/update-sticker-image/update-sticker-image.component.ts | 4 | 2 | — |
| src/app/common/modules/chat/chat-widget/chat-widget.component.ts | 4 | 2 | — |
| src/app/common/modules/file-upload-panel/file-upload-panel.component.ts | 2 | 1 | — |
| src/app/common/modules/inventory/inventory.component.html | 1 | — | |
| src/app/common/modules/inventory/inventory.component.ts | 6 | 3 | — |
| src/app/layout/header/panels/upload-docs-panel/upload-docs-panel.component.html | 2 | 1 | Binds uploadModule. Note this panel uses InventoryUnclassified, which has no module entry - see the open items. |
| src/app/layout/header/panels/upload-docs-panel/upload-docs-panel.component.ts | 8 | 1 | — |
| src/app/modules/cart/cart.component.html | 1 | — | |
| src/app/modules/cart/cart.component.ts | 4 | 1 | — |
| src/app/modules/cart/shipping-price/shipping-price.component.html | 1 | — | |
| src/app/modules/cart/shipping-price/shipping-price.component.ts | 3 | 1 | — |
| src/app/modules/copilot/bid-submit-collective/bid-submit-collective.component.html | 8 | — | |
| src/app/modules/copilot/bid-submit-collective/bid-submit-collective.component.ts | 6 | 3 | — |
| src/app/modules/copilot/bid-submit-collective/table/bid-collective-table.component.html | 1 | — | |
| src/app/modules/copilot/bid-submit-collective/table/bid-collective-table.component.ts | 3 | 1 | — |
| src/app/modules/copilot/bid-submit-tabbing/bid-submit.component.html | 8 | — | |
| src/app/modules/copilot/bid-submit-tabbing/bid-submit.component.ts | 6 | 3 | — |
| src/app/modules/copilot/copilot-action-logs/copilot-action-logs.component.html | 1 | — | |
| src/app/modules/copilot/copilot-action-logs/copilot-action-logs.component.ts | 3 | 1 | — |
| src/app/modules/copilot/copilot-forms/copilot-request-wizard/copilot-request-wizard.component.html | 2 | — | |
| src/app/modules/copilot/copilot-forms/copilot-request-wizard/copilot-request-wizard.component.ts | 3 | — | |
| src/app/modules/copilot/copilot-forms/copilot-request-wizard/wizard-equipment-builder/wizard-equipment-builder.component.html | 4 | — | |
| src/app/modules/copilot/copilot-forms/copilot-request-wizard/wizard-equipment-builder/wizard-equipment-builder.component.ts | 3 | 1 | — |
| src/app/modules/copilot/copilot-forms/equipment-assist/equipment-assist.component.html | 2 | — | |
| src/app/modules/copilot/copilot-forms/equipment-assist/equipment-assist.component.ts | 8 | 5 | — |
| src/app/modules/copilot/copilot-forms/equipment-row-editor/equipment-row-editor.component.html | 2 | — | |
| src/app/modules/copilot/copilot-forms/equipment-row-editor/equipment-row-editor.component.ts | 3 | 1 | — |
| src/app/modules/copilot/launch-rfp/launch-rfp.component.html | 2 | — | |
| src/app/modules/copilot/launch-rfp/launch-rfp.component.ts | 3 | — | |
| src/app/modules/denovo/add/components/questions-input/question-input.component.html | 1 | — | |
| src/app/modules/denovo/add/components/questions-input/question-input.component.ts | 3 | 1 | — |
| src/app/modules/denovo/edit/checkout/po-attachment-modal/po-attachment-modal.component.html | 1 | — | |
| src/app/modules/denovo/edit/checkout/po-attachment-modal/po-attachment-modal.component.ts | 3 | 1 | — |
| src/app/modules/denovo/edit/edit-project-info/edit-project-info.component.html | 4 | — | |
| src/app/modules/denovo/edit/edit-project-info/edit-project-info.component.ts | 3 | 1 | — |
| src/app/modules/denovo/edit/room-view/components/inventory/edit-denovo-inventory.component.html | 2 | — | |
| src/app/modules/denovo/edit/room-view/components/inventory/edit-denovo-inventory.component.ts | 5 | 2 | — |
| src/app/modules/denovo/edit/room-view/room-view.component.html | 1 | — | |
| src/app/modules/denovo/edit/room-view/room-view.component.ts | 3 | 1 | — |
| src/app/modules/denovo/external/external.component.html | 3 | — | |
| src/app/modules/denovo/launch-copilot-view/launch-copilot-view.component.html | 1 | — | |
| src/app/modules/denovo/launch-copilot-view/launch-copilot-view.component.ts | 3 | 1 | — |
| src/app/modules/developer/copilot-quote/list/copilot-quote.component.html | 2 | — | |
| src/app/modules/developer/copilot-quote/list/copilot-quote.component.ts | 5 | 1 | — |
| src/app/modules/feedback/add/add.component.html | 1 | — | |
| src/app/modules/feedback/add/add.component.ts | 3 | 1 | — |
| src/app/modules/inventory/add-edit/add-edit.component.html | 5 | 2 | — |
| src/app/modules/inventory/add-edit/add-edit.component.ts | 6 | 2 | — |
| src/app/modules/inventory/popup/transfer-popup.component.html | 3 | — | |
| src/app/modules/inventory/popup/transfer-popup.component.ts | 3 | 1 | — |
| src/app/modules/inventory/service-contracts-new/list.component.ts | 10 | 2 | — |
| src/app/modules/listing/sell-market-place/sell-market-place.component.html | 1 | — | |
| src/app/modules/listing/sell-market-place/sell-market-place.component.ts | 9 | 6 | — |
| src/app/modules/manage-inventory/components/divest-check-import-modal/divest-check-import-modal.component.ts | 9 | 1 | — |
| src/app/modules/manage-inventory/manage-inventory.component.ts | 10 | 2 | — |
| src/app/modules/manage-users/accounts/accounts.component.html | 2 | — | |
| src/app/modules/manage-users/accounts/accounts.component.ts | 9 | 3 | — |
| src/app/modules/manage-users/accounts/consumables-catalog-import-modal/consumables-catalog-import-modal.component.ts | 9 | — | |
| src/app/modules/manage-users/accounts/facility-data-import-modal/facility-data-import-modal.component.ts | 9 | 1 | — |
| src/app/modules/manage-users/add-account/add-account.component.html | 3 | — | |
| src/app/modules/manage-users/add-account/add-account.component.ts | 17 | 4 | — |
| src/app/modules/manage-users/channel-partner/list.component.ts | 18 | 5 | — |
| src/app/modules/notification/feedback/feedback.component.html | 1 | — | |
| src/app/modules/notification/feedback/feedback.component.ts | 3 | 1 | — |
| src/app/modules/offers/facility-vendor-offer/upload-pa-popup/upload-pa-popup.component.html | 1 | — | |
| src/app/modules/offers/facility-vendor-offer/upload-pa-popup/upload-pa-popup.component.ts | 3 | 1 | — |
| src/app/modules/profile/basic-info/basic-info.component.ts | 6 | 2 | — |
| src/app/modules/system-setting/shipping/shipping.component.ts | 10 | 3 | — |
| src/app/modules/transaction/details/details.component.html | 1 | — | |
| src/app/widgets/divestiture-my-to-dos/divestiture-my-to-dos.component.html | 2 | — | |
| src/app/widgets/divestiture-my-to-dos/divestiture-my-to-dos.component.ts | 50 | 38 | Presign gains moduleName, accountId and skipTemp; the call was reindented into a multi-line form. |
Import reflow and small model additions the sweep touched.
| Path | + | − | What to look at |
|---|---|---|---|
| src/app/modules/denovo/external/external.component.ts | 3 | 12 | Import reflow plus the uploadModuleEnum member the template needs. |
| src/app/modules/manage-inventory/models/inventory-item.model.ts | 1 | accountId added to the row model so a parent can pass it to the presign. | |
| src/app/modules/transaction/details/details.component.ts | 3 | 9 | Import reflow only; the detailExpand animation block was removed by the sweep and restored. |
Dead module, confirmed unreachable before deleting.
| Path | + | − | What to look at |
|---|---|---|---|
| src/app/modules/inventory-library-logs/inventory-library-logs-routing.module.ts | 33 | — | |
| src/app/modules/inventory-library-logs/inventory-library-logs.module.ts | 33 | — | |
| src/app/modules/inventory-library-logs/log/log.component.html | 171 | — | |
| src/app/modules/inventory-library-logs/log/log.component.ts | 196 | — |
What was actually run, so you know what the gates do and do not cover.
ShipmentTrackingInputDto errors come from the gitignored swagger client.shared/ imports no role bucket.HEAD and is identical — no new findings. Ten pre-existing ones remain.StorageService dependency, resolves each module’s providers by import path, and confirms the declaring module can resolve it. Zero gaps. This is what the boot failure taught: typecheck passes on a DI graph that cannot be constructed.folder and uploadModule.{SCOPE} survives a null, an undefined and an empty string.upload-targets.ts asserted the API map and the generated client agree on all 46 modules, in both directions.backendApi/CLAUDE.md, since capExpertApp generates its clients from it. The presign DTO changed, so this needs capturing before merge.Known and deliberate. None of it blocks the old-bucket path, which is what is live today.
accountId except four. Account-scoped uploads therefore record img/{SCOPE}/…. On the new bucket, one with skipTemp now throws at presign rather than writing a broken path — correct, but it will surface as soon as the flag flips.accountId needs an authorization check before it can be trusted.storedFilePath exists and USE_MEDIA_BUCKET is the single switch, but no front-end call site uses it yet. Until they do, flipping the bucket breaks image rendering.qrObjectPath, both use liveBucketName, and the URL cache is keyed by bucket so a cutover re-signs rather than serving the old object for six days. The front end never builds a QR path — it receives a signed URL.upload-docs-panel uses InventoryUnclassified, which has no module entry, so its derived module falls back to document-manager-file and records the wrong destination.document_manager.create with a caller-supplied transaction commits nothing — the record is not durable yet, so the caller must commit its own uploads.createAccount and updateAccount both branch on caller-supplied transactions with several update calls each; guessing the post-commit point there was the wrong trade.package.json one changes local dev for the import role.