# Deploying Plantoo_tech Handover instructions. Written 2026-09-03 for T17; the environment-specific values are marked `«fill in»` rather than guessed, because getting them wrong is worse than leaving them blank. ## What exists today | Environment | State | |---|---| | CI (GitHub Actions, `.github/workflows/ci.yml`) | **Real.** Runs the PHPUnit suite against MySQL 8 + MongoDB 7 service containers on every push and PR to `main`, plus an `env-diff` job that fails the build on an undocumented variable | | Testing / Staging | **Real**, driven by an external deploy tool (`plantoo_tech_testing`, `plantoo_tech_staging`), not by this repo | | `.github/workflows/staging-deploy.yml` | **A deliberate stub.** Gated behind `vars.STAGING_DEPLOY_ENABLED == 'true'`; its deploy and smoke steps echo and exit. Wire it or delete it — do not leave it looking like a pipeline | | Production | **Does not exist** (confirmed 2026-08-19, still true 2026-09-03). Building it is T18 | ## Requirements on the host - PHP **8.3** with `mbstring`, `intl`, `pdo_mysql`, **`mongodb`**, and — for the concurrency suite — `pcntl` + `posix`. Without those last two, `BookingConcurrencyTest` **skips silently** and a build goes green having never run the race it exists for. - MySQL **8** and MongoDB **7**. Both, always: `phpunit.xml` sets `DB_CONNECTION=sqlite`, but that only changes Laravel's *default* connection — every model pins `mysql` or `mongodb` explicitly. - Node for the asset build (`npm ci && npm run build`) — and the SAME `node_modules` is the Node worker's runtime, so it must survive the build rather than being pruned. Whoever runs the install must own the directory: on 21/09/2026 the testing box had a `node_modules` written by Jenkins and un-writable by the deploy user, and the worker crash-looped for hours on a missing `mysql2`. - **`node` must be executable by the SERVICE ACCOUNT, not just by you.** Under `on_demand` the queue worker (`www-data`) is what spawns it, and on 21/09/2026 `/usr/bin/node` was a symlink into `/home/arfu` that `www-data` could not traverse: `Permission denied`, from a path that worked perfectly for the deploy user. `sudo -u www-data node -v` is the check. `chmod o+x /home/arfu` fixes it; installing node system-wide is the version that does not depend on a person's home directory continuing to exist. `NODE_BINARY` exists for hosts where a different path is wanted — it does not help when the problem is traversal, since it is the same file behind the same directory. - **CJK fonts for the renderer.** A written document goes snapshot → HTML → headless Chromium, and a host with no Chinese font renders every Chinese glyph as a hollow box — silently, in a finished PDF nobody re-reads. Found on 21/09/2026 in the bilingual hire-note fixture, which exists to catch exactly this: `sudo apt-get install -y fonts-noto-cjk`, then restart the worker. - Three DNS names, one per panel: `ADMIN_DOMAIN`, `ORG_DOMAIN`, `USER_DOMAIN`. The panels are separated by **host**, not by path prefix, so one hostname is not enough. ## Environment file Copy `.env.example` and read the annotations — two blocks are easy to get wrong: - **`DB_*` is MySQL. `DB_*_ALT` is MongoDB** (`config/database.php`). The example file used to declare `DB_PORT_ALT=3306`, which pointed Mongo at the MySQL port; a fresh clone following it could not start. Both blocks are annotated now and CI's `env-diff` gates them. - `QUEUE_CONNECTION=database` and `SESSION_DRIVER=database` are **deliberate** for the MVP. Redis is full-product scope — do not "fix" this. ## ⚠️ The staging script as it stands (reviewed 2026-09-03) — three problems The build step currently running on staging has one destructive line and two omissions. ### 1. `php artisan db:seed --force` DELETES DATA ON EVERY DEPLOY — remove it `DatabaseSeeder` calls `AdminSeeder`, which does `table("admin")->truncate()`, and `OrganizationSeeder`, which does `table("organization")->truncate()` and then re-inserts **only** org 1 (Anoath) and org 2 (Day Day Help) with the password `abcd1234`. So every deploy to staging: - **deletes every organization created through the UI** — anything beyond ids 1 and 2; - **resets the passwords** of orgs 1 and 2 back to the seed value; - wipes and re-seeds `admin`; - leaves MongoDB **untouched**, so `user_org_belonging`, `chatroom`, `flow_run`, `flow_version`, `message_template` and every `*_detail` document goes on pointing at organization ids that no longer exist. This is where orphaned documents come from. Seeding is a **first-boot** action, not a deploy action. Run it by hand once when a box is built. ### 2. `config:clear` is missing before `config:cache` A box that ever ran a broken build keeps serving the cache that 404'd all three panels. Clearing first is cheap; not clearing is a panel-wide outage that presents as a routing bug. ### 3. Nothing here starts the queue or the scheduler `queue:restart` does **not** start a worker — it signals existing workers to exit after their current job so they reboot onto the new code. If nothing supervises `queue:work`, that line is a no-op. And there is no crontab entry, so `schedule:run` never fires, so `flow:timer-sweep` never runs: no chases, no give-ups, no appointment reminders, no reschedule chases, no slot-hold releases. See *Long-running processes* below — this is the failure that cost a morning on 2026-09-03. ### Bonus, found while reviewing it: uploaded files were 404ing on the server **Do not add `storage:link`** — this project serves uploads through Laravel, never through a symlink (owner decision, 03/09/2026). It already did, via `filesystems.php`'s `serve => true`… but the framework registers that URL as a **closure**, and `FilesystemServiceProvider::serveFiles()` begins `if ($this->app->routesAreCached()) return;`. The deploy runs `route:cache`. So on staging that route was never registered and **every chat attachment, evidence file and receipt 404'd** — while working perfectly in local development, where routes are not cached. Fixed 03/09/2026: `serve => false`, and `App\Http\Controllers\StorageController` owns `/storage/{path}` as an **invokable**, which is the repo's standing route rule for exactly this reason. `StorageServingTest` guards it, including an assertion that the route is not a closure. ### Corrected build step ```bash set -xe cd /var/www/html/plantoo_tech_staging git reset --hard git pull composer install --no-dev --optimize-autoloader --no-interaction --prefer-dist /home/arfu/.nvm/versions/node/v25.3.0/bin/npm ci /home/arfu/.nvm/versions/node/v25.3.0/bin/npm run build # db — migrations only. NEVER db:seed here: the seeders truncate admin and organization. php artisan migrate --force # cache — clear BEFORE cache php artisan config:clear php artisan config:cache php artisan route:cache php artisan view:cache # queue — LAST of the PHP steps. Restarts existing workers onto the new code; does not start one. php artisan queue:restart # worker — ONLY under WORKER_MODE=supervised, and then it is a supervisor program like the queue # worker. Under `on_demand` there is nothing to restart here: the queue worker runs it per job, and # `queue:restart` above already put that onto the new code. # sudo systemctl restart plantoo_testing_render ``` ## Deploy sequence Order matters, and every line here exists because leaving it out broke something once. ```bash git pull # or however the deploy tool delivers the code composer install --no-dev --optimize-autoloader --no-interaction npm ci && npm run build php artisan migrate --force # migrations are production-facing; see CLAUDE.md php artisan config:clear # BEFORE config:cache, always — see note below php artisan config:cache php artisan route:cache php artisan view:cache php artisan queue:restart # MUST be the last line — see note below ``` **`config:clear` before `config:cache`.** A box that ever ran the broken build keeps serving the cache that 404'd all three panels. Clearing is cheap; not clearing is a panel-wide outage that looks like a routing bug. **`queue:restart` last.** A long-lived `queue:work` keeps executing the code it booted with, so a deployed job-class change never takes effect — it presents as "the fix didn't work". It is the commit point of the deploy: with `set -xe`, an earlier failure should leave the old workers running on the old code rather than half-restarted onto a broken deploy. **Routes must be cacheable.** Every route is an invokable or a controller action, never a closure, because `route:cache` fails on a closure. Keep it that way. ## Long-running processes Three, and the product is visibly broken without the first two: ```bash php artisan queue:work --tries=3 # supervisor: autostart, autorestart, 1+ process php artisan schedule:work # or the standard crontab entry, below node worker/index.js # supervisor: autostart, autorestart, stopwaitsecs=30 ``` ```cron * * * * * cd /path/to/app && php artisan schedule:run >> /dev/null 2>&1 ``` **Under `WORKER_MODE=on_demand` the third one goes away** — the queue worker runs the Node worker when a step asks for a document — so a host needs exactly two things per environment: a supervised `queue:work`, and one cron line driving the scheduler. **Use a SYSTEMD UNIT, because the host already does.** The sibling application on the same box runs `anoath_testing_worker.service` and `anoath_staging_worker.service`, and its scheduler from the deploy user's crontab. A second convention on one host is a second thing to remember at three in the morning, so this follows it rather than inventing. (Its own runbook says supervisor; supervisor is not installed there. Believe the box, not the document — checked 21/09/2026.) `/etc/systemd/system/plantoo_testing_worker.service`: ```ini [Unit] Description=Plantoo Testing queue worker After=network.target mysql.service [Service] User=www-data Group=www-data WorkingDirectory=/var/www/html/plantoo_tech_testing ExecStart=/usr/bin/php /var/www/html/plantoo_tech_testing/artisan queue:work --sleep=1 --tries=3 --max-time=3600 Restart=always RestartSec=5 # NOT in the sibling's unit, so systemd's default 90s applies there. A document render can outlast # that, and a worker killed mid-render leaves a `render_job` row Running that nothing reclaims. TimeoutStopSec=600 [Install] WantedBy=multi-user.target ``` ```bash sudo systemctl daemon-reload sudo systemctl enable --now plantoo_testing_worker systemctl status plantoo_testing_worker # expect active (running) ``` One unit per environment, differing only in name and the two paths. `--sleep=1` rather than the sibling's 3: a queued render is something a customer is watching the chat for. And the scheduler, in the deploy user's crontab — the same shape the sibling uses: ```cron * * * * * cd /var/www/html/plantoo_tech_testing && php artisan schedule:run >> /dev/null 2>&1 ``` `queue:restart` in the deploy then does what it is for: the worker exits after its current job and systemd starts it again on the new code. **Linux keeps nothing alive by itself** — that is the only reason a supervisor appears in this document at all. The scheduler runs `flow:timer-sweep` every minute (`routes/console.php`). **Everything time-based depends on it** — reminder chases, give-ups, appointment reminders, reschedule chases and slot-hold releases are all `Timer` rows that something must come back and fire, and each firing schedules the next one. With no runner a reminder loop stops dead at its first link, which reads as a broken flow rather than a missing worker. It cost a morning on 2026-09-03. `failed_jobs` is the triage table; see the README runbook. ### The Node worker (FP-T26) A second runtime, and the only one. It does the two things PHP should not: fill an official PDF, and (when D2 lands) run a workbook's formulas. **One process, two job types** — the cost of supervising, deploying and watching a process is the same whether it does one job or two, which is why the second arrives as a handler rather than as a second service. - **It needs `npm ci --omit=dev` on the host.** The build already runs npm for the frontend; this adds a runtime dependency on `node_modules` remaining present after the build rather than being discarded. Three runtime packages: `pdf-lib`, `mysql2` and `@univerjs/preset-sheets-node-core`. - **A browser for LAYOUT, never for arithmetic**, and the two are separate decisions that happen to land in the same process. The spreadsheet ran in headless Chromium at one point; it worked, and it was replaced — the board's own words are *"a browser cannot be trusted with a statutory figure"*, and a browser adds a rendering pipeline, a font stack and a window's worth of state to something that is arithmetic. The sums are Univer's own engine, headless, in Node. **Rendering a written document is the opposite case**: a browser is the best text layout engine anybody has, and the figures reach it already decided. So `playwright` is a runtime dependency again, and the host needs its browser. - **Install the browser into a SHARED path, not a home directory.** Playwright resolves browsers under `$HOME/.cache/ms-playwright`, and on a server the user who installs one is never the user who launches it: you install as yourself, Jenkins installs as `jenkins`, and the render runs as `www-data`, whose home holds neither. The failure is `browserType.launch: Executable doesn't exist at /var/www/.cache/ms-playwright/...` on a checkout where the identical job worked by hand an hour earlier — 21/09/2026, and the flow took its `on_error` branch in front of a customer. So, once per host: ```bash sudo PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright npx playwright install --with-deps chromium sudo chmod -R a+rX /opt/ms-playwright ``` then `PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright` in `.env` (Laravel passes it down to the worker under both `on_demand` and `scheduler`) and the same line as `Environment=` in the unit under `supervised`. The build step installs into the same path rather than Jenkins's home: `PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright ./node_modules/.bin/playwright install chromium`. - **`playwright` is pinned to the same version as `@playwright/test`.** Two Playwright versions in one project fight over browser builds — installing one prunes the other's, and whichever ran last wins. Keep them equal when either moves. - **`stopwaitsecs=30`.** The loop finishes the job in hand before exiting, so a deploy that kills it mid-render leaves a row `Running` and a half-written file. - **Nothing reclaims a stuck `Running` row automatically**, deliberately: a document rendered twice is worse than one rendered late, and an official form filed twice is worse again. A row sitting in `Running` is how a crashed worker is found — check for one after any hard restart. - **Unlike the two above, nothing is broken while it is down.** Rendering is not on any customer's path today; jobs queue up in `render_job` and drain when it comes back. Worth knowing during a deploy, and worth an alert before anything depends on it. - Logs are one JSON line per job to stdout — `claimed`, `done` with the hash, `failed` with the reason. Point the supervisor's `stdout_logfile` somewhere and that is the whole observability story for now. `worker/README.md` carries the supervisor stanza. - **`WORKER_MODE` decides how this is driven**, and it is the setting to read before deciding what to supervise. Read through `config("plantoo.worker")`, never `env()` — see the panel-domain note above. | mode | what runs it | latency | what the host must supervise | |---|---|---|---| | `supervised` | a systemd unit keeps `worker/index.js` alive | a second | queue worker, scheduler, **worker** | | `scheduler` | the schedule runs `--once` every minute | up to a minute | queue worker, scheduler | | `on_demand` (default) | a QUEUED JOB runs `--once` when a step asks for a document, then collects | seconds | **queue worker only** | `on_demand` is the one that needs no scheduler for documents at all: `App\Jobs\RunRenderWorker` runs the worker and then calls `render:collect` itself, because producing a document and telling the conversation about it are two steps and a host with no scheduler would do only the first. It is `ShouldBeUnique`, so a burst of steps starts one process rather than ten. **The scheduler is still required for everything else** — `flow:timer-sweep` is what fires reminders and deadlines, and nothing dispatches that on demand. `node worker/index.js --once` also runs by hand, which is how a backlog is drained after any of this has been misconfigured — `routes/console.php` then runs it every minute under the same schedule as everything else, with `withoutOverlapping()` so a render that outlasts a minute cannot be joined by a second copy. Set `NODE_BINARY` too where node is not on the scheduler's PATH, which is every nvm host. That takes the box from three long-running processes to two, and costs a document up to a minute of latency plus Chromium's start-up on every run. Choose it where fewer processes matter more than seconds; supervise the worker where a customer is watching the chat for their statement. **Never both at once**: two workers racing one table is safe — the rows are claimed under a lock — but it is twice the CPU for nothing. - **CI/CD does not keep it alive — it only restarts it.** A deploy shell runs once and exits, so anything it starts dies with that session. Under `WORKER_MODE=supervised` the host needs a unit of its own beside the queue worker: ```ini [Service] User=www-data WorkingDirectory=/var/www/html/plantoo_tech_testing Environment=PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright ExecStart=/usr/bin/node worker/index.js Restart=always TimeoutStopSec=30 ``` `Environment=` is not optional here: nothing else sets it for a supervised process, and without it the unit looks for Chromium in `www-data`'s home. The other two modes get it from `.env`. `TimeoutStopSec=30` matters: the loop finishes the job in hand before exiting, and a deploy that kills it mid-render leaves a `render_job` row `Running` that nothing reclaims. Add `sudo systemctl restart plantoo_testing_render` to the deploy, since `git pull` changes its code too. Found on 21/09/2026: the deploy script had never mentioned the worker at all, and documents queued for 104 minutes while the panel correctly said the service was stopped. ## Seeding — read before running it anywhere with real data `AdminSeeder` and `OrganizationSeeder` **truncate**. Running `db:seed` on a live environment DELETES both organizations and leaves `user_org_belonging` documents pointing at ids that no longer exist. Safe on a fresh box, destructive the moment anyone has created anything. A full reset is two commands **in this order**: ```bash php artisan db:wipe --database=mongodb --force # FIRST php artisan migrate:fresh --seed --force ``` `migrate:fresh` only drops MySQL. Without the Mongo wipe first, every document survives what looks like a complete reset — and the result is a MySQL schema with no rows pointing at a Mongo store still full of the old ones. A clean seed produces exactly **one admin and two organizations**, and nothing else: no demo flow, no demo service, no demo user. That is deliberate — building those through the UI *is* the test (`docs/mvp-test-plan.html`). ## Post-deploy smoke check Five minutes, in this order. Anything failing here is a broken deploy, not a flaky test. 1. `curl -fsS https://«org host»/api/health` → `{"status":"ok"}`. 2. Load each of the three panel hosts signed out. Each shows its login form; **none returns 500** (an unauthenticated navigation returning 500 is a regression the browser suite has caught before). 3. Sign into the org panel. The dashboard renders — a blank page means the bundle died on an un-imported constant, which has happened and which no PHPUnit test can see. 4. Confirm both processes are up: `php artisan queue:work` and the scheduler. Then check `select count(*) from timer where status = 1 and fire_at <= now()` is not growing — a rising count is the signature of a dead sweep. 5. Run one flow end to end as a customer in a throwaway organization, and check the message arrives in the org Inbox. Never smoke-test in a real organization. Build a dedicated testing org. ## Demo walkthrough Six scenarios, in order, against a freshly built environment. Preserved here from the MVP release card because the content is release-time work, not a task. **It needs the content built through the UI first**: a clean seed creates two EMPTY organizations — no demo flow, no service, no user. 1. **Tenancy.** Two organizations; suspend one; its chat entry shows unavailable; reactivate. 2. **1-way.** An announcement flow; the message appears in the customer panel. 3. **Booking.** Complete the flow in org A; the appointment shows in *My Schedule* and in the org panel; the slot is gone from availability. 4. **2-way.** Go quiet on a choice node; the timer fires; the reminder arrives; acknowledge. (Needs the scheduler running — see *Long-running processes*.) 5. **3-way.** Upload evidence; approve it in the org's Verifications; the receipt appears; confirmed. 6. **Isolation and concurrency.** A cross-org URL is not-found; parallel booking gives 1×201 and 19×409. ## Not automated, on purpose `npm run e2e` (Playwright, 13 specs) is not in the deploy path. It needs a running app with real hosts, and wiring it is T12's open checkbox — see `docs/progress.md`.