Deployment
From an order to a running, licensed, backed-up, monitored cPanel server with no hands on it. One resumable job, 24 idempotent steps, every step with a probe, a failure rule and a rollback.
Deploy gate
Nothing is invoiced in advance, but AWS cost starts at launch. The gate is the account, not a payment.
Card on file
OrderController::paymentMethodRefusal() already refuses manual gateways for Rochen One; it extends to Dedicated::handles(). No stored card, no order.
Good standing
Any overdue invoice on the account refuses the order with 422 "Account has overdue invoices". The existing fraud checks run as for every order.
Immediate dispatch
Dedicated::store() writes the pool row as queued and dispatches DeployDedicatedInstanceJob afterCommit, the way Hostings::create() dispatches site deployments.
Non-payment later
Monthly capture with the stored card; failure follows the normal overdue path: suspension stops the instance, termination after the usual grace. Usage accrues while suspended.
Job contract
App\Jobs\DeployDedicatedInstanceJob, same shape as MigrateServerJobdedicated, a new Horizon supervisor with timeout 1800 and 2 processes; queue.connections.redis.retry_after raised to 1900 so Redis never re-delivers a running jobShouldBeUnique per hosting id for 2 hours, plus cache lock dedicated:deploy:{hosting} held for the job timeout + 60 s$timeout 1500 · retryUntil() = 8 hours · $maxExceptions 5 · backoff 60 / 300 / 900 / 1800 sWAIT; the job releases itself for 60–120 s. A release is not an exception, so hours of polling never exhaust the retriesdedicated_instances.plan: {version, steps: [{key, status, attempts, started_at, finished_at, output}]}; current_step and attempts mirrored for listingsResumeInterruptedDedicatedJobs listener on WorkerStarting, or php artisan dedicated:resume {hosting}. The runner skips every step whose verify() probe already passesServers\AWS\ClientFactory: management credentials from resources/data/aws.json, member-account clients through OrganizationAccountAccessRole. Tests inject SDK mock handlers hereSequence
sequenceDiagram
autonumber
participant C as Client (app)
participant A as rochen-api
participant J as DeployDedicatedInstanceJob
participant O as AWS management account
participant M as AWS member account
participant L as Licence providers
participant S as New server
C->>A: POST /v1/order (464, region, hostname, template, disk)
A->>A: card + standing checks, pool row (queued)
A-->>C: 201 (no invoice)
A->>J: dispatch after commit
J->>O: CreateAccount r{userid}-{hostingid}, SSO assignments
J->>M: contacts, IAM role, VPC IPv6, security groups, key pair
J->>M: AllocateAddress (Elastic IP)
J->>A: tblservers row for the pool (hostname, IP, root password, disabled)
J->>L: initial tiers: cPanel Solo, Imunify 1 user, LiteSpeed by instance, CL/KC keys, JetBackup, Softaculous
J->>M: PutParameter secrets (SSM)
J->>M: RunInstances (cloud-init, no credentials)
M->>S: boot
S->>M: wait until IMDS shows the Elastic IP
S->>M: ssm get-parameter (root password, keys)
S->>S: cPanel, CloudLinux, KernelCare, JetBackup, MailChannels, CSF
S->>M: bootstrap-status = done
J->>M: poll bootstrap-status, verify WHM :2087
J->>M: Zabbix host, Cloudflare DNS + PTR
J->>A: enable the pool's tblservers row (WHM token stored)
J->>A: create change applied (billing starts), pool Active
A-->>C: "Your server is ready" mail
The 24 steps
Every step is a class with verify() (is it already done?), run() and rollback(). The plan records each outcome. Order matters in two places: the Elastic IP exists before any licence is issued, and licences exist before the instance boots.
| # | Step | Does | Already done when… | On failure | Rollback (abort path) |
|---|---|---|---|---|---|
| 1 | account | Organizations CreateAccount named r{userid}-{hostingid}, email aws+{name}@rochen.com; polls the request (WAIT); stores aws_account_id and the aws_id custom field | an account with that name exists | email already used → adopt; account limit → unrecoverable | close account through the quota queue |
| 2 | sso | Identity Center assignments for the Engineering and Support groups | assignments listed | retry | delete assignments |
| 3 | contacts | alternate contacts BILLING / OPERATIONS / SECURITY | always (upsert) | retry | none |
| 4 | iam | role + instance profile AmazonSSMManagedInstanceCore with an inline policy limited to /rochen/dedicated/{hosting}/* parameters; 15 s wait for propagation | role and profile exist | retry | detach, delete profile and role |
| 5 | network | default VPC: Amazon IPv6 /56, a /64 per subnet, IPv6 route, DNS records on launch; stores vpc and subnet ids | association state and route present | retry | none (account closes) |
| 6 | security_groups | cPanel, SSH, Outbound and ICMP, rules from config/dedicated.php | groups found by name | retry | delete groups after the instance is gone |
| 7 | key_pair | ed25519 key pair hkp-{sld}-{hostingid}; PEM stored as an SSM SecureString, never on disk | key and parameter both exist | retry | delete key and parameter |
| 8 | elastic_ip | allocate before launch, tagged with the hosting id; picks the generated hostname adjective-noun from the word lists, unique across pools | address with that tag exists | address limit → unrecoverable | release (or hold 3 days if the pool once ran) |
| 8b | register_server | inserts the pool's own tblservers row now that the name and IP exist: name and hostname = {adjective-noun}.directrouter.com, ipaddress = Elastic IP, generated root password WHMCS-encrypted, same type as the Rochen One servers, "Dedicated Pools" server group kept out of fill, disabled = 1; the pool's tblhosting.server is repointed to it. This row is the only store of hostname, IP and credentials | row with that hostname exists | retry | delete the row, repoint to the region row |
| 9 | mailchannels | sub-account and SMTP password (kept only in SSM) | sub-account listed | retry, unrecoverable after 5 | delete sub-account |
| 9b | wasabi | bucket jb-{hostname} and an access key for JetBackup with a create/read/write-only policy; the deployer's own Wasabi credentials cannot delete buckets | bucket exists | retry | none: buckets are never deleted by the system |
| 10 | secrets | SSM parameters for the bootstrap only: root password (read from the tblservers row), MailChannels password, CLN key, Wasabi key, bootstrap-status = pending; they are deleted in step 17 once the server reports done. The same set is written as one item into the 1Password inbox vault through a service account scoped to that vault only (staff move it later) | always (overwrite / item exists) | retry; 1Password failure never blocks the build, it opens a ticket | delete parameters; the 1Password item is kept |
| 11 | licences | issues the initial tiers for a new, empty server: cPanel Solo (Manage2 add-to-IP), Imunify360 1 user + CloudLinux + KernelCare (CLN keys), JetBackup (JetLicense), Softaculous to the IP (NOC API to confirm), LiteSpeed by instance size (Site Owner on m7i.large, ticket until the API is confirmed). Ids into dedicated_instances.licences; the tracking options set | a licence for this IP already exists at the provider | provider error → retry | expire / delete key / cancel |
| 12 | launch | RunInstances with a ClientToken; AMI and type from dedicated_templates; gp3 root 40 GB and /dev/sdf home at the ordered size, IOPS and throughput; volumes kept on termination; IPv6; tags hosting_id, client_id, rochen:line=dedicated; CreditSpecification=standard only on burstable types; user-data rendered from the cloud-init template | a non-terminated instance carries the hosting tag | capacity or quota → retry, then unrecoverable | terminate and delete volumes |
| 13 | wait_running | WAIT until running (10 min cap) | — | timeout → failed | — |
| 14 | associate_eip | attach the Elastic IP the moment the instance runs, so cPanel licenses against the right address | association present | retry | disassociate |
| 15 | tag_volumes | tag root mount=/ and home mount=/home; store volume ids and sizes | always (upsert) | retry | — |
| 16 | wait_status_ok | WAIT until both instance status checks pass (15 min cap) | — | timeout → failed | — |
| 17 | wait_bootstrap | WAIT on bootstrap-status written by the script (90 min cap); then verify WHM answers on :2087 and whmapi1 version succeeds over SSM | parameter says done | failed:{stage} or timeout → unrecoverable (a rerun would reinstall cPanel); the ticket carries session instructions | — |
| 18 | zabbix | read-only IAM user and key for Zabbix, host.create with the templates MCS uses; stores zabbix_host_id | host found by name | retry | delete host, key, user, policy |
| 19 | dns | A and AAAA records for {adjective-noun}.directrouter.com through the Cloudflare API (token scoped to the directrouter.com zone, DNS edit only), then reverse DNS on the Elastic IP (ModifyAddressAttribute) | records resolve to the Elastic IP and PTR set | Cloudflare error → retry; PTR refused → retry after DNS propagates | delete the records |
| 20 | enable_server | creates the WHM API token over SSM and stores it WHMCS-encrypted on the pool's tblservers row (accesshash), deletes the bootstrap parameters from SSM, sets the row disabled = 0. From here the Rochen One site code and the worker address the server through that row | token present, parameters gone | token creation fails → retry | disable the row |
| 21 | activate | pool Active; dedicated_instances.status = active, provisioned_at; the create change becomes applied with applied_at = now — billing starts here, not at launch | status | — | — |
| 22 | notify | "Your pool is ready" mail (hostname, IPs, how to add sites from the app; no WHM or root details, the client has neither); a handover Maintenance ticket if anything was skipped: manual LiteSpeed step, or DNS records the zone API refused | — | — | — |
directrouter.com zone lives.cloud-init contract
Template
resources/views/dedicated/cloud-init.sh.blade.php, versioned; the version is stamped into dedicated_instances.cloud_init_version. Content is today's keys/init.sh (cPanel, CloudLinux, KernelCare, JetBackup, MailChannels, CSF, LVM for /home, MySQL on /home) made parameterised.
Placeholders only
hostname, elastic_ip, hosting_id, region. Nothing else is rendered in. The legacy module's habit of embedding keys/aws.json in user-data is the one thing this design must never repeat.
Secrets at boot
The script reads the root password, the MailChannels password and the CLN key with aws ssm get-parameter --with-decryption, allowed by the instance profile only for its own path. Those parameters exist only until the bootstrap reports done; afterwards the root password and WHM token live in one place, the pool's tblservers row, WHMCS-encrypted, never shown to the client.
Ordering guard
The first stage loops until the instance metadata reports the Elastic IP (15 min cap), so the cPanel installer licenses against the address step 11 registered.
Progress
Each stage writes bootstrap-status: failed:{stage} on any non-zero exit, done before the final reboot. Step 17 reads it.
AMI prerequisite
The SSM agent and AWS CLI v2 must be baked into the AMI listed in dedicated_templates; without them the script cannot fetch its secrets.
Failure & recovery
Transient
Any exception marks the step failed with its message and rethrows. Horizon retries with the backoff; the next run re-probes every step and continues from the first that is not done.
Final failure
After 5 exceptions or 8 hours: status = failed, plan kept, ProvisioningFailedTicket in the Maintenance department (one open ticket per service, like the disk-mismatch tickets), devops mail with the plan dump.
Automatic teardown
The failed build is torn down at once by the termination job in abort mode: licences bound to the IP expired, instance terminated, volumes deleted, IP released, DNS removed, MailChannels sub-account removed. The member account, the Wasabi bucket and the 1Password item are kept. dedicated:resume then starts a fresh build from step 1 with a new IP and new licences (Chris and Ben's rule).
Refunds
Abort raises no automatic credit; a credit note is a support decision recorded on the ticket.
Commands
| Command | Purpose |
|---|---|
dedicated:resume {hosting} [--step=key] | re-dispatch a failed or interrupted deployment, optionally from one step |
dedicated:abort {hosting} | tear down a failed deployment |
dedicated:show {hosting} | print configuration, plan progress and the change log |
dedicated:check-catalogue | validate regions, option groups, templates and prices |
dedicated:retier-licences | daily: reconcile the pool's site rows with the accounts on the server and move licence tiers up or down |
dedicated:release-ips | daily: release Elastic IPs whose 3-day hold after termination has passed |
All commands are documented in public/docs/cmd.html; the endpoints in public/docs/api.html.