Fan-out topology · Half-sync detection · Lease arbitration · Retry budget · With Merge Queue docs
Teams running shared remote Mac runners often move IPA bundles, dSYM sets, and layered caches through ad-hoc scp, then hit half-sync: the pointer names a new build while bytes are still landing, yet consumer nodes already start signing archives. This guide frames fan-out under a Mac Mesh pool, contrasts rsync with object storage on latency and cost, and lists dual-field manifests, lease arbitration, and a six-step runbook. Read it with Merge Queue vs runner labels and artifacts and cache proximity without duplicating queue policy detail.
Pools serving nightly regressions plus interactive debugging usually break on byte fan-out before CPU: one publisher must push multi-gigabyte archives across regions in a narrow window. Failures masquerade as TCP jitter, UID mapping, or stale ETags—treat fan-out edges and compute edges as separate capacity tables.
Single-repo flows stay tractable; multi-repo contention on one runner label collides cache prefixes and signing identities unless uplink leases exist. We encode read-only consumer contracts—no writes back into publisher prefixes—so consistency rides pointer moves only.
Pair this with the seat lock and mutex TTL article: locks decide who owns compile cores; this article decides how verified bytes leave the machine after compilation. If you still mix interactive sessions with unattended pipelines, validate sleep policies and screen-lock behavior so SSH control paths stay alive during fan-out jobs.
Hidden bandwidth debt: Merge Queue depth looks healthy while Actions queues stay red—often because telemetry watches GitHub only, not cross_region_bytes.
Half-sync tears: pointers flip before objects finish; consumers read truncated tarballs and see Codesign or bitcode linker errors.
UID and GID swamps: rsync preserve attrs fights provider multi-tenant accounts on shared volumes, leaving undeletable ghost directories.
Missing leases: two publishers write the same stage prefix; the later job silently overwrites so manifest hashes diverge from bytes.
LIST invoices: object storage is abused like a directory tree; API cost and latency spike yet blame lands on Apple toolchains.
Dimensions below are trimmed for shared pool fan-out, not generic cloud coursework. Few readers inside a controlled SSH allowlist usually keep rsync plus staged pointers cheapest on TCO; more than three readers across oceans often favors object storage with strong read anchoring instead of naive LIST. Immutable versioning and bucket policies tilt the decision when auditors demand durable evidence.
| Dimension | SSH rsync push | S3-compatible object storage |
|---|---|---|
| Latency shape | RTT-linear; great for few large chunks; optional compression | First-byte depends on DNS, TLS, regional edge; great for parallel readers |
| Cost model | Mostly engineer time and opportunity cost; night fan-out may hurt interactive users | Egress gigabytes, request count, and lifecycle tiers; LIST is the silent bill |
| Audit posture | SSH logs plus rsync module ACLs; cross-region copies need your ledger | Bucket policies, access logs, object locks, replication rules mature faster |
| Failure surface | TCP drops, partial files, key rotation windows | Credential blast radius, accidental public reads, wrong-version reads |
| Half-sync risk | inplace writes without temp dirs expose torn reads | Pointers flip before multipart complete yields phantom manifests |
Assume SSH trust from shared pool onboarding; split ci-merge lanes before tuning fan-out. Each step emits machine-verifiable artifacts.
Freeze publish triple: short commit SHA, build id, and toolchain fingerprint land in manifest.header; any rebuild bumps the build id.
Acquire uplink lease: register the fan-out window and byte budget with your scheduler; lease id is logged and embedded in stage paths.
Write to stage prefix only: temporary paths receive bytes; consumers must not subscribe to stage wildcards.
Dual-field validation: manifest lists tarball SHA-256 and logical size; mismatch blocks pointer promotion.
Atomic flip: pointer files or latest tags move only after gates pass; flip actions are audited separately.
Fan-out retry budget: cap exponential backoff, global timeout, and dead-letter logging on runners—never swallow failures silently.
RSYNC_RSH="ssh -o ServerAliveInterval=25 -o ServerAliveCountMax=3"
/usr/bin/rsync -az --partial --temp-dir="/var/tmp/rsync-stage-${LEASE_ID}" \
./publish/${BUILD_ID}/ consumer@${HOST}:inbox/stage/${BUILD_ID}/
Tip: Pair SSH ServerAliveInterval with chunked tarballs on jittery paths; for millisecond-grade visibility flips, prefer multipart-complete driven pointers on object storage.
Consumer nodes should fetch with read-only credentials and never hold publisher signing keys. Split code artifacts from debug symbols: ship the former through CDN-friendly small objects, move dSYM bundles through overnight bandwidth windows. If consumers still run interactive Xcode builds, avoid syncing DerivedData back into publisher prefixes or cache keys lose monotonicity.
When signing fails, compare manifest toolchain sections with consumer xcode-select paths before revisiting byte integrity; many incidents blamed on Apple upgrades are half-sync reads. Cross-read SSH versus VNC handoff: interactive bandwidth budgets are usually an order of magnitude looser than fan-out budgets—do not reuse thresholds.
Parallel XCTest interleaves disk writes with fan-out reads; chart disk write latency beside cross_region_bytes. Treat eventually consistent object clients as a distinct failure class from runner noise. Map idempotency keys into scheduler envelopes (see observable task chains) so duplicate webhooks do not rewrite stage trees in parallel.
Warning: Never purge stage trees until leases release and consumers drop handles; blunt rm -rf belongs behind two-person review.
Consumer gate: Pull only when lease state is released and pointer generations increase monotonically.
Failure classes: Retry network errors; validation errors must halt and open an incident.
Rollback: Keep two generations of manifests online for instant pointer rewind.
Bands below support capacity reviews, not SLAs; replace with your histograms. Always chart queue depth, disk write latency, and fan-out throughput together.
Night fan-out plus agent batches can spike disk latency while GPUs look idle—review both on one capacity board.
| Reader topology | Ocean crossing share | First pragmatic choice |
|---|---|---|
| 1→2 same metro | < 10% | rsync + stage + dual-field manifest; fixed SSH keepalive |
| 1→4 multi-shore | 40–70% | Immutable object versions + anchored reads; LIST replaced by index files |
| Many publishers and consumers | Any | Mandatory lease arbitration + dedicated observability tiles; forbid shared stage roots |
Fan-out over personal laptops and temporary accounts fails audit, sleep policy, and signing isolation simultaneously; even perfect algorithms cannot compensate for unreliable nodes.
Owned datacenter hardware locks you into depreciation cycles; borrowed notebooks cannot satisfy multi-region concurrency with isolated keys. Teams that must ship iOS and macOS continuously while reserving cycles for AI agents usually find VpsMesh cloud Mac Mini rental the stronger operating model: regions are selectable, nodes are dedicated and auditable, and fan-out metrics become negotiable like runner metrics.
Start from reader count and ocean-crossing share; tight SSH allowlists favor rsync; many readers needing anchored reads favor object storage. Compare regions on the pricing page.
Validate dual manifest fields against bytes, then audit pointer flip timestamps; if still broken revisit idempotency keys in observable task chains.
Use the help center for remote access guidance and the order page before provisioning nodes.