Keep Remote Mac Jobs Running After SSH Disconnects

You start a long-running build on a cloud Mac over SSH, close your laptop, and reconnect later only to find that the terminal is gone. The biggest problem is not having to rerun the job—it is not knowing whether the job failed, completed, or is still running in the background. The reliable approach is not to keep extending connection timeouts, but to separate the network connection, terminal session, and actual job into three layers: SSH provides access to the node, tmux preserves the terminal context, and logs plus exit codes provide proof of the outcome.

Identify Which Layer a Disconnect Affects

A Remote Desktop or SSH disconnect only means that the interactive channel between the client and the cloud Mac has been interrupted. It does not mean the physical node has stopped running. However, a foreground process attached directly to an SSH pseudo-terminal may receive a hangup signal when the session closes. A terminal opened through the graphical interface is also unsuitable as the entry point for an unattended job, because closing the window or signing out of the desktop changes its lifecycle.

Start with a ten-minute test job to validate the environment instead of experimenting with a production build:

mkdir -p "$HOME/jobs/session-check"
cd "$HOME/jobs/session-check"
date -u +"start=%Y-%m-%dT%H:%M:%SZ" > run.log
sleep 600
date -u +"finish=%Y-%m-%dT%H:%M:%SZ" >> run.log

After starting it, disconnect intentionally, reconnect, and inspect run.log. This baseline test confirms whether the node itself continues running, but it is not a substitute for persistent session management.

SSH keepalives determine how quickly a dead connection is detected. tmux determines whether a recoverable terminal remains available after that connection fails. Neither replaces the other.

Configure SSH Keepalives and Explicit Timeouts

Create a dedicated host entry for the node in ~/.ssh/config on your local Mac. Replace the address and username with the actual values provided in the console:

Host minid-node
    HostName <node-address>
    User <system-user>
    ServerAliveInterval 30
    ServerAliveCountMax 3
    TCPKeepAlive yes
    ConnectTimeout 10

ServerAliveInterval 30 tells the client to send an application-level probe every 30 seconds. If three consecutive probes receive no response, SSH terminates the dead connection instead of leaving the terminal frozen indefinitely. This does not reconnect automatically or guarantee that a foreground process will survive.

Before connecting, inspect the effective configuration:

ssh -G minid-node | grep -E 'serveralive|tcpkeepalive|connecttimeout'
ssh minid-node

If you switch networks frequently, avoid extending the timeout to tens of minutes. Detecting a dead connection sooner makes it easier to reconnect and re-enter tmux to check the job’s actual progress.

Run Long-Running Jobs in tmux

First check whether tmux is available. If it is not, install it using the package management method supported by the current environment:

command -v tmux || brew install tmux
tmux new-session -s ios-build

Use a session name that describes the job, such as ios-build, integration-test, or asset-export. Avoid reusing an ambiguous name such as work indefinitely. Once inside the session, create a dedicated directory and send output to both the terminal and a log file:

job_dir="$HOME/jobs/ios-build-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$job_dir"
cd "$job_dir"

set -o pipefail
caffeinate -i /path/to/run-build.sh 2>&1 | tee build.log
status=${PIPESTATUS[0]}
printf '%s
' "$status" > exit-code.txt
date -u +"%Y-%m-%dT%H:%M:%SZ" > finished-at.txt

Press Control-b, followed by d, to detach from the session without terminating the job. After reconnecting to the node, run:

tmux list-sessions
tmux attach-session -t ios-build

caffeinate -i prevents idle system sleep only while the command is running. Keep it tied to the specific job rather than running it permanently for convenience. Command-line builds do not require the display to remain awake.

Avoid Starting Duplicate Jobs

After a network interruption, run tmux list-sessions and pgrep -fl run-build first. Restart the job only after confirming that the previous instance no longer exists. Otherwise, two builds may write to the same derived-data directory, cache, or artifact path at the same time, leaving behind mixed results that are difficult to reproduce.

Use Logs and Exit Codes as Acceptance Criteria

Seeing terminal output continue to scroll after reconnecting is not proof of success. Every long-running job should record at least its start time, end time, complete log, and exit code. Only an exit code of 0 indicates that the script completed successfully according to its contract. A particular word appearing at the end of the log is not an adequate substitute.

Use the following sequence to assess the job quickly:

The script should also create a new directory for every run instead of overwriting the previous log. When cleanup is necessary, delete confirmed, archived directories by date rather than recursively deleting the entire ~/jobs directory.

Run a Planned Disconnect Drill Before Production Use

Before using the setup for production work, start a safely repeatable test job and verify SSH disconnection, Remote Desktop closure, and local network switching one at a time. After each reconnection, check the tmux session, target process, log growth, and exit-code file.

If the job depends on a graphical interface, test separately whether it continues after Remote Desktop disconnects. Command-line tools and graphical applications do not have the same lifecycle. If a job ends early, inspect the script’s exit code and log first, then determine whether the terminal was closed, and finally check storage capacity and process resources. Do not attribute every failure to the network.

A production-ready checklist should include:

• The SSH configuration can be verified with ssh -G;
• Each type of long-running job uses a dedicated tmux session;
• Job directories are timestamped and do not overwrite previous results;
• Standard output and standard error are written to the same searchable log;
• The exit code and UTC completion time are saved separately;
• After a disconnect, recover the existing session before starting another job;
• When the job is complete, close unused sessions and archive the artifacts.

Frequently asked questions