Tag Archives: scripts
title-icon-code

Introducing Counterpart: a live bootable clone tool for OS X

These days, backing up Mac OS X Server is easier than ever, with plenty of options available for taking incremental backups of system and service data, and great support from Time Machine’s ServerBackup process in making sure that databases and Open Directory are included in hourly backups.

All of these great options have one thing in common though: restoration. In a production environment, as lovely as backups are; admins are often hounded by management and users when hardware fails, and time elapses whilst operating systems are re-installed and service data rolled back from backup drives. To combat this downtime, I like to keep a fully bootable backup connected to each production server that is as up to date as possible. In many situations, this allows you to get a slightly data-delayed version of your environment up and running until you can schedule proper downtime to restore backed up data. For years, I utilised Mike Bombich‘s fantastic tool Carbon Copy Cloner, along with it’s excellent Scheduled Tasks functionality for this exact purpose. With it’s recent move to shareware, I decided that there really should be a free, open source script for cloning Mac OS X that can be scheduled with launchd, and provides proper logging and statistical data. It is with this that I announce the release of Counterpart; a wrapper script for rsync on OS X that is capable of producing bootable clones of live Mac systems. I have been using this for a little over 6 months to backup hundreds of OS X systems, and it has made my life so much easier a couple of times during hardware failures.

Counterpart utilises rsync, the fantastic data and synchronisation utility, and wraps it in a script that provides the correct settings and filesystem exceptions to create a bootable clone of a live Mac system, whilst error checking and providing comprehensive logging, statistics and monitoring data. It is bundled with detailed documentation and instructions on scheduling clones. Coming soon is a companion Nagios plugin script to monitor clones and provide performance data, meaning you can be sure clones are completing successfully, and get insights into your backup data like this:

Counterpart Clone Statistics

It should also be said that whilst Counterpart was first envisaged to backup Mac OS X Server instances, there is nothing stopping it being used to back up standard OS X clients, and it would be a great, fully scriptable, free way to create a bootable clone of your OS before an upgrade or significant modification to your system.

I have uploaded Counterpart as a new project on GitHub, and welcome any feedback that you may have on using it. I am also happy to answer any questions on getting it set up in your environment, and I am best contacted using this form.

Counterpart on GitHub

title-icon-code

Monitoring launchd tasks with Nagios

Over the last 4 major releases of Mac OS X, launchd has cemented itself as the virtual engine room of service control, becoming responsible for the execution of most critical processes in the operating system.

From bootstrap all the way through to iCloud and Time Machine backup, launchd is somehow involved in almost every important task. It therefore makes sense that the comprehensive overview provided by launchd of it’s current tasks and exit statuses become central to the monitoring of a critical Mac server or client system. Particularly on Mac OS X server, where processes such as Postgres and Apache now take such a central role in the provision of services, monitoring of launchd can be a game changer in identifying issues quickly.

I have added a new script to my OSX-Monitoring-Tools project which checks launchctl, and reports on non-zero exit codes for tasks. This is a lifesaver on Mac OS X Server, and can alert you very quickly to processes controlled by launchd that exit badly (very good at picking up ‘Throttling respawn’ tasks).

CRITICAL - critical daemons (org.postgresql.postgres,com.apple.devicemanager) exited with a non-zero code! | active=113;inactive=161;error=4;

In the example above, bad permissions are stopping PostgreSQL from starting, which is also causing the failure of Profile Manager. As you can see, the script returns performance data for task counts.

Because some Apple provided launch daemons happily exit with non zero exit codes as status identifiers (com.apple.xprotectupdater, com.apple.suhelperd), an array of exceptions are built into the script. You can also supply your own exceptions and critical identifiers as arguments.

check_osx_launchd.sh on GitHub

title-icon-code

Checking Time Machine and CrashPlan backups with Nagios

Latest Backup: Failed....

When monitoring client servers, I find it very valuable to keep track of of the currency of backups, particularly where there may be long periods between us getting hands on with equipment.

I can still feel the pain of clients who inadvertently disconnect their local backup solution, only to have a major crash weeks later.

In some of the small business deployments that we actively monitor, local Time Machine and CrashPlan backups are popular, and where appropriate, we like to get a notification if one of these has failed to complete a backup within the last day.

I have added the scripts we use to GitHub today, and hope you find them useful. They actively parse and analyse the completed backup timestamp logged by each service, so they are a little more reliable than checking file age on a specific directory or file within the backup.

Visit OSX-Monitoring-Tools on GitHub