ck's Weblog: My World, Meditated


Now Playing:

Updated Database Backups

Monday, August 4 2014

I've updated my previously-shared database backup script to take advantage of modern technology:


DATE=`date +%Y%m%d`
DAYOFWEEK=`date +%u`
DATABASES=`mysql -ss -e 'SHOW DATABASES' | grep -v information_schema`
TMPDIR=`mktemp -d`

for DB in ${DATABASES}; do

mysqldump --events ${DB} | bzip2 -9 - > ${FILE}

# Copy this backup to Amazon S3 for durable external storage.
s3cmd put ${FILE} s3://${BUCKET}/nightly/

# If it's Sunday, do a weekly backup as well.
if [ $DAYOFWEEK = 7 ]; then

s3cmd put ${FILE} s3://${BUCKET}/weekly/


rm ${FILE}


rmdir ${TMPDIR}

All of the rotation and deletion functionality has been removed. I've replaced it with S3 lifecycle rules:

  • Files in the 'nightly' directory are automatically deleted a few days after upload.
  • Files in the 'weekly' directory are kept online for a few weeks, then archived to Glacier for a few months before deletion.

There's a small but important difference in those rules compared to the previous system, which kept the most recent three nightly and five weekly backups. If my system goes offline for a period of time, or if my backup script fails, backups will continue to be expired with nothing coming in to replace them. As such, I expect to continue to tweak the lifecycle rules as I watch data going through the system to make sure I'm comfortable with the availability.


AWS Marketplace Copy Protection

Saturday, June 21 2014

Amazon Web Services announced a new SSD-based storage option this week. Based on my utilization, the new option would provide better performance at a slightly lower cost.

I found some documentation on migrating to a different type of storage, and the process pretty much goes like this:

  1. Shut down your instance.
  2. Take a snapshot of the volume you want to migrate.
  3. Create a new volume from that snapshot on the appropriate type of storage ("gp2" in this case).
  4. Detach your original volume from your instance.
  5. Attach the new volume.
  6. Start your instance.

There are some cleanup steps to take afterwards so you're not paying to store extra volumes and snapshots you don't need, but it should be pretty simple, requiring only a few minutes of downtime.

However, when I got to step 5, I found that I wasn't permitted to attach my new volume. For that matter, I wasn't permitted to reattach the original volume.

After some research (and a little panic), I found the explanation. My original volume had a marketplace code:

AWS volume manager screenshot showing marketplace code

That code reflected that I'd purchased a CentOS image from the CentOS maintainers. As a purchased product, it makes sense that if I want another one, I should be required to purchase a second copy rather than duplicating it myself.

In this case, CentOS images are sold for $0.00 with a recurring cost of $0.00 per hour, so the copy protection isn't really necessary.

I was able to call Amazon Web Services and open a support case. Once I shared my snapshot with the AWS marketplace team and provided some additional information, they were able to verify that the copy protection wasn't strictly necessary and provided me a snapshot with the marketplace code cleared. I was immediately able to use that snapshot to create a new volume free of the previous restrictions:

AWS volume manager screenshot showing absent marketplace code

Once that was done, it was a simple matter to attach the new volume and restart my instance.

The whole process took about two days, during which my instance was powered off. That probably would have been shortened had I chosen to pay for a higher tier of support.

Even with the surprise limitation and unexpected downtime, AWS support's handling of the issue has built my confidence in using their platform.


Nuclear Trendsetting

Saturday, May 24 2014

I had the pleasure of attending the CIC Operations & Infrastructure meeting last week. I met some great people and learned a whole lot, but there was one thing that stood out:

No other CIC school has a nuclear device sharing a wall with a data center.

We're trendsetters!


15-Month Overdue Bugfix

Saturday, May 24 2014

When I moved to the cloud last year, I also moved from Gentoo Linux to CentOS. I failed to notice that:

  1. CentOS had a version regression in PHP compared to Gentoo.
  2. The CentOS version of PHP was lacks support for a particular htmlentities() argument I'd been using on Gentoo.
  3. As a result, comment posting was failing without notifying the commenter that anything was wrong.

It's now functional.



Saturday, May 3 2014

How I spent my Saturday:


On the bright side:

  1. Plumbing now works as expected.
  2. I won't have as much mowing this summer.
  3. I'm most of the way to a right-sized deck.


Copyright © 2001-2014 Chris Kuehn