August 25, 2008

In Which I Run A Marathon For The Very First Time


Those of you who haven't seen my other, far more boring blog are possibly unaware that I have been training to run a marathon. On Saturday, I actually ran said marathon. I bring it up here in a brazen attempt to solicit more recognition for this rather impressive feat.

Please feel free to read my write-up here, then review the rest of that blog if you're at all interested in my training saga.

Then, you should feel free to offer me any sort of congratulations or kudos that you deem appropriate. I feel no shame in pointing out how cool I am, and you shouldn't either.

August 18, 2008

One Backup Solution I Won't Be Considering

My recent post on the subject of backups brought out a couple of helpful comments that I'd like to address separately from the main post.

One such comment was from Roberts G Keeling, who writes:

Aaron, did you look at Togglenetwork's TrueVault? www.togglenetwork.com,
Totally automatic,encrypted (448 bit) onsite and fully secure (level 4)offsite, totally pain free trial.
Incremental backup and restore and easily programmable both as to what is backed up and what is restored. Inexpensive due to efficient design and compression.


(I assume Mr. Keeling is affiliated with Toggle Networks in some way. That's not a bad thing; the info's still helpful either way. A little disclosure would be nice, though.)

I had not heard of TrueVault before. URL is www.togglenetworks.com (with an 's'), by the way.

I went to the web site, and spent a couple of minutes looking around, but then I tripped and fell into a giant pool of buzzwords and drowned. I have a general feeling that if I can't figure out what a product is and how it's different after a few minutes, then the web site/brochure/other marketing material isn't doing it's job. I also go by the old adage that "If you have to ask what it costs, you can't afford it." There's not a hint of pricing on the website, so I assume it's out of reach for this small business.

But the most annoying thing about the Toggle Networks website is that if you click in any text portion (besides a link), you get a popup that says "The contents of this page are copyright protected".

Really, Toggle Networks? You're really choosing to do this? You really think someone's going to steal your whole website if not for the little bit of JavaScript protecting you? And if I want to send an email to someone at my company to say, "Hey, we should consider paying unspecified amounts of money for Toggle Networks' TrueVault" and want to copy and paste a paragraph from your website to show why we should, you really want to keep me from doing that?

Sorry, Toggle Networks' True Vault, that's strike three for you. You're out.

August 17, 2008

Gold Medal Embarrassment

Becki and I were watching a particular Olympic event the other day when a particular athlete with a rather unusual name came up for their turn in the event. Becki and I both commented how stupid we thought the name was and placed the blame solely on the parents for making up such a silly name. At one point, one of us probably said "What kind of mother would give their kid a stupid name like that" or something equally insulting towards the mother.

Today, we found out said athlete's mother is in our ward.

Whoops.

August 16, 2008

Backup Manifesto & CrashPlan Review

Whenever I'm dealing with a fairly complex problem over a long period of time, I find it helpful to write a manifesto to lay out exactly what I want to to accomplish and how to go about doing that. It seems to organize my thought process in such a way that I more clearly understand what the desired result really is, and lets me achieve that desired result sooner. Here for your perusal is the manifesto that I worked on for several days on the subject of revising my backup strategy at work (and at home). If either of my readers can add anything to my thinking here, I'd definitely appreciate the feedback.

At work a couple of years ago, we started scanning all paper that would otherwise be filed, and also started taking all x-rays digitally. This has huge advantages, and I'd never consider going back, but the big current drawback is we've now outgrown the capacity of the DAT72 tapes we were using for backup.

At the same time, I've been trying to reevaluate my backup needs at home. I decided I wanted to get all my data onto my laptop, including my music collection, so I never had to use our flaky sucky desktop computer. So, I upgraded to a 320GB drive in the laptop, which is great, but it's now the biggest drive in the house, so I can't conveniently back it up to an external drive like I used to without spending more money on a new, bigger external drive.

I've also just got a new flash memory based Hi-def camcorder (the Canon HF100, which I'm very enamored with so far). Flash is convenient, but I can't just keep a box of tapes around the house anymore, and I'm sure as heck not going to buy a new card every time it fills up (at today's prices, anyway). So, I needed to calculate some way to store my "magical moving graven images" as part of my backup strategy.

My needs at home are fairly simple. I just want all my data backed up, and I don't want to have to fiddle with anything to make it happen. I want a backup off-site, as well. I used to use Time Machine (OS X built-in backup magic), but I'm stuck with a couple of it's limitations, now. Time Machine's usefulness only comes when the external drive you're backing up to is larger than your computer's drive. Time Machine can back up a smaller data set than your whole drive, but unlike other backup programs, you don't tell Time Machine what you want it to back up. Instead, it defaults to the whole drive, and you have to go in and tell it what you don't want backed up. I find the interface for setting exclusions to be a little cumbersome and time-consuming, especially if the only thing you actually want to back up is buried several directories deep on your hard drive. Additionally, Time Machine doesn't provide me any off-site backup unless I use a different external hard drive and take it somewhere, which isn't a huge hassle, but gets out of date unless I bring it back and swap frequently.

My needs at work are simple also. I need all the patient data to never go away or go bad ever for any length of time. The actual implementation of this ideal is a little more complicated though.

The server's got a mirrored RAID, so that addresses the problem of immediate hard drive failure, but then I need something external to the server to handle the case of complete server meltdown or database corruption, neither of which RAID can help with. So, I back up to tape every night. But, there may be cases where some problem or change happens that we don't catch right away, and need to go back a few days. So, I set up a week's worth of backup tapes to be able to go back to any point during that week. I may need to go back farther than that, so I've got a set of tapes alternating weekly backups so that I can go back a couple of weeks if necessary (although not necessarily to the precise date that I want). And I need offsite for disaster recovery, so I alternate a set of monthly tapes and take one with me every month.

There are a number of drawbacks with the current setup. One, if someone doesn't put the right tape in on the right day, the backup fails. I assumed this wouldn't be a problem, but I've left it to other people to do instead of doing it myself, and I'm only getting like 90% success that way. That's not a huge deal on a daily backup, but if I miss a monthly and somehow don't catch it, I might end up with an off-site backup that's three months old. With my luck, that would be the month we have the fire, and then we're really screwed. Two, this only addresses our patient record database, and the x-rays. This doesn't back up any other stuff like our accounting records, patient correspondence, internal documents or other stuff that we'd really miss if it were lost.

The bigger drawback, though, is that even though we're just backing up patient records and x-rays using this method, our data set is just too big for those tapes which formerly seemed so capacious. I can fix that a couple of ways. I can get an autoloader, and keep using DAT72. That would fix the compliance issue as well, but cost a lot. I could get a new drive and tapes in some higher capacity. That would still have the compliance issue, and also cost money. Or, if I didn't want to spend any more money, then I could reconfigure the daily backups to be differentials instead of doing a full backup each day. Then, I'd only do a full backup once per week, scheduling that backup to run overnight as usual, but then finishing the next day after someone changes the tape. This would mean even more tape changes for someone to forget to do, and would actually get part of the backup running during business hours, which I don't want to do. Our hardware isn't exactly top of the line anymore, and I want the server to not be spinning its disks feeding a backup at the same time three other people are trying to feed it an x-ray.

So, what I really want to do at work is take the humans out of the equation, and get something automated with a more up to date offsite backup. So, what I really want is to separate the tasks into some sort of local backup coupled with an online backup service. Local backup only should be easy to do, because I have plenty of unused disk space on the network here. I can designate one or more machines as the home for backups, and then just configure my backup software to copy the data to them with the right combination of full backups and differentials to get as far back as I want to go. That part of the problem's solved with $0 expenditure for at least a few more years.

For online, the first place I looked was the vendor of the practice management software. They have an online backup client and storage available, but the client is hideous and ridiculously complicated. Worse, the storage component is ludicrously expensive. We'd back up maybe 40GB, but because it's priced in tiers, we'd have to sign up for the 50GB tier at $150/month. That's after a $100 startup fee, too. That's asinine. This is Dentrix eBackup, by the way. I need to name it here in case anyone is insane enough to consider it and googles the name.

I then checked out Mozy, because I heard a few good things about them. They had a lot going for them, because they were cheaper (even in their business offering), and had a cross platform client. But, when I looked closer, I didn't like what I saw.

They had a free trial, but it's capacity limited, so I had to pony up for 1 month at $26.95 just to test out how it would deal with my full backup set. The pricing is also tiered, although much more reasonable than Dentrix. I still don't like this, though, because it's capped at the level you buy. If your backup set starts to exceed that level, I don't know what will happen, but it seems like your backup will fail. Sure, you'll probably get notified that it's time to upgrade your plan, but I don't want to deal with that. Also, the Mac client was second-class to the Windows client, which was already nothing to write home about.

When I started Googling for reviews, I discovered that they evidently have some really huge issues restoring files. You can restore through the client, but there's some sort of packaging that needs to be done, and that takes a while and apparently doesn't work all the time. You have the option of restoring from a web page, which is really nice, except again you have to wait for the files to be packaged, which could take a day or more for a long backup set, delaying what will already be a painful download. Again, it doesn't work all the time, either.

A third option for restore is for them to burn your data to DVD and Fedex it to you. If you've got 50GB of data, this seems like an excellent solution whatever the price, since it would take several days to download 50GB on a T1. However, the net is full of stories of people who ordered restore DVDs, and then didn't get them for weeks. That's completely unacceptable. If you're going to offer a service like that, you have to automate the process, allocate x minutes for burning each DVD, then when the order is placed, have your order system do the math to determine whether it makes that day's cutoff or gets bumped to the next. Display that delivery date to the customer, and then stick to it!. If you can't do this, do not offer the service. If you do offer a service that you know you can't deliver in the way a customer would expect, you might as well have your order confirmation page be a big ASCII drawing of a middle finger, because that's the kind of contempt you're showing for your customers.

Aside from the DVD thing, a backup system without a bulletproof restore process is no backup system at all. So, after reading all of those horrible things about Mozy, I didn't even wait for the first backup to finish and went to cancel my account. There is no link online to cancel a pro account. I had to email, and the guy told me where to go to find the link to cancel, but it wasn't there, and I sent him a screenshot to prove it. He said he'd cancel it manually, but I'll believe it when I don't see a charge next month.

Sorry, Mozy. I never really got the chance to get to know you myself, but evidently you really suck.

Many of the online reviewers of Mozy mentioned that once they kicked Mozy to the curb, they switched to CrashPlan and loved it. So, that was my next stop. I'm not in love yet, but I do kind of have a crush on CrashPlan.

CrashPlan is a little more of a philosophy than a backup program. CrashPlan's philosophy is that what everyone needs is a nice simple program that can run in the background and backup your data to your one or more friends' computers over the Internet. I've seen some programs with this idea before, but CrashPlan's is by far the most polished and simple. The thinking is that you'll know someone with extra space on their computer who'll host a backup for you, and you might have some extra space to return the favor for them. The common scenario they suggest if you don't have the free space is to each buy an external disk and station it at each other's houses.

Online backup is all it does, over LAN or Internet. No backup to a local disk. No backup to external drives. Just online backups. The software has a one-time charge of $20$25 or $60 for the more customizable Pro version, then no other charges as long as you're supplying your own backup location.

If you don't have any friends, fear not, because CrashPlan will offer to be your friend for a fee. You don't have to host their backups for them, either. Their CrashPlan Central service provides integrated hosting for your backups at a flat fee of 10 cents per GB per month, with a minimum charge of $5. That's about half the price of Mozy, and 1/30th the price of Dentrix.

The other features that I like are:
  • 30 day free trial, of the software and unlimited CrashPlan Central
  • All of the data is compressed, encrypted, and deduplicated before being sent to minimize bandwidth. Files that have changed will only send the portion of the file that has changed. They even claim that if multiple computers are backing up to the same location, it will not send any data that's duplicated between them. (note: I had read this in someone else's review. Turns out it's not true. Sorry)
  • Versioning, with the option to keep x versions, or unlimited.
  • A pretty simple restore interface. You just pick the date and time you want to restore to off a calendar and go from there.
  • Backs up in order of modified date. An intelligent prioritization algorithm that puts smaller, recently changed files ahead of larger, stale files. Even if the first backup's not done, it'll make sure that recently changed files get backed up first, and re-backed up if they change again before moving all of your super old files.
  • Will watch files in real-time, and back them up as soon as they change (or after a user-configurable number of minutes)
  • If you're using their storage, it's just billed by total usage at the end of the month. If you back up 100GB and pare down to 40GB by the end of the month, you only pay for 40GB
  • If you have multiple computers, they can use the same CrashPlan Central account, which pools the usage. So, you could back up two computers, each with 30GB (for example), and only pay $6 instead of 2 X $5.
  • There are a lot of customizable options as well, such as when it runs, how long after a change a file will get backed up, how much CPU and bandwidth it uses, QOS, etc.
  • It's cross-platform, and works the same on all platforms. That should be expected since it's written in Java, though. (Clarification: It's possibly only the front-end that's written in Java. The back-end engine appears to be a platform specific daemon or service.)
  • A local restore option, in which you get the backup archive onto the machine you want to restore to, moving physical drives if necessary, then run the restore locally. This in theory would go much faster than even doing the restore over a LAN.


I've been using it for a few weeks now, and I believe it will suit my needs well. I've already got it backing up my desktop at work and the server to the CrashPlan Central. I could also install it on the backup machine at work to do my local backups through it, but I'm pretty sure I won't. There's just no need to run through the extra CPU overhead for compression and encryption for local backups, plus I don't want my backups on the LAN to be encrypted. If I need that data, I want at it fast, without anything standing in my way. So, I'll probably still use NTBackup and scheduled jobs to do the local backups even though it's not as easy to use or polished as CrashPlan. However, CrashPlan seems like the way to go to install on the two computers that I need backed up offline.

I'm not 100% convinced, though. I'm still having some concerns and some unanswered questions. The web site is very sparse and the documentation equally so. This is a testament to the ease of use of the product, but there are some options whose interaction could be fairly complex, and it would helpful to have clear documentation about how they work. Either that or the wording in the client could be clarified.

Another problem with the website is that there's lots of references to a business product (as opposed to their Pro product?), but no real information about what it is or why I should use it. Most of the links about it go to pages that describe the Pro product. The few oblique references I could find made it seem like a VMWare server image used as a client/server thing for backing up multiple desktops in your organization. I only need to back up one desktop and a server; all other computers here are glorified terminals. So, maybe I don't need it. But someone probably does, and you can't sell your product if they don't know what it is/does.

The program is dog slow at backing up over the LAN. I've been tryng to have my latop seed the backup to the server here at work so I don't spend three weeks doing the initial backup from home. I'm not allocating it all of the CPU (unknown whether or not it supports dual processor), so maybe that's the bottleneck, but it's barely faster on the LAN than over the cable modem at home. It's certainly not 30-50 times faster than the Internet as they claim on their web site. 1.30-1.50 times faster, maybe...

Another thing is that it doesn't do VSS or have any way of backing up open files. I have scripts shutting off all the computers at 6 PM, so there shouldn't normally be any open files in the practice database by the time the backup starts, but I do work late frequently, and I'm sure that will be an issue at some point. If the backup goes often enough it's not much of an issue, but still. They have a beta client with VSS support, but it's XP only, so apparently doesn't work with Windows Server.

The versioning only lets you specify to retain unlimited versions, or to specify an actual number. That's useless to me. Our database is a set of files that's all got to be in sync or massive corruption will occur. So, having the 7 most recent versions of a file that changes every 2 months extends pretty far back, but then I would also have the latest 7 versions of a file that changes hourly. So, no matter how many versions I set that setting to, I can only go back as far as the most frequently updated file. So, I have to set that to unlimited or none at all. I'd really rather have a time based option like "retain the last x days worth of versions" or something. Think Time Machine.

This is an issue because I can't find out if the CrashPlan Central usage that you pay for is the actual total of all versions of all files on the server, the actual de-duped disk usage of all versions on the server, or just the usage of the current set without regard to versions or deleted files. If it's all including versions and deleted files, and you cycle through these files a lot, you could quickly find yourself with a set that takes 20GB on your box but takes up 200GB at CrashPlan Central.

And then there are the options that might look self-explanatory, but really aren't, especially when you try to figure out how they interact:

There's an option whether or not to keep deleted files at CrashPlan Central and for how long, but if you choose to never remove them, how long are deleted files kept? Is it really forever? Or is it in some way tied to the versions number? If really forever, how does that figure in to disk usage? And what happens in the case of the file that keeps getting deleted and then recreated. I might see that as multiple versions. CrashPlan may see that as a bunch of separate deleted files. If I don't want to see versions, but do want to retain deleted files, what happens there?

There's a setting for "Back up changed files after:" set to some number of minutes. On the surface, this seems useful to control the number of versions of a file I end up with. If I've got a file that changes every minute. I might only want 1 version per hour at most, not 60. But how does this option really work? If this is set to 60, does this mean the file won't get backed up until a 60 minute has elapsed with no changes to the file? Or does this mean that the file won't get backed up until 60 minutes have elapsed since the last time it was backed up? If it's set to 60, and the last time it was backed up was 2 days ago, and the file changes, how long after the change will it get backed up? Less than or greater than 60 minutes?

The program has by default a realtime scan for changes in files, as well as a set interval to do a full filesystem scan, looking for any changed files that the real-time scan missed. If real-time scanning is off, and "Verify Backup Selection Every:" is set to only every 7 days, will nothing get backed up in between? In other words, when real-time scanning is off, is the verify scan the only way that CrashPlan knows to back up a file?

So, if real-time is off, and a file is changed at 5:55 PM on Tuesday, and "Back up changed files after:" is set to 60, and the verify scan is set to run at 6:00PM every 7 days and the scan actually starts at 6:00PM that Tuesday, what happens? Does the file get caught by that scan and backed up? Does it get caught, but not get backed up because it hasn't been 60 minutes since the change? Does it get flagged for backup once 60 minutes has elapsed? Or, does it not get backed up until the next scan catches it the following Tuesday?

I'm good at QA and setting up test scenarios, so given a few days I could answer these questions by myself. However, I shouldn't have to. They should have clear wording for the options themselves, and enough info in the documentation to be able to answer them.

When the documentation (PDF) says something unclear, it has what looks to be a link to further information. For example, when talking about versions, it has a blue sentence afterwards that says "What happens if I keep all the versions of a file?". Hey, that's what I'd like to know. It looks like a link to a FAQ or something, but it's not; it's just blue text. I thought maybe these were links and they broke them in the PDF conversion, but the more I look at it, the more I think these look like notes from QA or a technical writer of questions that they needed answered by development. Maybe some developer wrote the first draft, and then some tech writer cleaned it up and added some notes to answer later, then got hit by a bus. Not knowing the tech writer wasn't actually done with it before his untimely demise, they just threw it straight on the web. Also, the screenshots don't match the currently shipping product. Not good form, guys.

As for my home needs, I've got enough space left over on the server at work to do a full online backup of my latop through CrashPlan free of charge (after initial software license fee) to that server. So, now I know that whatever happens, I'm fully backed up off site in a very current fashion. That's never been the case before, and I'm totally excited by that. I'd like a more readily accessible local backup, though, so I'll probably still use Time Machine and it's crappy exclusions interface just to get my more crucial stuff in locally accessible form. In the long term, I'll either get a nice big external drive, or a Time Capsule, or something like that and do it better.

As for the camcorder, I was holding off on its purchase for a while because I assumed flash would end up way more expensive by the time I backed up the files in a reliable (i.e. redundant and off-site) way. Once I ran the numbers, though, I saw that although initial investment in the backup media may be more, it's actually cheaper on a per-hour basis.

DV tapes, bought in bulk from the Price Club^H^H^H^H^H^H^H^H^H^HCostco might run about $3 a piece, with each tape holding an hour. This camcorder at highest settings uses a little under 8GB for an hour of footage. If I look around for the right deal, I can get a 750GB external drive for $100, or about $1/hour. Drives are inherently less reliable than tapes, so I need to get another one to compensate for that, but then, I'm still only at $2/hour. And, if I take one drive to work, I've got the safety of off-site storage, which I never had with tape. Yeah, a tape will last forever with at most a couple of drop outs or sparklies, but if my house burns down, it's gone.

In the short term (like for the first few GB of footage), I'll just store the files from the camcorder on my hard drive and let them get backed up with everything else, either locally or off-site. Then, when that pile gets too big, I'll move them to two of the external drives I already have, and carry one off-site. When the file pile gets too big for those (the smallest external I currently have is 40GB), only then will I have to shell out for new external drives dedicated for storing video. At that time, pricing might be close to 50 cents per hour, making it way cheaper, indeed.

In the longer term, when I've got a 4TB drive in my laptop, and a 100mbit/s upload bandwidth on my internet connection, I'll reintegrate all my movie footage to my laptop the same way I keep all my photos and music there, and just let it get backed up with everything else. In the meantime, I'm fine with using offline storage for the video because I really don't need access to every minute of video footage I've ever shot all the time.

August 13, 2008

Olympic Fever... Catch It!


Becki and I have been watching a shedload of Olympics coverage the past few days. This is very uncharacteristic for me, as I don't watch a lot of sports in general, and certainly not those featuring people, teams, or sports with which I am unfamiliar. But, the last few days, I've been glued to the TV for hours every night. Actually, I've been glued to the TV for hours every night for many years now. But, I'm glued to the TV watching Olympics every night now.

I watched the 1980 Winter Olympics, saw Eric Heiden win all his medals, and then desperately wanted to be an Olympic speed skater when I grew up. In 1984, I watched a lot of the LA Olympics, but I think that was mainly because I was a kid, there was nothing else on TV, and it coincided with a huge promotion at McDonalds that gave free food if the US won a medal in the event listed on your game piece (or something like that).

I didn't really watch any of the 1988, or 1992 Olympics, nor the 1994 Winter, nor the 1996 Summer. I saw none of the 1998 or 2000 Olympics. And even though the 2002 Olympics were in my backyard, and I even went to one of the medal ceremonies, I saw nothing on TV beyond the opening ceremonies. Everything since my first kid was born is a blur in my mind, so I can't remember whether I saw any of the 2004 or 2006 Olympics, but I'm guessing not.

So, with a long history of Olympic apathy, why am I so interested in it this time out? That's a very good question. I'm glad I asked.

I've thought a lot about it, and narrowed it down to two factors. One, is just the quality and quantity of the TV coverage. On DirecTV, there's something like 9 or 10 channels that cover the Olympics, 8 of which are in HD. I've watched a lot of crap that I otherwise would never consider watching just because it's in HD, and maybe some of the Olympics fall into that category. Either way, I'm just fascinated by the ability to get both tuners on the DVR recording two different channels, then scan through them while two more chunks record.

Although I didn't watch any of the 1992 Olympics, I was a big fan of the concept of the NBC Olympic TripleCast, a pay-per-view package of three channels of 24 hour Olympic coverage. If you like watching the Olympics, that really seemed like the way to go. Nowadays, with all these channels, and the VOD replays available, and the internet coverage, the TripleCast almost seems quaint.

So, massive availability always draws me in, just like I'll always eat more than I should at a buffet, even if the food is crap. But, I think the real reason I'm watching so much is that I got suckered in by the opening ceremonies.

We have a Chinese student living with us right now, so we were obligated to watch the opening ceremonies with him anyway. But, after hearing just how much money and effort was spent on putting them together, I was intrigued to see what they would actually pull off. It was really stunning, for lack of a better word, and I was hooked from the first fireworks blast.

Dear Chinese Olympics opening ceremonies,
I was blown away by your massive floor TV, your flying people, your lit up people, your bouncing up and down printing block people, and your running sideways on the Death Star people. But, you had me from insanely massive pyrotechnics.

The opening ceremony was really representative of a lot of things, but mainly representative of what you can accomplish in the singular focus of a totalitarian society. The massive transformation of Beijing in the last few years is another testament to that. I guess that means the one thing we can all look forward to after Russia's current return to totalitarianism is a really kicking Winter Olympics in Sochi in 2014.

Much has also been written about how this was China's big coming out to the West, but I think it's a little more than that. It's more like China's big spectacle to kick off their upcoming domination of everything: all Olympic sports, economics, world military power, whatever.

So, I've watched a lot. I watched rowing for two straight hours the other day. I've watched what seems like a dozen hours of swimming. I've watched badminton, whitewater rafting, boxing, a lot of gymnastics, volleyball, and a lot of stuff I can't remember. I've seen some pretty cool things too. The amazing come from behind US victory in the 4x100m men's freestyle relay was really cool to see. I've seen tons of records broken in swimming, which is kind of old hat since everyone's wearing that crazy new suit now. But, I saw a cool pair of races where one guy broke another guy's record, and the other guy broke it back in the next race. I watched an entire USA-Australia softball game last night that ended up as a no-hitter.

But, I think my favorite parts are the ones where a commentator has been talking all day and has just given up on sounding professional and just starts to fill time. Those times come often with this level of coverage. Just yesterday I listened to the women's volleyball commentator rail on about how bad the USA team was, and how it was shameful that they're really dishonoring their coach by sucking so much. Beach volleyball has Karch Kiraly providing "analysis", and he's usually good for something stupid. Today, the play-by-play guy asked Karch why a particular athlete would be wearing a watch, trying to find out why she wouldn't take it off for the match. Karch replied by listing all the things a volleyball player would be able to do before the match if they knew what time it was. Basically answering that watches are good for telling time. Thanks, Karch.