Games designer, author and all-round very smart man Raph Koster has written a rather interesting piece on his personal site laying down some framework for just when you can call something a video game (or a game in general) and when you can't.
Spurred on by the likes of Dear Esther, which I thought was pushing the limits of what we can reasonably call a "game", and Farmville, which loads of people say is not a real "game", he offers up a clear definition for the term and a super-handy venn diagram illustrating it.
Playing a game is the act of solving statistically varied challenge situations presented by an opponent who may or may not be algorithmic within a framework that is a defined systemic model.
It's not a universal rule of course, just one man's opinion, but if this kind of stuff floats your boat it's well worth a read.
"X" isn't a game! [Raph Koster]