Bingo! This is not a myth. The phrase “passing the buck” does come from the card table. According to the Random House Historical Dictionary of American Slang, a buck was a token used in poker to indicate the dealer. According to the New Oxford American Dictionary, the buck was originally a “buck-handled knife,” by which they presumably mean a deer antler. It was the next person’s turn to deal, you passed the buck, or the knife, to him. Theoretically, someone who didn’t want the responsibility of dealing could pass the buck to another player.
The term seems to come into use around the time of the Civil War. Mark Twain’s Roughing It (1871) says, “I reckon I can’t call that hand. Ante and pass the buck.”
Previous comments:
History Myths Debunked 
I don’t always post a comment, but I always enjoy this blog. Thanks for the amusing information.
Well! What a pleasant surprise! I thought for sure this one would be a myth. How interesting!
The version I read some fifty years ago was that the ‘buck’ was a $2 bill. Some people thought $2 bills were unlucky (why?), and in order to avert the bad luck that possessing one would bring a receiver of such would tear a corner off before passing it along. Once all four corners had been torn it would be difficult to find someone willing to accept it in payment, so frequently the buck would stop there (with the fourth tearer).