Bingo! This is not a myth. The phrase “passing the buck” does come from the card table. According to the Random House Historical Dictionary of American Slang, a buck was a token used in poker to indicate the dealer. According to the New Oxford American Dictionary, the buck was originally a “buck-handled knife,” by which they presumably mean a deer antler. It was the next person’s turn to deal, you passed the buck, or the knife, to him. Theoretically, someone who didn’t want the responsibility of dealing could pass the buck to another player.
The term seems to come into use around the time of the Civil War. Mark Twain’s Roughing It (1871) says, “I reckon I can’t call that hand. Ante and pass the buck.”