A character in C. S. Lewis's posthumously published novel fragment THE DARK TOWER asserts it is. (Granted, one faction within Lewis scholarship maintains THE DARK TOWER wasn't actually written by him, but I don't find that claim convincing. Anyway, the issue doesn't affect the point of the story.) He argues that physical travel to the past or future can't be done for a basic, irrefutable reason: A corporeal trip into a different time necessarily carries all the atoms in one's own body into that other time. But in the past, all those particles existed in other entities in the physical world, whether inanimate objects, living creatures, liquids, gasses, whatever. In the future, those same particles will again be distributed through the environment. The only way you could materialize in a different moment would be if duplicates of each of your atoms, molecules, etc. existed in the same place at the same time. According to the laws of physics as we know them, that's impossible. Therefore, physical time travel is forever, irrevocably ruled out, unless we invoke magic rather than science.
That story is the only place where I've encountered this argument, which strikes me as highly convincing. On this hypothesis, other temporal "locations" could be only viewed, never visited. Accordingly, Lewis's character has invented a device for viewing other times, although it turns out the true situation is more complicated than he believed.
While I've come across other stories of observing rather than traveling to some non-present time, I don't remember any that offer a theoretical grounding for the impossibility of temporal travel in the flesh. It's not unusual in time-travel fiction, however, for a traveler to be unable to exist in the same location more than once in the same moment. In Dean Koontz's LIGHTNING, a traveler can't visit a place/time where he already is/was. He's automatically shunted away from that point. In Connie Willis's series about time-traveling historians from a near-future Oxford University, the same prohibition applies, but it's not clear whether the simultaneous existence of two of the same person is outright impossible or would produce a catastrophic result if it accidentally happened. In such works as the Harry Potter series, THE TIME TRAVELER'S WIFE, and Robert Heinlein's "By His Bootstraps," on the other hand, any number of you can be in the same point in space/time at once.
To me, the former rule seems more plausible, because it makes the issue of the same material object being in two places at once less obvious, although I've enjoyed lots of fiction in the second category. One possible way to get around the problem raised in Lewis's DARK TOWER: Instead of a corporeal leap into a different time, travelers might project their consciousness and build temporary bodies in the other time by "borrowing" stray particles from the surrounding air, water, and earth. When the traveler released the borrowed matter to return to his or her point of origin, the particles would dissipate harmlessly into the environment. Another method of bypassing the problem shows up in the new QUANTUM LEAP series: The leaper's consciousness occupies the body of a person in the past, presumably suppressing the host's personality in a sort of temporary, benign possession. (The time-shift operated differently in the original series, while this version does leave unanswered the question of where the leaper's body is while his immaterial consciousness travels to multiple past eras.)
Margaret L. Carter
Carter's Crypt