There are actually some pretty clever methods of analysis in figuring out past climate, not necessarily temperature by itself. One of the better-known ones is tree ring analysis, which by overlapping the patterns in living trees with samples in artifacts can trace back several thousand years, which doesn't necessarily tell you that much about temperature but more about growing conditions, such as the combination of sun and rainfall. Less well-known, ice cores from glaciers and ice caps can go back tens of thousands of years and the microbubbles in the ice can be analyzed to determine atmospheric composition, like what the percentages of oxygen and carbon dioxide were during periods of before, during, and after the ice ages...however that produces a correlation, the causation part is a lot sketchier. Another method that does get a good picture of temperature ranges is examination of pollen grains in sediments, which can also go back thousands of years, pollen grains are damn' near indestructible, they are distinctive for different species of plants, and the temperature/humidity ranges in which particular plants flourish or fail are pretty well established.
DAT, I agree with pretty much all of what you said; however, I still feel that anything prehistoric is speculation. Perhaps accurate. Perhaps not so accurate. I'll try to explain my position. A scientist can go out tomorrow and cut down a tree. He can examine the growth rings of said tree. He can pull the recorded climate data for that area. Then he can go cut down another tree from another location and examine those growth rings. He can, once again, pull the recorded climate data for that area. He can do the same several times in several area, and after those comparing climate conditions and growth rings he can deduce that if you see characteristic "x" in the tree ring it corresponds with climate conditions "y". The problem is that just because "y" causes "x" today it does not necessarily mean that "y" caused "x" 10,000 years ago.
On a similar note, growing up I enjoyed eating oysters. I ate them prepared in many different ways. Based on that evidence it could safely be deduced that I am not allergic to oysters. Around age 12 something happened. Any time I ate oysters, no matter how they were prepared, I'd break out in hives and swell up everywhere. For several years I tried to eat oysters. Every time it was the same reaction. Hives and swelling. Based on that evidence it could safely be deduced that I am indeed allergic to oysters so I stopped eating them. Then when I was 20 years old I was dating a girl who wanted me to eat oysters. She knew I was allergic so I'm suspicious that she was curious about the swelling part. Being young and stupid I ate some for her. To her regret nothing happened. I've eaten oysters now for about 30 years without any problems. Based on that evidence, it could safely be deduced that I am NOT allergic to oysters. Once "y" didn't cause "x". Then "y" did cause "x". And then "y" didn't cause "x".