Several years back my GP and I had a conversation about this. Turns out that after paying malpractice insurance premiums and some other overhead she took home less per year than I did and she worked much longer hours. Specialists do make the big bucks but your average GP doesn't make all that much compared to some other professions.
I don't work in the medical field, but my impression is that, possibly excepting specialists, the insurance companies and government have squeezed a lot of the profitability out of the medical profession. I'm well aware of all the touchy-feely blather about profit and medical practice, but I also know that if doctors cannot make a good living in proportion to their education, skills, and hours, they're going to leave practice. I think we are seeing this happen, though at this point it's often early retirement rather than career changes.