I have found that stepping up to the plate to take on the hard ones that need to be done will hurt in the short run, but will pay off over and over again in multiple ways over the long haul. And, if you let opportunity in when it knocks, it tends to be a more and more frequent visitor.
I've had the lucky route. When I was working as an Electronics Technician, there was a point where my budget was the only one on the project that showed a profit. Then they gave me a manager who violated policy and charged thirty hours a week to my project (most of his hours were spent running the rec committee and feeling up various female workers) and had the mechs start charging my code to fix all of their screw ups as I caught them.
The PM yanked the work out of the building, and another group who had observed me picked me up to be on the PM team. Doing PM work proved to be enjoyable, the tasks vary, there is interesting work to do, and generally your off to something new every six months or so.
The hardest part of that line of work for me is putting up with the Electrical Engineers. I have no idea what color the sky is in their world.