Using that same logic, it's a good thing Americans didn't give up on Ronald Reagan in the very early 1980s, when things from the CarterDepression kept going downhill. Just as the Great Depression got worst after Franklin Roosevelt became president, so too did the CarterDepression 1981-1983. after the Incompetent One was gone.
Oops, I erred, though.
After a landslide victory in 1980, the Republicans suffered some substantial setbacks in the mid-term elections of 1982. (Unlike the Democrats in 1934, two years after the first Roosevelt landslide.)
But fortunately for America, and the world in general, the course had been righted by 1984, leading to the longest peace-time expansion of the economy and affluence in American history.