Like the whole stupid "he didn't build that" thing. They KNOW what "that" refers to but intentionally take it out of context, every single time. And I call them on it every single time.
Either it was a poorly thought-out series of sentences (well, that's basically a given) or your relatives are correct. "That" could not possibly refer to "roads, bridges, etc." As others have pointed out, he's wrong even if you consider the overall point of the speech.
As I've noted, the spin on the meaning of the word that is nothing but a pathetic attempt at damage control, and this guy has bought hook, line, and sinker.
Oh, and I might as well mention again, for Obama to add " many wealthy, successful businessmen want to give something back" takes the speech to another level of absurdity.
It has just occured to me that I might not be so irritated if not for the number of people who buy what Obama is selling! I swear, some of his supporters even seem to live vicariously through him.