I agree, although it's probably cultural, both of us having been born and raised in the upper Great Plains states, where a boy takes his "man" name about the time he's 10-12 years old.
I can respect that it's a southern tradition. We had our own "JohnnyReb" here, to whom I always referred to as "John." I just didn't feel decent, addressing a man whom I greatly admired, by a boy's name.
In real life, if I really knew and interacted with this character, I'd call him "Rob" to his face--"Robert" or "Robin" being a little too formal, but better than his little-boy's name--and he'd grow to like it.