My family all lived in Middlesex County, New Jersey (within sight of New York City) up until 1950. They were small farmers, cabinet makers, tavern owners, etc... pretty much middle class. I have a number of their wills from the pre-Civil War era. Every one of them owned a couple of slaves that they willed to their children.