There were of course a few downsides, such as every felony being a capital offense (And virtually every offense that is a misdemeanor now was a felony), the possibility of being called up for military service on the King, Duke, or Baron's demand for a given number of days per year (Essentially to serve as cannon fodder, if they'd had field artillery in the 12th Century, but you get the idea - the job of the peasant levies was to tie up the real soldiers on the other side with so much killing that the chivalry and household troops could make a successful shock attack), and of course being regarded as a 'resource' to be destroyed by any invader conducting a chevausee in what passed for strategic warfare in that period.