There was nothing safe about farming for food. Which is why people banded together for mutual defense and protection and usually depended on someone more militaristic to envelop them in a physical envelope, such as a walled city. Those who went it alone out in the boonies were low-hanging fruit.
Maybe thats why those psychopath/sociopath genes got weeded out. Those folk tend to anti herd and would have got killed off quickly.