Who neglects the lats certainly not bodybuilders?
Those legs will shrink as the weight loss comes off, and as they do they will become weaker and less functional. Muscle that isn't worked while losing weight is cannibalized.
Are you not doing a full body workout?
Your not doing this...