Perhaps you are driving your self too hard on your diet program? Keep in mind that men and women had a different set of nutrients that was necessary in their body.
Just wondering though? are you an athletic person? because of you are, you probably doesn't have enough energy giving nutrient that you need in your body.
and a p.s.
If you are rather worrying about your "physique" that is why you avoid eating well, take a note on this:
our metabolism could be greatly affected by your eating routine.. try to eat on the right time.