Using a BMI calculator, your BMI is 18.4, which is 'underweight.' Why would you worry about 4 pounds when you are underweight?
I'm pretty sure that if you ask your doctor, his response would come from the best place of knowledge of your health and history. But I'd be surprised if he thought your gaining 4 pounds was a cause for alarm, given your BMI. Check with him anyway - if you want to know when to start worrying, he's your best source of info.