I don't know anyone who actually eats the peel. A 4 cal difference seems like a darn good estimate to me.
I always assume I am probably eating 100+ more cals than I log each day.
Now if someone chews gum everyday should they count it? Depends. I do low carb and a 1.5g piece of sf gum (the little square kind) has 1g carb and 2.5 cals. I usually chew at least 2 pc a day. I don't count them. They fall into my 100+ safety area.
Some people may chew a whole pack everyday and not count it. A 20 pack is 50 cals. Not that big of a deal to me but the 20 carbs is huge chunk of the amount I allow for myself each day.
When you add up all the "little things" most of us don't think to log it's easy to see why weight loss can be slowed down when we think we are doing everything right.
I don't think there could realistically ever be a way to know your exact cal/macro values even if you were able to track the weight on a .001oz. scale.
Would be kinda tempting to try though, huh?
"Just Do It"