I feel like when I'm straining to do a sit up and especially a plank my ab muscle sticks out instead of staying flat. Trying to hold in my abs while really working hard isn't easy. As the workout continues it gets harder. Is this part of the process of building my ab muscles? Or am I doing something wrong? Are there specific ab workouts that can help this? Do I need to ahotel on my core? I need help please.

You aren't doing anything wrong. Everyones Ab muscles stick out when working. What you are probably seeing is the effect of fat stores above and below your Ab muscles. These don't contract like muscles, it is more like squeezing a pillow and having it bulge out everywhere else. As you lose the body fat their will be less and less to squeeze and the bulge will thin as just the muscle is contracted. You may want to consider squats and deadlifts as these are great at building the core muscles and not just the 6 pack group.