The conventional wisdom is to eat something small about half an hour before you exercise.
But it really comes down to you. Do you feel lethargic or low energy on your ride? Do you get over-hungry after a ride and eat too much? If not, then keep doing what you are doing. If it works for you the way you do it now, then don't change it.
When I run in the mornings, sometimes I just have coffee with milk beforehand and I am good. Or I eat a 100-calorie granola bar about 5 minutes before the run. It doesn't seem to sit in my stomach and it gives me a little energy. And I have been doing this for years.
You could take gels or chews on your bike to eat in case you get hungry. Or drink a little milk beforehand to get a little nutrition without it sitting in your stomach.