It's fine. People keep hijacking the terms though. What you're referring to is most likely plant-based diet folks. Should be called vegetarians except they started eating eggs, milk, etc yet still want the label. Then you have nutty vegans who rank living organisms and don't mind their own business.
Anyhoo, plant based diet folks are cool and don't mind others' business.
No, it’s not. I’m vegan and people often assume I’ll be offended if they meat or certain types of meat in front of me. Honestly, I’m not bothered by their food choices at all. It gets embarrassing when some of them apologize. So don’t worry, it’s not rude. They have made their choice and you make yours.