There's something very wrong if you never see users using your product.
For one thing, if you're in B2B you should have a whole network of customers and end-users at your customer's organizations. You should be out of the office and on the ground for at least several hours a week. Why? Because you don't learn things actionable about why people use (or don't use) your product or what value it creates for them if you're sitting around your own office.
And on a completely different not, I would say that for me personally, watching users use/abuse my products is far and away the most fulfilling thing about being a product manager.