I’ve done both; getting a proper fit is worth it.
This is something that may or may not save your life, and may or may not distract from your riding pleasure.
real world example: I’d love to buy an Arai. Some of them are supposedly Shoei shape. I’ve tried them on. Unfortunately my head isn’t QUITE the shape Arai had in mind and for a correctly sized Arai I get slight pressure points on the back of my head. Could I ride like that? Maybe. But it wouldn’t be anywhere near as pleasant.