If we search for the definition of sport in the dictionary, we can see one of the main components of it is the physical exertion. Cambridge dictionary defines it as "a game, competition, or similar activity, done for enjoyment or as a job, that takes physical effort and skill and is played or done by following particular rules"

https://dictionary.cambridge.org/us/dictionary/english/sport

So we can understand both skill and physical effort are part of what can be considered a sport, and this is why activities such as golf, which lack this component can't be considered sports. Sure golf can be enjoyable and can be considered a display of skill, but it doesn't need physical exertion to practice it even at a professional level, so this is why it shouldn't be considered a sport.