It is really up to you, whether you take it or not.
Not everyone gets all of the side effects.
You could try and see. The Oncologists' advice sounds appropriate.
There are also other drugs to try that are similar, should you need to change.
I would double check the percentage of difference, taking or not taking the drug.
My difference is 11%. So I am continuing with it. (I'm on Exemestane)
The other thing is you can start and see, you can stop later. But keep in touch with your Medical team.
All the best, whichever you do.