Your dilemma is one shared by many of us. The thing is we are all different and reactions and side effects can vary a lot. There are some people who are fortunate to have very few effects and others who get the lot!
You can ask your oncologist for the statistics for your particular situation regarding the difference that hormone therapy might make to your chance of recurrence. For me the difference was very significant and that was very persuasive.
I then tried to keep an open mind about what might occur side effects wise. I hoped for the best but did find a few issues. Luckily I have found things that help minimise these effects and I think your body does learn to cope with them better over time. I now find my quality of life is just fine and the few things I have to do to live more comfortably with the hormone therapy are just a part of my day to day life now.
If you do give it a try and find you have a few effects, the ladies on here are great for advice. It is also important to let your oncologist know about issues as many things can have solutions. Don't just think there is nothing that can be done. Sometimes help is just there for the asking. I hope you find the right choice for you.