I'm with @Afraser. Give it a go for six months and see what happens. There's a reason it's prescribed: it improves survival rates.
The ultimate question to ask yourself, if you don't take it and your cancer metastasises or recurs, will you be tortured by your decision? Not to put too fine a point on it, but I would imagine you'd have to be able to look your loved ones in the eye with confidence if the worst happens.
The decision is yours and yours alone. It is your body, your life. You have to be comfortable with your decision, either way. I wish you the best of luck with this difficult decision. K xox