Hi there,
I'm also 37 (9,6 &4 yr olds) and have been on tamoxifen since November. I think it definitely affects my moods too, but I am too scared to go off it! I don't want to get secondaries (of course no one does), and for me I don't know how I would live with myself if I didn't do all that I can to stay healthy. In saying that, I still haven't changed my diet dramatically or started running marathons! I guess you need to consider how much better your life will be if you go off it, and whether you are prepared to risk it (only improves our stats slightly I think). Maybe you could talk to your psychologist about it as they may be able to help give you a more balanced opinion.
I didn't have chemo or radiation therapy after my mastectomy and so had a very short treatment period. I felt very emotional and abandoned when I wasn't seeing my breast surgeon anymore, but I have felt better emotionally as time goes on despite the tamoxifen.
Its a tough drug to take for some more so than others, and remember I'm pretty sure there are other drug options if the side effects get too bad. Talk to your onco too at your next appointment if it's not too far away.
Good luck, and lots of love to you,
Sam xxx