If your memory is so bad to the point that your safety is at risk then I suggest you address this concern with your oncologist. I don't accept that it should go with the territory of being on hormone therapy treatment.
I also put up with other side effects, including brain fog while going through chemo treatment but the memory loss on an AI for me was out of this world! I sympathise with you and please get this addressed by your oncologist. They may have good suggestions for your individual needs. I also was diagnosed with hypothydoism, which also may have exacerbated my symptoms being on two different hormone therapies for two different reasons. Lots to consider.
Wishing you all the best.