The breast can swell as a result of radiotherapy, I think. Do you have a breast care nurse you could discuss this with?
I have ended up with breast lymphoedema, for which I see a lymphoedema therapist, and a masseuse that specialises in lymphatic massage for lymphoedema. I'm not suggesting that you have lymphoedema but a visit to a lymphoedema therapist may be worthwhile. If you don't know of one in your area, follow this link
https://www.lymphoedema.org.au/the-register/find-a-practitioner/Type in your postcode to find your closest therapist
Also, is your breast showing any signs of infection? Maybe see your GP to get that side of things checked out, too.
The shrinking of the breast, if it happens, happens slowly over time. It doesn't happen to everyone, in fact, I don't know what the frequency would be but I'd imagine it would be a minority.
Take care