You could also discuss this and your confusion with your GP. Sometimes it is easier to have a discussion with them, rather than the surgeon, it might help you to feel more comfortable with the findings.
My sister had a Phyllodes tumour. She was given the option of having it removed or just leaving it there and monitoring it. She chose to remove it, but apparently the surgeon wasn't too concerned about it.
She has yearly mammograms because of it.
You can chose to have more frequent mammograms and ultrasounds if you want, just get your GP to do a referral. You just have to pay for them, that's all, and have them done at PRP or similar private diagnostic rooms.
Maybe you might like to have one done in a year and if it is okay you will feel better about going back to two yearly, or be guided by your GP.