I agree @CRM. It would be really nice if relatives asked if you minded if they contacted the specialist you see and even nicer if they gave the specialist a heads up about the relationship, but in the final analysis your confidence in your specialist is a great bit of advertising so it’s hard to blame someone if they act on it. You have every right to remind your specialist of the confidentiality of your medical relationship or quite specifically raise your concern so he/she knows your position.