Normally each chart plotter which is able to display AIS has got the possibility to set an alarm. This alarm is called “CPA” = closest point of approach. Normally you can set this to 5nm and every vessel which comes closer would initiate an alarm at the plotter. Also it is the same with the most PC navigation software, which are able to display AIS. There should also be a setting.
If you do not use, either a plotter, nor a software I could recommend “Yacht AIS” from the company Y-tronic. They have a very simple, but very nice software which shows the other ships in a radar screen and which gives you also an alarm on the PC if the ship is in your CPA range.