Short answer is no. It is dangerous and illegal to do this. They are governed under the same CAA laws as planes.
However this is good news as you can report someone to the CAA if they are flying illegally and I would recommend you do. The best advice I can give you is to take your cell phone and take video of where they are flying if possible and then follow it back. Most of the “toys” only have a 15-20min flight time. Film it landing and wait for it to land before talking to the pilot.
In most cases the pilots just did not know. That is the biggest problem with these things being sold as toys.
If they are flying over your property they must have your permission to do so. Make them aware of this and refer them to this site or one of the following.
If they remain arrogant or get violent back away as this person should not be a pilot and file a complaint with the CAA so they can follow up on it. The do follow up and the fines are not nice to receive if they find the pilot guilty.
If you were worried about privacy take the same video evidence to your local police station and let them know you have also filed it with the CAA. Full size pilots and us legal operators will thank you.
If you have any questions please don’t hesitate to drop them here. But please don’t shoot them down unless you want the police knocking at your door.