Sometimes a look or a remark like, "What's your point?" will stop the person in their tracks who is giving you the backhanded compliment or in any way trying to bring you down.