Those who pray for your downfall are concentrating negative thought towards you without cognisance of slippery ground in which they are standing,which could lead to their downfall..
There are many reasons of such kinds of thoughts. Out of them two main reasons are most important. Some who have no negativity just advising or expressing their views for improvment. I often take it in positive way. On the other hand some views, we feel smell of jealousy. I just over look them.