A big reason a lot of these diets fail is because people do them short term. To truly lose weight you need to make a permanent change in how you live. A heavy, but steady, weight is preferable to a fluctuating one.
Actually, I thought the big reason a lot of these diets failed was because it required you to battle your own body, whereas a diet that worked would be one that worked with your body.
Depends on the diet in question, but to some extent any weight loss plan is battling your body. From a physiological perspective our bodies will do whatever it can to avoid losing weight and whatever it can to put on more weight.
I'm pretty sure that mostly applies during times of stress (the whole "must gain weight and never lose weight") but not so much during times of not-stress.
Anytime your body feels stressed, it goes into a preservation mode. Where it preserves everything as fat. One of the things that can cause this is "starvation mode"
Essentially, it's when your body thinks it is starving, but in reality it isn't. Some things that can cause this are losing weight too quickly, depriving yourself suddenly of specific nutrients, making any major reduction to your diet... when the body goes into starvation mode, it essentially clings tightly to the fat cells, holding onto them for as long as it possibly can.
...The trick isn't to wait it out, unless you have an immensely strong psyche (And if you had an immensely strong psyche and still got fat, then there's probably reasons why you're fat that no diet can really help with) but rather, to increase the amount of food you take in to end the starvation mode. Not too much, of course.
It's easy to say "It's healthiest to eat only x y and z", it's another to actually put that into practice. And that is where most fad diets fail. They focus on the goal and not the method you get there with.
The best "diet" is to eat less of everything... but not to cut it out entirely. Like chocolate? Have some every day. Just limit yourself on it to only a little bit. Craving that immensely delicious meal your parents make once a week? Have some, eat less of it.
The problem is that we (as a society) spend so much time vilifying fat people for being fat that we don't focus on the cause or the cure. We focus on the end goal, and that is what fad diets all aim towards.
People who only look at the goal will trip over every obstacle along the way. And that's terrible.