I get the idea, I just don't think it's possible to actually think of everything ahead of time.
I think you can be in at least 99% of the cases.
Probably, yeah, but you have to handle that 1% of cases (or whatever the number is). You need to have something that steps in and notices that this is the one-in-N case where the system fails and to stop it.
This is guesswork, but I'd say that the current system works adequately for the ninety-something percent of cases, as well, and it doesn't fail catastrophically on most of the remaining ones. I don't have any hard figures on this, obviously, so I could be over- or under-estimating by a lot. Still, my intuition here is that
1) Most cases are unexceptional, and could be handled equally well either way.
2) For every system, there's a small fraction of cases where one would be clearly worse than the other.
3) For each of the above sets, there a small fraction where one system leads to awful (as opposed to just bad) results, and the other doesn't.
4) I don't have a decent estimation for the relative sizes the set of cases where each set fails, but it seems likely that the human-judge system fails catastrophically much less than the alternative.
This is, like I said, intuition. At this point, all I can say is that my (limited) experience in trying to formalise things human care about in an objective way tells me it tends to be harder than it appears at first, that it's easy to mistakenly judge the project complete, and that when it fails, it does so rather horribly. This experience is more relevant to ethics in general than to judicial rulings, which is probably an easier subset to deal with, so I could be overestimating the difficulty of the problem. I cannot think of an easy way to test any of my assumptions. The
obvious way of testing it would be to build your system and then check it against judicial rulings in a large set of cases, (and find some way to determine failure and catastrophic failure, which is non-trivial), but that's not practical.