[FoRK] Super-Intelligent Humans Are Coming

Stephen Williams sdw at lig.net
Fri Oct 24 00:48:24 PDT 2014

On 10/23/14, 9:58 PM, J. Andrew Rogers wrote:
>> On Oct 23, 2014, at 5:57 PM, Stephen D. Williams <sdw at lig.net> wrote:
>> One super-intelligent human has a <50% of being moral (the Superman Problem I'll call it) while a group (SuperFriends?) is very likely to be moral due to fundamental and derived effects.  If the first generation isn't, they'll likely kill each other off.  But some generation will get it right.
> Define “moral”.
> Human morality is baskets of evolved heuristics that reflect biology and other local constraints. Islands of stability and game theoretic optimality are not even consistent across human populations. And a lot of popular human morality is predicated on near-parity of intelligence, which is being discarded as an assumption here. We would lack the ability to even understand the context in which a super-intelligent person is evaluating the morality of an action.
> See also: human morality as evaluated by mice in a mouse context
> Compounding this, humans can rationalize almost any action as “moral” if placed in the proper context. A super-intelligent human could be expected to be superb at constructing the necessary context. Normal humans are able to do this regularly.

What example did you have in mind for "predicated on near-parity of intelligence"?
I would divide things people call "morals" into categories.  Some categories are like common law and natural law. (Murder, 
theft, etc.)  Some are probabilistically correct in some context.  Some are correct according to some mythical or fashionable 
story.  Some are new theories yet to be proven. (Various PC)

Each of those levels may have different applicability to widely disparate entities and cultural boundaries, just as humans have 
those divisions now.  But many of them will overlap.  Besides the mythical and fashionable ones, those we can drop now.

The ability of humans or any intelligence to falsely rationalize rather than correctly applying known facts is either laziness, 
lack of ability, or the presence of mendacity.  Your "constructing the necessary context" really sounds like falsely 
rationalizing by a method like straw man, false premises, false equivalence, correlation used as causation, or selective memory.


More information about the FoRK mailing list