Another aspect is that a computer can only ever understand concepts like "The Greater Good" or "The Sanctity of the Individual" in terms of facts/numbers. The "Trolley Problem" that we agonize over is just another math problem to a computer. I has no soul so it'll just address it as math. "2 meatbags over here vs 7 meatbags over there. Choose to kill the two to save the seven."
Yes it could definitely be true AI if it chose to defy the programming.
But I would add that I disagree with your point on the programmer. It would seem from my reading that your assertion includes an assumption that the programmer would fully know his nature. And in my experience, they don't.
Sounds like we agree on all of this.
Another aspect is that a computer can only ever understand concepts like "The Greater Good" or "The Sanctity of the Individual" in terms of facts/numbers. The "Trolley Problem" that we agonize over is just another math problem to a computer. I has no soul so it'll just address it as math. "2 meatbags over here vs 7 meatbags over there. Choose to kill the two to save the seven."
Yes it could definitely be true AI if it chose to defy the programming.
But I would add that I disagree with your point on the programmer. It would seem from my reading that your assertion includes an assumption that the programmer would fully know his nature. And in my experience, they don't.
"You are what you eat". :)