Super Computers Could Take Over the World

San Francisco - The co-founder of one of Silicon Valley's top technology companies believes scientific advances may be ushering humanity into a nightmare world where super smart machines force mankind into extinction.

In a heartfelt appeal published in the April issue of Wired magazine, Sun Microsystems chief scientist Bill Joy urges technologists to reconsider the ethics of the drive toward constant scientific innovation.

"We are being propelled into this new century with no plan, no control, no brakes," Joy writes. "The last chance to assert control - the fail-safe point - is rapidly approaching."

Joy's article comes as a rare cry of caution in an industry that thrives on relentless and often unplanned advances and is now riding the boom of a "new economy" expansion attributed to technological progress.

The warning is all the more disturbing because of the author's own impressive tech credentials. A leading computer researcher who developed an early version of the Unix operating system, Joy has more recently pioneered software technologies like Java and was co-chairman of a US presidential commission on the future of information technology.

Joy's fears focus on three areas of technology undergoing incredibly rapid change.

THINKING, BREEDING ROBOTS

The first, robotics, involves the development of "thinking" computers that within three short decades could be as much as a million times more powerful than those now available. Joy sees this as setting the groundwork for a "robot species" of intelligent robots that create evolved copies of themselves.

The second, genetics, deals with scientific breakthroughs in manipulating the very structure of biological life. While Joy says this has led to benefits such as pest-resistant crops, it also has set the stage for new, man-made plagues that could literally wipe out the natural world.

The third, nanotechnology, involves the creation of objects on an atom-by-atom basis, which before long could be harnessed to create smart machines that are microscopically small.

All three of these technologies share one characteristic absent in earlier dangerous human inventions such as the atomic bomb: they could replicate themselves, creating a cascade effect that could sweep through the physical word in much the same way a virus spreads through the computer world.

"It is no exaggeration to say we are on the cusp of the further perfection of extreme evil," Joy writes. "An evil whose possibility spreads well beyond that which weapons of mass destruction bequeathed to nation states on to surprising and terrible empowerment of extreme individuals."

OTHER INDUSTRY LEADERS NOT CONVINCED

Executives attending PC Forum, a technology industry conference in Scottsdale, Arizona, had mixed reactions to Joy's grim view on technology.

"You can't argue with the incredible improvements in thought, civilisation, productivity. The world is changing and I think people would say dramatically for the better," said Michael Campbell of SAP America, which develops software to help corporations manage their business.

Steve Kirsch, the founder of Infoseek and now chief executive of a new start-up called Propel, said he saw many instances where science and research is not moving fast enough, such as in the area of embryonic stem cell research, where the US government is too restrictive.

"If anything ... we are moving the opposite way, not faster," he said. "Lives are going to be lost."

But others said Joy's fears of technology running out of control were not entirely off the mark.

"From the sense that technologists fail to fully study or bother to understand the long range implications of technology, Bill Joy is right on the mark," said Steve Larsen, senior vice president of marketing at Net Perceptions, which develops personalisation software for the Internet.

UNABOMBER'S IDEAS

Joy says his new, darker vision of the potential threat to humanity posed by technology - one he notes is shared in part by convicted Unabomber Theodore Kaczynski - has led him to reconsider his own contributions to the field.

"I have always believed that making software more reliable, given its many uses, will make the world a safer place," Joy writes. "If I were to come to believe the opposite, then I would be morally obligated to stop this work. I can now imagine such a day may come."

Joy does hold out some hope, saying humanity's effort to control nuclear and biological weapons was evidence of the strength of the species' self-preservation instinct.

But he urges a wider dialogue on the implications of new tech advances and specifically asks that they be incorporated into the programme at the annual Pugwash Conferences, which began in 1957 as a forum for scientists to discuss the threat posed by nuclear weapons.

"The experiences of the atomic scientists clearly show the need to take personal responsibility, the danger that things will move too fast, and the way in which the process can take on a life of its own," Joy says. "We can, as they did, create insurmountable problems in almost no time flat. We must do more thinking upfront if we are not to be similarly surprised and shocked by the consequences of our inventions."
 

[Story originally published: March 14 2000 ]