Vernor Vinge
                          Department of Mathematical Sciences
                             San Diego State University

        

         (c) 1993 by Vernor Vinge
                    (This article may be reproduced for noncommercial
                        purposes if it is copied in its entirety,
                               including this notice.)    

   
                          The original version of this article
                        was presented at the VISION-21 Symposium
                       sponsored by NASA Lewis Research Center and
                     the Ohio Aerospace Institute, March 30-31, 1993.
                       A slightly changed version appeared in the
                       Winter 1993 issue of _Whole Earth Review_.

                             

   
                                       Abstract

                     Within thirty years, we will have the technological
                means to create superhuman intelligence. Shortly after,
                the human era will
be ended.
 
                     Is such progress avoidable? If not to be avoided, can
                events be guided so that we may survive?  These questions
                are investigated. Some possible answers (and some further
                dangers) are presented.

             _What is The Singularity?_
 
                  The acceleration of technological progress has been the central
             feature of this century. I argue in this paper that we are on the edge
             of change comparable to the rise of human life on Earth. The precise
             cause of this change is the imminent creation by technology of
             entities with greater than human intelligence. There are several means
             by which science may achieve this breakthrough (and this is another
             reason for having confidence that the event will occur):
                o There may be developed computers that are "awake" and
                  superhumanly intelligent. (To date, there has been much
                  controversy as to whether we can
create human equivalence in a
                  machine. But if the answer is "yes, we can", then there is little
                  doubt that beings more intelligent can be constructed shortly
                  thereafter.)
                o Large computer networks (and their associated users) may "wake
                  up" as a superhumanly intelligent entity.
                o Computer/human interfaces may become so intimate that users
               
       may reasonably be considered superhumanly intelligent.
                o Biological science may provide means to improve natural
                  human intellect.
 
                  The first three possibilities depend in large part on 
             improvements in computer hardware. Progress in computer hardware has
             followed an amazingly steady curve in the last few decades [17]. Based
             largely on this trend, I believe that the creation of greater than
             human intelligence will occur during the next thirty years.  (Charles
             Platt [20] has pointed out that AI
enthusiasts have been making claims
             like this for the last thirty years. Just so I'm not guilty of a
             relative-time ambiguity, let me more specific: I'll be surprised if
             this event occurs before 2005 or after 2030.)
 
                  What are the consequences of this event? When greater-than-human
             intelligence drives progress, that progress will be much more rapid.
             In fact, there seems no reason why progress itself would not involve
             the creation of still more intelligent entities -- on a still-shorter
             time scale. The best analogy that I see is with the evolutionary past:
             Animals can adapt to problems and make inventions, but often no faster
             than natural selection can do its work -- the world acts as its own
             simulator in the case of natural selection. We humans have the ability
             to internalize the world and conduct "what if's" in our heads; we can
             solve many problems thousands of times faster than natural selection.
            
Now, by creating the means to execute those simulations at much higher
             speeds, we are entering a regime as radically different from our human
             past as we humans are from the lower animals.
 
                  From the human point of view this change will be a throwing away
             of all the previous rules, perhaps in the blink of an eye, an
             exponential runaway beyond any hope of control. Developments that
             before were thought might only happen in "a million years" (if ever)
             will likely happen in the next century. (In [5], Greg Bear paints a
             picture of the major changes happening in a matter of hours.)
 
                  I think it's fair to call this event a singularity ("the
             Singularity" for the purposes of this paper). It is a point where our
             old models must be discarded and a new reality rules. As we move
             closer to this point, it will loom vaster and vaster over human
             affairs till the notion becomes a commonplace.
Yet when it finally
             happens it may still be a great surprise and a greater unknown.  In
             the 1950s there were very few who saw it: Stan Ulam [28] paraphrased
             John von Neumann as saying:
 
                  One conversation centered on the ever accelerating progress of
                  technology and changes in the mode of human life, which gives the
                  appearance of approaching some essential singularity in the
                  history of the race beyond which human affairs, as we know them,
                  could not continue.
 
                  Von Neumann even uses the term singularity, though it appears he
             is thinking of normal progress, not the creation of superhuman
             intellect. (For me, the superhumanity is the essence of the
             Singularity. Without that we would get a glut of technical riches,
             never properly absorbed (see [25]).)
 
                  In the 1960s there was recognition of some of the implications of
             superhuman intelligence. I. J. Good
wrote [11]:
 
                  Let an ultraintelligent machine be defined as a machine
                  that can far surpass all the intellectual activities of any
                  any man however clever.  Since the design of machines is one of
                  these intellectual activities, an ultraintelligent machine could
                  design even better machines; there would then unquestionably
                  be an "intelligence explosion," and the intelligence of man
                  would be left far behind.  Thus the first ultraintelligent
                  machine is the _last_ invention that man need ever make, 
                  provided that the machine is docile enough to tell us how to
                  keep it under control.
                  ...
                  It is more probable than not that, within the twentieth century,
                  an ultraintelligent machine will be built and that it will be
                  the last invention that man need make.
 
                  Good has captured the essence of the runaway, but does not pursue
            
its most disturbing consequences. Any intelligent machine of the sort
             he describes would not be humankind's "tool" -- any more than humans
             are the tools of rabbits or robins or chimpanzees. 
 
                  Through the '60s and '70s and '80s, recognition of the cataclysm
             spread [29] [1] [31] [5]. Perhaps it was the science-fiction writers
             who felt the first concrete impact.  After all, the "hard"
             science-fiction writers are the ones who try to write specific stories
             about all that technology may do for us.  More and more, these writers
             felt an opaque wall across the future. Once, they could put such
             fantasies millions of years in the future [24].  Now they saw that
             their most diligent extrapolations resulted in the unknowable ...
             soon. Once, galactic empires might have seemed a Post-Human domain.
          
       Now, sadly, even interplanetary ones are.
 
                  What about the '90s and the '00s and the '10s,
as we slide toward
             the edge? How will the approach of the Singularity spread across the
             human world view? For a while yet, the general critics of machine
             sapience will have good press. After all, till we have hardware as
             powerful as a human brain it is probably foolish to think we'll be
             able to create human equivalent (or greater) intelligence. (There is
             the far-fetched possibility that we could make a human equivalent out
             of less powerful hardware, if we were willing to give up speed, if we
             were willing to settle for an artificial being who was literally slow
             [30]. But it's much more likely that devising the software will be a
             tricky process, involving lots of false starts and experimentation. If
             so, then the arrival of self-aware machines will not happen till after
             the development of hardware that is substantially more powerful than
             humans' natural equipment.)
 
                  But as time passes, we should
see more symptoms. The dilemma felt
             by science fiction writers will be perceived in other creative
             endeavors.  (I have heard thoughtful comic book writers worry about
             how to have spectacular effects when everything visible can be
             produced by the technologically commonplace.) We will see automation
             replacing higher and higher level jobs. We have tools right now
             (symbolic math programs, cad/cam) that release us from most low-level
             drudgery. Or put another way: The work that is truly productive is the
             domain of a steadily smaller and more elite fraction of humanity. In
             the coming of the Singularity, we are seeing the predictions of _true_
             technological unemployment finally come true.
 
                  Another symptom of progress toward the Singularity: ideas
            
     themselves should spread ever faster, and even the most radical will
             quickly become commonplace.  When I began writing science fiction in
             the middle '60s,
it seemed very easy to find ideas that took decades
             to percolate into the cultural consciousness; now the lead time seems
             more like eighteen months. (Of course, this could just be me losing my
             imagination as I get old, but I see the effect in others too.) Like
             the shock in a compressible flow, the Singularity moves closer as we
             accelerate through the critical speed.
 
                  And what of the arrival of the Singularity itself? What can be 
             said of its actual appearance? Since it involves an intellectual
          
       runaway, it will probably occur faster than any technical revolution
             seen so far.  The precipitating event will likely be unexpected --
             perhaps even to the researchers involved. ("But all our previous
             models were catatonic! We were just tweaking some parameters....") If
             networking is widespread enough (into ubiquitous embedded systems), it
             may seem as if our artifacts as a whole had suddenly wakened.

And what happens a month or two (or a day or two) after that? I 
             have only analogies to point to: The rise of humankind. We will be in
             the Post-Human era. And for all my rampant technological optimism,
             sometimes I think I'd be more comfortable if I were regarding these
        
         transcendental events from one thousand years remove ... instead of
             twenty.

   
             _Can the Singularity be Avoided?_
 
                  Well, maybe it won't happen at all: Sometimes I try to imagine
             the symptoms that we should expect to see if the Singularity is not to
             develop.  There are the widely respected arguments of Penrose [19] and
             Searle [22] against the practicality of machine sapience.  In August
             of 1992, Thinking Machines Corporation held a workshop to investigate
             the question "How We Will Build a Machine that Thinks" [27]. As you
             might guess from the workshop's title, the participants were not
             especially supportive
of the arguments against machine intelligence.
             In fact, there was general agreement that minds can exist on
             nonbiological substrates and that algorithms are of central importance
             to the existence of minds.  However, there was much debate about the
             raw hardware power that is present in organic brains. A minority felt
             that the largest 1992 computers were within three orders of magnitude
             of the power of the human brain.  The majority of the participants
           
      agreed with Moravec's estimate [17] that we are ten to forty years
             away from hardware parity. And yet there was another minority who
             pointed to [7] [21], and conjectured that the computational competence
             of single neurons may be far higher than generally believed. If so,
             our present computer hardware might be as much as _ten_ orders of
             magnitude short of the equipment we carry around in our heads. If this
             is true (or for that matter, if the Penrose or Searle critique
is
             valid), we might never see a Singularity. Instead, in the early '00s
             we would find our hardware performance curves beginning to level off
             -- this because of our inability to automate the design work needed to
             support further hardware improvements. We'd end up with some _very_
             powerful hardware, but without the ability to push it further.
             Commercial digital signal processing might be awesome, giving an
             analog appearance even to digital operations, but nothing would ever
             "wake up" and there would never be the intellectual runaway which is
             the essence of the Singularity. It would likely be seen as a golden
             age ... and it would also be an end of progress. This is very like the
             future predicted by Gunther Stent.  In fact, on page 137 of [25],
             Stent explicitly cites the development of transhuman intelligence as a
             sufficient condition to break his projections.
 
                  But if the technological
Singularity can happen, it will. Even 
             if all the governments of the world were to understand the "threat"
             and be in deadly fear of it, progress toward the goal would continue.
             In fiction, there have been stories of laws passed forbidding the
             construction of "a machine in the likeness of the human mind" [13].
             In fact, the competitive advantage -- economic, military, even
             artistic -- of every advance in automation is so compelling that
             passing laws, or having customs, that forbid such things merely
             assures that someone else will get them first.
 
                  Eric Drexler [8] has provided spectacular insights about how far
             technical improvement may go. He agrees that superhuman intelligences
             will be available in the near future -- and that such entities pose a
             threat to the human status quo. But Drexler argues that we can confine
             such transhuman devices so that their results can be examined and
            
used safely.  This is I. J. Good's ultraintelligent machine, with a
             dose of caution. I argue that confinement is intrinsically
             impractical. For the case of physical confinement: Imagine yourself
             locked in your home with only limited data access to the outside,
             to your masters. If those masters thought at a rate -- say -- one
             million times slower than you, there is little doubt that over a
             period of years (your time) you could come up with "helpful advice"
             that would incidentally set you free. (I call this "fast thinking"
             form of superintelligence "weak superhumanity". Such a "weakly
             superhuman" entity would probably burn out in a few weeks of outside
       
          time. "Strong superhumanity" would be more than cranking up the clock
             speed on a human-equivalent mind.  It's hard to say precisely what
             "strong superhumanity" would be like, but the difference appears to be
            
profound. Imagine running a dog mind at very high speed. Would a
             thousand years of doggy living add up to any human insight? (Now if
             the dog mind were cleverly rewired and _then_ run at high speed, we
             might see something different....) Many speculations about
             superintelligence seem to be based on the weakly superhuman model. I
             believe that our best guesses about the post-Singularity world can be
             obtained by thinking on the nature of strong superhumanity. I will
             return to this point later in the paper.)
 
                  Another approach to confinement is to build _rules_ into the
             mind of the created superhuman entity (for example, Asimov's Laws
             [3]). I think that any rules strict enough to be effective would also
             produce a device whose ability was clearly inferior to the unfettered
             versions (and so human competition would favor the development of the
             those more dangerous models).  Still, the Asimov dream is a wonderful
            
one: Imagine a willing slave, who has 1000 times your capabilities in
             every way. Imagine a creature who could satisfy your every safe wish
             (whatever that means) and still have 99.9% of its time free for other
          
       activities. There would be a new universe we never really understood,
             but filled with benevolent gods (though one of _my_ wishes might be to
             become one of them).
 
                  If the Singularity can not be prevented or confined, just how bad
             could the Post-Human era be? Well ... pretty bad. The physical
             extinction of the human race is one possibility. (Or as Eric Drexler
             put it of nanotechnology: Given all that such technology can do,
             perhaps governments would simply decide that they no longer need
             citizens!). Yet physical extinction may not be the scariest
             possibility.  Again, analogies: Think of the different ways we relate
             to animals. Some of the crude physical abuses are implausible, yet....
             In
a Post-Human world there would still be plenty of niches where
             human equivalent automation would be desirable: embedded systems in
             autonomous devices, self-aware daemons in the lower functioning of
             larger sentients. (A strongly superhuman intelligence would likely be
             a Society of Mind [16] with some very competent components.) Some
             of these human equivalents might be used for nothing more than digital
             signal processing. They would be more like whales than humans. Others
             might be very human-like, yet with a one-sidedness, a _dedication_
             that would put them in a mental hospital in our era.  Though none of
      
           these creatures might be flesh-and-blood humans, they might be the
             closest things in the new enviroment to what we call human now. (I. J.
             Good had something to say about this, though at this late date the
             advice may be moot: Good [12] proposed a "Meta-Golden Rule",
             which might be paraphrased as "Treat
your inferiors as you would be
             treated by your superiors."  It's a wonderful, paradoxical idea (and
             most of my friends don't believe it) since the game-theoretic payoff
             is so hard to articulate. Yet if we were able to follow it, in some
             sense that might say something about the plausibility of such kindness
             in this universe.)
 
                  I have argued above that we cannot prevent the Singularity,
             that its coming is an inevitable consequence of the humans' natural
             competitiveness and the possibilities inherent in technology.  And yet
             ... we are the initiators. Even the largest avalanche is triggered by
             small things. We have the freedom to establish initial conditions,
             make things happen in ways that are less inimical than others. Of
             course (as with starting avalanches), it may not be clear what the
             right guiding nudge really is:

             _Other Paths to the Singularity: Intelligence Amplification_

When people speak of creating superhumanly intelligent beings, 
             they are usually imagining an AI project. But as I noted at the
             beginning of this paper, there are other paths to superhumanity.
             Computer networks and human-computer interfaces seem more mundane than
             AI, and yet they could lead to the Singularity. I call this
             contrasting approach Intelligence Amplification (IA). IA is something
             that is proceeding very naturally, in most cases not even recognized
             by its developers for what it is. But every time our ability to access
             information and to communicate it to others is improved, in some sense
             we have achieved an increase over natural intelligence. Even now, the
             team of a PhD human and good computer workstation (even an off-net
             workstation!) could probably max any written intelligence test in
             existence.
 
                  And it's very likely that IA is a much easier road to the 
             achievement of superhumanity
than pure AI. In humans, the hardest
             development problems have already been solved. Building up from within
          
       ourselves ought to be easier than figuring out first what we really
             are and then building machines that are all of that. And there is at
             least conjectural precedent for this approach.  Cairns-Smith [6] has
             speculated that biological life may have begun as an adjunct to still
             more primitive life based on crystalline growth.  Lynn Margulis (in
             [15] and elsewhere) has made strong arguments that mutualism is a
             great driving force in evolution.

        Note that I am not proposing that AI research be ignored or less
             funded. What goes on with AI will often have applications in IA, and
             vice versa.  I am suggesting that we recognize that in network and
             interface research there is something as profound (and potential wild)
             as Artificial Intelligence. With that insight, we may see projects
             that are not
as directly applicable as conventional interface and
             network design work, but which serve to advance us toward the
             Singularity along the IA path.
 
                  Here are some possible projects that take on special 
             significance, given the IA point of view:
                o Human/computer team automation: Take problems that are normally
         
             considered for purely machine solution (like hill-climbing
                  problems), and design programs and interfaces that take a
                  advantage of humans' intuition and available computer hardware.
                  Considering all the bizarreness of higher dimensional
                  hill-climbing problems (and the neat algorithms that have been
                  devised for their solution), there could be some very interesting
                  displays and control tools provided to the human team member.
                o Develop human/computer symbiosis in art: Combine the graphic
                  generation capability of modern machines and the esthetic
                 
sensibility of humans. Of course, there has been an enormous
                  amount of research in designing computer aids for artists, as
                  labor saving tools.  I'm suggesting that we explicitly aim for a
                  greater merging of competence, that we explicitly recognize the
                  cooperative approach that is possible. Karl Sims [23] has done
                  wonderful work in this direction.
                o Allow human/computer teams at chess tournaments. We already
                  have programs that can play better than almost all humans. But
                  how much work has been done on how this power could be used by a
                  human, to get something even better? If such teams were allowed
                  in at least some chess tournaments, it could have the positive
                  effect on IA research that allowing computers in tournaments had
                  for the corresponding niche in AI.
                o Develop interfaces that allow computer and network access without
                  requiring the
human to be tied to one spot, sitting in front of a
        
              computer. (This is an aspect of IA that fits so well with known
                  economic advantages that lots of effort is already being spent on
                  it.)
                o Develop more symmetrical decision support systems. A popular
            
          research/product area in recent years has been decision support
                  systems. This is a form of IA, but may be too focussed on
                  systems that are oracular. As much as the program giving the user
                  information, there must be the idea of the user giving the
                  program guidance.
                o Use local area nets to make human teams that really work (ie,
                  are more effective than their component members). This is
                  generally the area of "groupware", already a very popular
                  commercial pursuit. The change in viewpoint here would be to
                  regard the group activity as a combination organism. In one
                  sense, this
suggestion might be regarded as the goal of inventing
                  a "Rules of Order" for such combination operations. For instance,
                  group focus might be more easily maintained than in classical
                  meetings. Expertise of individual human members could be isolated
                
      from ego issues such that the contribution of different members
                  is focussed on the team project. And of course shared data bases
                  could be used much more conveniently than in conventional
                  committee operations. (Note that this suggestion is aimed at team
                  operations rather than political meetings. In a political
                  setting, the automation described above would simply enforce the
                  power of the persons making the rules!)
                o Exploit the worldwide Internet as a combination human/machine
                  tool. Of all the items on the list, progress in this is
                  proceeding the fastest and may run us into the Singularity before
                 
anything else. The power and influence of even the present-day
                  Internet is vastly underestimated. For instance, I think our
                  contemporary computer systems would break under the weight of
                  their own complexity if it weren't for the edge that the USENET
                  "group mind" gives the system administration and support people!
                  The very anarchy of the worldwide net development is evidence of
                  its potential. As connectivity and bandwidth and archive size and
                  computer speed all increase, we are seeing something like Lynn
                  Margulis' [15] vision of the biosphere as data processor
                  recapitulated, but at a million times greater speed and with
                  millions of humanly intelligent agents (ourselves).
 
                  The above examples illustrate research that can be done within 
             the context of contemporary computer science departments. There are
             other paradigms. For example, much of the
work in Artificial
             Intelligence and neural nets would benefit from a closer connection
             with biological life. Instead of simply trying to model and understand
             biological life with computers, research could be directed toward the
             creation of composite systems that rely on biological life for
             guidance or for the providing features we don't understand well enough
             yet to implement in hardware. A long-time dream of science-fiction has
             been direct brain to computer interfaces [2] [29]. In fact, there is
             concrete work that can be done (and is being done) in this area:
                o Limb prosthetics is a topic of direct commercial applicability.
                  Nerve to silicon transducers can be made [14].  This is an
                  exciting, near-term step toward direct communication.
                o Direct links into brains seem feasible, if the bit rate is
                  low: given human learning flexibility, the actual brain neuron
                  targets might not
have to be precisely selected.  Even 100 bits
                  per second would be of great use to stroke victims who would
                  otherwise be confined to menu-driven interfaces.
                o Plugging in to the optic trunk has the potential for bandwidths
                  of 1 Mbit/second or so. But for this, we need to know the
                  fine-scale architecture of vision, and we need to place an
                  enormous web of electrodes with exquisite precision.  If we want
                  our high bandwidth connection to be _in addition_ to what paths
                  are already present in the brain, the problem becomes vastly more
                  intractable. Just sticking a grid of high-bandwidth receivers
                  into a brain certainly won't do it.  But suppose that the
                  high-bandwidth grid were present while the brain structure was
                  actually setting up, as the embryo develops.  That suggests:
                o Animal embryo experiments. I wouldn't expect any IA success
                  in the first
years of such research, but giving developing brains
                  access to complex simulated neural structures might be very
                  interesting to the people who study how the embryonic brain
      
                develops.  In the long run, such experiments might produce
                  animals with additional sense paths and interesting intellectual
                  abilities.

                  Originally, I had hoped that this discussion of IA would yield 
             some clearly safer approaches to the Singularity. (After all, IA
             allows our participation in a kind of transcendance.) Alas, looking
             back over these IA proposals, about all I am sure of is that they
             should be considered, that they may give us more options. But as for
             safety ...  well, some of the suggestions are a little scarey on their
             face. One of my informal reviewers pointed out that IA for individual
             humans creates a rather sinister elite. We humans have millions of
             years of evolutionary
baggage that makes us regard competition in a
             deadly light. Much of that deadliness may not be necessary in today's
             world, one where losers take on the winners' tricks and are coopted
             into the winners' enterprises. A creature that was built _de novo_
             might possibly be a much more benign entity than one with a kernel
             based on fang and talon. And even the egalitarian view of an Internet
             that wakes up along with all mankind can be viewed as a nightmare
             [26].
 
                  The problem is not simply that the Singularity represents the 
             passing of humankind from center stage, but that it contradicts
             our most deeply held notions of being. I think a closer look at the
             notion of strong superhumanity can show why that is.

             _Strong Superhumanity and the Best We Can Ask for_
 
                  Suppose we could tailor the Singularity. Suppose we could attain
             our most extravagant hopes. What then would we ask for:
            
That humans themselves would become their own successors, that
             whatever injustice occurs would be tempered by our knowledge of our
             roots. For those who remained unaltered, the goal would be benign
             treatment (perhaps even giving the stay-behinds the appearance of
             being masters of godlike slaves).  It could be a golden age that also
             involved progress (overleaping Stent's barrier). Immortality (or at
             least a lifetime as long as we can make the universe survive [10]
             [4]) would be achievable.  
 
                  But in this brightest and kindest world, the philosophical 
     
            problems themselves become intimidating. A mind that stays at the same
             capacity cannot live forever; after a few thousand years it would look
             more like a repeating tape loop than a person.  (The most chilling
             picture I have seen of this is in [18].)  To live indefinitely long,
             the mind itself must grow ... and when it becomes great enough, and
             looks
back ... what fellow-feeling can it have with the soul that it
             was originally?  Certainly the later being would be everything the
             original was, but so much vastly more. And so even for the individual,
             the Cairns-Smith or Lynn Margulis notion of new life growing
             incrementally out of the old must still be valid.
 
                  This "problem" about immortality comes up in much more direct
             ways.  The notion of ego and self-awareness has been the bedrock of
             the hardheaded rationalism of the last few centuries. Yet now the
             notion of self-awareness is under attack from the Artificial
             Intelligence people ("self-awareness and other delusions").
             Intelligence Amplification undercuts our concept of ego from another
             direction.  The post-Singularity world will involve extremely
             high-bandwidth networking. A central feature of strongly superhuman
             entities will likely be their ability to communicate at variable
            
bandwidths, including ones far higher than speech or written messages.
             What happens when pieces of ego can be copied and merged, when the
             size of a selfawareness can grow or shrink to fit the nature of the
             problems under consideration?  These are essential features of strong
             superhumanity and the Singularity. Thinking about them, one begins to
             feel how essentially strange and different the Post-Human era will be
             -- _no matter how cleverly and benignly it is brought to be_.
 
                  From one angle, the vision fits many of our happiest dreams:
             a time unending, where we can truly know one another and understand
             the deepest mysteries.  From another angle, it's a lot like the worst-
             case scenario I imagined earlier in this paper.
 
                  Which is the valid viewpoint? In fact, I think the new era is
             simply too different to fit into the classical frame of good and
             evil. That frame is based on the idea of isolated,
immutable minds
             connected by tenuous, low-bandwith links. But the post-Singularity
             world _does_ fit with the larger tradition of change and cooperation
             that started long ago (perhaps even before the rise of biological
             life). I think there _are_ notions of ethics that would apply in such
             an era. Research into IA and high-bandwidth communications should
             improve this understanding.  I see just the glimmerings of this now
             [32]. There is Good's Meta-Golden Rule; perhaps there are rules for
          
       distinguishing self from others on the basis of bandwidth of
             connection. And while mind and self will be vastly more labile than in
             the past, much of what we value (knowledge, memory, thought) need
             never be lost. I think Freeman Dyson has it right when he says [9]:
             "God is what mind becomes when it has passed beyond the scale of our
             comprehension."

 
 
             [I wish to thank John Carroll of San Diego
State University and Howard
             Davidson of Sun Microsystems for discussing the draft version of this
             paper with me.]
 
             _Annotated Sources [and an occasional plea for bibliographical help]_
 
         [1] Alfve'n, Hannes, writing as Olof Johanneson, _The End of Man?_, 
          
          Award Books, 1969 earlier published as "The Tale of the Big
                Computer", Coward-McCann, translated from a book copyright 1966
                Albert Bonniers Forlag AB with English translation copyright 1966
                by Victor Gollanz, Ltd.
 
         [2] Anderson, Poul, "Kings Who Die", _If_, March 1962, p8-36.
                Reprinted in _Seven Conquests_, Poul Anderson, MacMillan Co., 1969.
 
         [3] Asimov, Isaac, "Runaround", _Astounding Science Fiction_, March 1942,
                p94. Reprinted in _Robot Visions_, Isaac Asimov, ROC, 1990.
                Asimov describes the development of his robotics stories in this
                book.
 
         [4] Barrow, John D. and Frank J. Tipler,
_The Anthropic Cosmological
                Principle_, Oxford University Press, 1986.
 
         [5] Bear, Greg, "Blood Music", _Analog Science Fiction-Science Fact_,
                June, 1983. Expanded into the novel _Blood Music_, Morrow, 1985.
 
         [6] Cairns-Smith, A. G., _Seven Clues to the Origin of Life_, Cambridge 
               
     University Press, 1985.
 
         [7] Conrad, Michael _et al._, "Towards an Artificial Brain", _BioSystems_,
                vol 23, pp175-218, 1989.
 
         [8] Drexler, K. Eric, _Engines of Creation_, Anchor Press/Doubleday, 1986.
 
         [9] Dyson, Freeman, _Infinite in All Directions_, Harper && Row, 1988.
 
        [10] Dyson, Freeman, "Physics and Biology in an Open Universe", _Review
                of Modern Physics_, vol 51, pp447-460, 1979.
 
        [11] Good, I. J., "Speculations Concerning the First Ultraintelligent 
     
               Machine", in _Advances in Computers_, vol 6, Franz L. Alt and
                Morris Rubinoff, eds, pp31-88, 1965,
Academic Press.
 
        [12] Good, I. J., [Help! I can't find the source of Good's Meta-Golden
                Rule, though I have the clear recollection of hearing about it
                sometime in the 1960s. Through the help of the net, I have found
                pointers to a number of related items. G. Harry Stine and Andrew
                Haley have written about metalaw as it might relate to
      
              extraterrestrials: G. Harry Stine, "How to Get along with
                Extraterrestrials ... or Your Neighbor", _Analog Science Fact-
                Science Fiction_, February, 1980, p39-47.]

        [13] Herbert, Frank, _Dune_, Berkley Books, 1985. However, this novel was
                serialized in _Analog Science Fiction-Science Fact_ in the 1960s.
 
        [14] Kovacs, G. T. A. _et al._, "Regeneration Microelectrode Array for
                Peripheral Nerve Recording and Stimulation", _IEEE Transactions
                on Biomedical Engineering_, v 39, n 9, pp 893-902.
 
        [15] Margulis, Lynn and Dorion
Sagan, _Microcosmos, Four Billion Years of
                Evolution from Our Microbial Ancestors_, Summit Books, 1986.
 
        [16] Minsky, Marvin, _Society of Mind_, Simon and Schuster, 1985.
 
        [17] Moravec, Hans, _Mind Children_, Harvard University Press, 1988.
 
        [18] Niven, Larry, "The Ethics of Madness", _If_, April 1967, pp82-108.
               
     Reprinted in _Neutron Star_, Larry Niven, Ballantine Books, 1968.
 
        [19] Penrose, Roger, _The Emperor's New Mind_, Oxford University Press, 
                1989.
 
        [20] Platt, Charles, Private Communication.
 
        [21] Rasmussen, S. _et al._, "Computational Connectionism within Neurons: 
                a Model of Cytoskeletal Automata Subserving Neural Networks", in
                _Emergent Computation_, Stephanie Forrest, ed., pp428-449, MIT
                Press, 1991.
 
        [22] Searle, John R., "Minds, Brains, and Programs", in _The Behavioral and
                Brain Sciences_, vol 3, Cambridge University Press, 1980. The
               
essay is reprinted in _The Mind's I_, edited by Douglas R.
                Hofstadter and Daniel C. Dennett, Basic Books, 1981 (my source
                for this reference). This reprinting contains an excellent critique
                of the Searle essay.
 
        [23] Sims, Karl, "Interactive Evolution of Dynamical Systems", Thinking 
                Machines Corporation, Technical Report Series (published in _Toward
                a Practice of Autonomous Systems: Proceedings of the First European
                Conference on Artificial Life_, Paris, MIT Press, December 1991.
 
        [24] Stapledon, Olaf, _The Starmaker_, Berkley Books, 1961 (but from
                the date on forward, probably written before 1937). 
 
        [25] Stent, Gunther S., _The Coming of the Golden Age: A View of the End
                of Progress_, The Natural History Press, 1969.
 
        [26] Swanwick Michael, _Vacuum Flowers_, serialized in _Isaac Asimov's
                Science Fiction Magazine_, December(?) 1986 - February 1987.
                Republished
by Ace Books, 1988.
 
        [27] Thearling, Kurt, "How We Will Build a Machine that Thinks", a workshop
                at Thinking Machines Corporation, August 24-26, 1992. Personal
                Communication.
 
        [28] Ulam, S., Tribute to John von Neumann, _Bulletin of the American 
                Mathematical Society_, vol 64, nr 3, part 2, May 1958, pp1-49.
 
        [29] Vinge, Vernor, "Bookworm, Run!", _Analog_, March 1966, pp8-40.
                Reprinted in _True Names and Other Dangers_, Vernor Vinge, Baen
                Books, 1987.
 
        [30] Vinge, Vernor, "True Names", _Binary Star Number 5_, Dell, 1981.
       
             Reprinted in _True Names and Other Dangers_, Vernor Vinge, Baen
                Books, 1987.
 
        [31] Vinge, Vernor, First Word, _Omni_, January 1983, p10.
 
        [32] Vinge, Vernor, To Appear [ :-) ].