Bibliographic information:

Kasian, Stefan. 1995. Virtual Reality and Neurosurgery. Vertices 10(2): 7-8,60.

Imagine putting on a helmet and a pair of gloves and traveling to the far corners of the earth, visiting the great wonders of the world, touring the most beautiful museums... all without even leaving your home.

Imagine you decide to get in touch with your friend overseas, and in an instant his image appears in front of you, in full sensory detail. You decide to play a 3-D video game, and the two of you chase one another in spaceships through the far reaches of the galaxy.

Imagine that before making an important purchase, such as a wedding dress or a sprawling mansion, you have the opportunity to try your purchase on for size: you can walk through the rooms of your dream home in a sumptuous wedding dress.

Imagine no longer. Entertainment, business, technology, medicine ... many facets of life and many industries and professions could be changed by the revolutionary technology known as virtual reality (VR).

Virtual reality, in its final form, would produce a fully interactive world that appears "real" to the viewer. VR would simulate the five senses, and would augment or replace one's actual surroundings, in real time, as the event actually occurs. Typically, a simple helmet and a pair of gloves would interact with the user to achieve this effect. Virtual reality, in its purest form, is a dream technology that could revolutionize our lives. Developers have made much progress, but have yet to overcome many tough obstacles. The technology is still in its infancy, and its great potential is only beginning to be realized.

Virtual reality, in its cruder forms, is currently making its way into diverse fields. The military, for example, has used flight simulators for decades at places like the Army's National Training Center. This new technology saves time and damage to costly equipment and speeds training considerably. A decade of virtual reality-aided training has resulted in a 10% increase in average tank gunnery test scores. The military's December 1993 Interservice/Industry Training Systems and Education Conference achieved Disney-like proportions in its ability to transform fiction into reality:

In a showroom resembling a video arcade of the gods, the small exhibit is hardly noticeable. There amidst the most advanced systems... is a helmet mounted display. Place it on your head, strap a small device to the palm of your hand, and you are transported into a synthetic world of stunning detail and color.

Put your hand in front of your face, and an animated version appears before your eyes. Point in the direction you want to go, and with a flick of a finger you're flying down a broad avenue, darting through stone arches and skimming tree branches. You leap tall buildings with a single bound. With an upward turn of the wrist you soar dizzyingly upward; point downward, and you are plummeting to the ground like Superman, master of your domain. Totally interactive. Surprisingly real.

As you reluctantly relinquish your grasp on this new world, you notice a funny thing: The hand-held device is soaked with sweat. That simple response -- a physiological reaction to a synthetic environment -- reveals the tantalizing and as yet-unfulfilled promise of virtual reality. (Kitfield 1994)

The same conference hosted the largest-ever demonstration of Distributed Interactive Simulation, a sysem which enables participants at 59 simulators located around the globe to enter a common virtual environment--a virtual combat--and to interact with one another in real time. The system was not without imperfections, as crashes resulting from technical differences among the simulators were frequent, but the implications of even the partial success of the demonstration are astounding: all the virtual vehicles involved in the combat performed on the synthetic battlefield as one would expect them to in real combat, and all aspects of the battle could be immediately reviewed since the entire simulated battle is saved on a computer. The December conference served to confirm what many computer scientists and researchers have been saying for years: the potential of virtual reality technology is tremendous.

The military, while one of the first industries to make extensive use of the young technology, by no means has a monopoly on potential applications of VR. Modern medicine, which relies heavily on information acquired through diagnostic techniques, stands to benefit greatly from virtual reality applications.

The Department of Computer Science at the University of North Carolina at Chapel Hill (UNC-CH) is among the leaders in research and development of this new technology. The department was recently awared a three-year contract from the Advanced Research Projects Agency (ARPA) to investigate medical applications of image-guided procedures, including needle biopsies, fetal examination, and radiation treatment planning.

Ultrasound, a common diagnostic tool used in medicine today, also takes on a new dimension with virtual reality. Currently UNC is developing a system to project ultrasound images within 3-space around a subject. An observer wears a head-mounted display equipped with a small video camera. A computer receives the output of the camera and returns to the viewer the ultrasound images properly geared to the viewer's current position.

Research differs from practice, however, and significant obstacles must be overcome before virtual reality becomes as commonplace in medicine as the x-ray. When asked about the use of virtual reality in surgery today, Elizabeth Bullitt, a neurosurgeon at UNC, commented:

I don't think that surgery is ready for virtual reality yet... if you define virtual reality as the interactive software, then surgery is nowhere near that, nowhere close....

The U.S. government, however, is interested in robotics, having robot surgeons in the front line, and real surgeons running behind the scenes manning the robots. And then one must consider what happens when the lines go down. The control and sensitivity of the operating equipment must be drastically improved over the joystick that we have today. The sense of tissues and touch are critical as well.

Virtual reality surgery is a very long way away, and actually in terms of medical imaging, still far from getting computers in the operating room.

Dr. Bullitt notes four major problems that must be resolved before virtual technology becomes practical in the operating room. The first is registration--that is, the ability to map one image onto another and still preserve accurate alignment when the initial image is rotated. Related to this are problems of display such as interposing a CAT scan onto another image. This display problem is far from trivial, and is currently an active area of research.

The second problem is hardware related: computer speed and accurate simulation of touch. Virtual reality requires that a scene be repeatedly rendered as more data is input in order to accurately portray an image and reflect the changing position of the subject. Even the fastest hardware in the world can still only yeild crude renderings at best. For example, in a promising system of rendering using pixel planes, developed by Henry Fuchs at the Department of Computer Science at UNC, a chair would perhaps be described with several rectangles. This unfortunately does not yeild a very real appearance. As Dr. Bullitt explains, "the software attempts to maximize the hardware; however, the system is still limited by the speed of the hardware." More powerful hardware is being developed: parallel processors, for instance, multiply the rate at which information is processed. Nonetheless, no existing processor is fast enough to provide a realistic simulation.

Once images can be updated accurately, Dr. Bullitt explains, work must be done on simulating touch: "The touch, feel, and resistance of tissue is very important to a surgeon. Simulating touch to a surgeon's degree of sensitivity and precision is a long way off."

The third major challenge is particular to neurosurgery, though analogous situations arise in other surgical procedures. This is the problem of aligning the patient's head with images from a scan. This is not a simple problem, as the patient exists in 3-space but the images from the scan are rendered in two dimensions. How, for example, should the patient and the image from the NMR or CAT scan be aligned so that when a surgeon touches the patient's nose, the surgeon will be able to efficiently interact with the corresponding images?

So far, there are a number of different ways to go about this:

Make use of external fiducial (coordinate) systems that hold the patient still with markers, and bring the patient into the operating room. Then use a rotation matrix to align the patient's head with the same markers that are on the CAT or NMR scans.

Pizer has created a workable system using ultrasound. It requires the entire ceiling to be wired with transmitters that can send a beam of light, and cameras that aim toward the ceiling. A hand-held ultrasound monitor also contains external computer controlled cameras. The cameras and light beams can be used to determine the location and angle of both the ultrasound unit and the patient's head. Ultimately, the ultrasound unit would be mounted on a set of goggles.

Another way to reconcile the 3-space head with the 2-space image is to mix surgery with computer graphics. For example, suppose a probe is placed into the brain. If the CAT scan or NMR dataset is precomputed and stored, it is easy to represent that probe from any angle. This can be done with an articulated mechanical arm. The patient's head is fixed, the arm moves and passes the probe into the brain, and the probe's position can be represented on the computer. Ultimately, though, the mechanical arm is pretty clunky.

The fourth major obstacle is updating image changes during surgery. For example, during the removal of a tumor, the original CAT scan becomes less and less accurate as the surgery progresses. Standard NMR or CAT scans cannot be used to update the images during surgery. Ultrasound could be used, but it would produce a lower quality image. The problem of how best to mix the updated (lower-quality) ultrasound images with the old (higher-quality) CAT scan data is a very complex one.

Another way to update changes during a procedure is to use magnetic resonance machines with open magnets in the operating room. Before starting the operation, the patient's head would be put into the magnet and scanned continually during the operation. Unfortunately, these machines are very large and interfere with a surgeon's mobility. They also present an added challenge in preserving a sterile environment. However, combining ultrasound with an open magnet is one of the methods people are looking at for updating in-surgery changes.

Although these problems may appear difficult, they are far from insurmountable. Dr. Bullitt offers the following perspective of the progress that medical technology has made in the last few decades:

Medical imaging has come a long way. It began with the angiography and encephalogram... you would insert the dye into a patient's head with needles, and see ventricles. Cat scans came later, at the end of my medical training. Initially they were crude, and contained about 100 pixels per image, but this quickly improved in quality. Now with spiral CAT scans and NMR we can obtain 3-D data sets that can be rotated in real time at any angle. This added tremendously to patient diagnostics, treatment of disease, and long-term care. Computers have played a principal role in the progress we have seen in medical imaging. Virtual reality is mostly in computer science now, and this is where most of the problems will be solved.

The technology of virtual reality is still in its infancy, and numerous practical problems must be overcome, yet VR has tremendous potential to impact on many aspects of human life, from global business interaction to precision microsurgery. Virtual reality promises to enable the exploration of unknown worlds, both within the brain and without.

Reference:

Kitfield, James. 1994. Government Executive, p.60.

At the time this article was written, Stefan Kasian was a Trinity College junior majoring in computer science.

©1995 Duke University Undergraduate Publications. Reproduction, except for personal use, is strictly prohibited without prior written consent of the author(s) or Duke Undergraduate Publications. For more info...

Winter 1995 contents

Vertices web central