User Tools

Site Tools


keckcaves:caves_trouble_shooting_guide

H1 KeckCAVES User's Manual v0.92

H2 General Care Information

* The CAVE walls are thin flexible projection screens and are very sensitive. Additionally, since they cannot be cleaned, it is imperative that they never be touched or bumped into. Users are advised to be careful when walking inside the CAVE, especially when stepping backwards (to avoid backing into a screen). The CAVE floor is made from a material strong enough to walk on, but to protect the image quality of the floor protection, always take off your shoes or wear the disposable booties provided in the CAVE room before stepping into the CAVE. * The CAVE wand and the head tracker/stereo glasses are very sensitive. They must be handled carefully and never be dropped. The stereo glasses run on their own batteries independent from the head tracker. To preserve battery life, the glasses' ear pieces should be folded in when the glasses are not in use (there is a power switch between the right earpiece and the frame that turns the glasses on as soon as they are unfolded). * The CAVE wand and the head tracker have their own rechargable batteries. To recharge them, they have to be plugged into one of the charge power adapters on the workstation table. The charge connector on the CAVE wand is next to the power switch, the charge connector on the head tracker is on the belt clip case next to the cable going to the head tracker. The charging LEDs next to the power connectors light up until the batteries are fully recharged. * The CAVE structure is stable, but the projector mounts are sensitive and carefully aligned with the screens and must never be touched or leaned on. The mirrors and projector lenses cannot be cleaned or dusted, and must never be touched. It is best to stay away from the structures around the CAVE entirely.

H2 Starting the CAVE

# Turn on control workstation screen. # Log into control workstation.\\The user name is “guest”. # Turn on all projectors.\\Double-click on the “Projectors ON” icon. The projectors will take about a minute to warm up and show an image. The default image is the KeckCAVES logo on a white background on the walls, and a grid of one foot by one foot white tiles on the floor. # Turn on tracking devices.\\The CAVE wand has a sliding switch on the right side of the case; the “on” position is towards the front. When on, the right LED on the case will light up green. Once the wand has established contact to the base station, the left LED will light up orange. If the middle LED lights up, the wand needs to be recharged.\\The head tracker has a sliding switch on the belt clip case. When on, the “Power” LED will light up green. Once the head tracker establishes contact to the base station, the “In Range” LED will light up orange. If the “Low Battery” LED lights up, the head tracker needs to be recharged.

H2 Starting a VR Application

* Each currently installed demo application has an icon in the “CAVE Demo Programs” folder on the control workstation's desktop. Double-clicking the icon will open the folder and show all demo applications. * Each demo application can be started by double-clicking its icon. If the application starts up successfully, a control window will appear on the control workstation's screen, and the default image on the CAVE screens will be replaced with the application's display. * If an application does not start up after double-clicking its icon, refer to the troubleshooting section. * Before stepping into the CAVE, a user should “gear up” by firmly attaching the head tracker's belt clip to some piece of clothing, carefully unfolding and putting on the stereo glasses, and grabbing the CAVE wand. It will take a few seconds until the tracking system locks onto the head tracker and the image apperars correctly. * If the image on the screen appears “warped” or does not change when a user moves her/his head, the head tracker might not be able to communicate with the tracking rails (above the CAVE, with the blue “blinkenlights”). The head tracker has two tiny microphones on the top front corners. The tracker will not work properly if the microphones are obstructed, for example by a visor. (Although not recommended as a fashion statement, it is advisable to wear baseball caps backwards, or not at all.) * If the image on the screen appears “double” and there is no depth perception, the stereo glasses might not be able to communicate with the projectors. The stereo glasses are controlled via an infrared signal; the infrared receiver is located between the eye glasses. The line of sight between the receiver and the projectors (behind the screens) must not be obstructed (headgear advisories apply as above). Sometimes the glasses do not turn on properly; in that case, taking them off, folding, and unfolding them again usually fixes the problem.

H2 Using a VR Application

Although VR applications have different purposes and do not typically work alike, they do have common grounds when performing basic tasks (analogous to how most desktop user interface systems provide more or less consistent “look and feel”). All VR applications only listen to the CAVE wand - the keyboard and mouse are completely ignored except for closing an application as described below - and all interactions are triggered by pressing single buttons or button combinations on the wand, or pushing the wand joystick. The current applications have no need for “clicking” any of the buttons; performing tasks typically involves pressing a button until something happens, and then releasing it again.

The VR programming toolkit used to write all VR applications is quite flexible in what a user has to do to perform certain tasks (and almost everything can be changed while an application is running), but all applications start up with a default configuration, which is described in the following sections and illustrated in Figure 1.

Basic interactions with VR applications fall into the following categories:

H3 Navigation

Navigation is the process of changing how the “real” coordinate system of the CAVE relates to the “virtual” coordinate system of the 3D environment displayed by a VR application. In other words, navigation allows a user to walk/fly through a 3D environment, or to pick up and move an entire environment. In the default configuration, VR applications offer two different modes of navigation.

H4 “Model-on-a-Stick” This mode allows a user to pick up a 3D environment by pressing the yellow (bottom-left) wand button, and move it by translating and/or rotating the wand while the button is pressed. This is much more difficult to explain in detail than to actually use.

In addition to translating and rotating, it is also possible to uniformly scale the 3D environment by pushing the red (top-left) wand button while the yellow button is already pressed (this is easiest when pressing the yellow button with the back of one's thumb, and the pushing the red button with the top of the thumb). While both buttons are pressed, moving the wand in the direction it is pointing will shrink the 3D environment, while moving the wand in the opposite direction will enlarge it. The center point of scaling is the tip of the wand. This can lead to a confusing effect when trying to scale a 3D object that is relatively far away from the wand: when enlarging, the model will not only become bigger, but also move away from the wand position, and the effect of enlarging will be offset by the model disappearing in the distance (this is unintuitive because objects in the real world can typically not be scaled). To avoid this confusing effect, it is best first to move the object to be enlarged to the wand position (poking it with the wand), and then to scale it.

H4 Fly-through This mode allows a user to fly through larger 3D environments by pushing the wand joystick. If the joystick is pushed towards the front of the wand, the user will fly in the direction the wand is pointing, with a velocity proportional to how much the joystick is pushed. Pulling the joystick towards the back will fly backwards. Pushing the joystick left or right will turn around the wand's up axis, with an angular velocity proportional to how much the joystick is pushed. The flying velocity is typically quite fast, so flying is less useful when examining smaller models, and more useful to navigate expansive environments (such as terrain maps at large scales).

H3 Program Control

Program control allows a user to change the behavior or look of a VR application - it is essentially a fancy way to refer to using an application's main menu. The VR toolkit provides a 2.5D user interface and menu system (essentially flat menus and dialogs just as on the desktop, but floating in space) that are used just as if the CAVE wand were a regular mouse.

Pressing the blue (top-right) wand button will bring up an application's main menu; when first opening, it will appear floating several inches in front of the wand, facing the user. While the blue button is pressed, the VR toolkit will display a red “laser ray” coming from the tip of the wand. This ray is used to select entries from the menu just as in desktop programs. If the blue button is released while the red ray is hitting a menu entry, the function associated with that button will be executed. To close the main menu without executing a function, one just points the ray away from the menu and releases the blue button.

The menu functions provided by the VR applications vary widely (depending, of course, on the application's function), there are some typical menu entries shared between applications. Most applications have a “Center Display” menu entry that will reset the application's coordinate system to the state at the start of the program.

H3 Locating

Locating is the process of executing a program function at a particular point in space (for example, evaluating a volumetric function or creating a streamline, slice or contour surface). What happens when locating depends on the application, but a common factor is that locating typically starts when some button is pushed, is continually updated while that button is pressed, and stops when the button is released. For example, when the locator function is to create a slice, the slice will be created at the wand's position when the button is pushed, will move along with the wand while the button is pressed, and then stay fixed once the button is released.

Note: The default environment configuration does not map any wand buttons to locating; to do so, a locator tool has to be created (see below).

H3 Dragging

Dragging is the process of picking up a single object or collection of objects in a VR application, and moving them in a fashion analogous to navigation. Similar to locating, dragging is initiated when some button is pushed. At this point, the VR application selects which object(s) a user wants to drag (typically by selecting the object that is poked by the wand, or is inside some region of space around the wand), and remembers the relative position of that object to the wand. Then, when the wand is rotated and/or translated while the button is pressed, that object will follow along. Dragging stops when the button is released.

The difference between locating and dragging is sometimes subtle; in general, the distinction is that locating performs a function at a position in space, whereas dragging selects an object (such as an atom in a molecular dynamics simulation) and performs a function on that object (typically moving it).

Note: The default environment configuration does not map any wand buttons to dragging; to do so, a dragging tool has to be created (see below).

H3 Tool Creation and Destruction

As mentioned above, the relation between buttons on the wand and program functions such as navigation or locating is not fixed and can be changed according to a user's needs during the application's runtime. The basic idea is that each button (or joystick axis) on an input device such as the wand can be connected to a “tool” that will perform a certain function when that button is pressed (or the joystick axis is pushed). Tools not only define what action happens when a particular button is pressed, but also how that action happens. For example, there are several different navigation tools a user can select from. All of them navigate, but they do it in different ways; which tool works best is sometimes dependent on what a user wants to do.

When a user pushes a button that currently has no tool assigned to it (in the default environment, these are the green (bottom-right) button, the trigger button on the wand's handle, and the “button” that is activated when the joystick is pushed down), the tool selection menu will appear. This works the same way as an application's main menu; a user can select a tool from any of the listed categories and map it to the pressed button using the red selection ray. If a tool was selected, and matches the pressed button(s) (some tools require more than one button), the tool will be created. Henceforth, pushing that button will activate the tool. For example, a user can map a locator tool (such as the 6-DOF Locator) to the trigger button and subsequently use the trigger to evaluate 3D functions or create slices or contour lines.

To release a mapped tool (so it can be replaced with a different one), a user has to press the tool's button while the wand is inside the “tool destruction zone,” which in the current default environment is an invisible sphere about one foot to the right of the user's head. Moving the wand into the zone, pushing, briefly holding, and then releasing a button will release the tool mapped to that button, such that the next press of that button outside the zone will display the tool selection menu again.

{image:worksite:/Images/CAVEWandDiagram.png|CAVEWandDiagram.png}

Figure 1: Diagram of the functions mapped to the CAVE wand's buttons and joystick in the default configuration. The “hotspot” is the point that applications think the wand is pointing at. It is also denoted by a grey cone displayed in the CAVE; normally, the grey cone's tip should appear exactly at the position of the small nose on the front of the wand (maybe a couple of millimeters off).

H3 Interplay between Tools and Application Functions

Some more complex applications have multiple different functions that can be mapped to locator or dragging tools. For example, a locator tool could be used to create slices, contour surfaces, or streamlines. Typically, applications handle this by offering a setting in their main menu which will influence how future tools are created. For example, a visualization application might offer creating slices and streamlines. When a user creates a locator tool, that tool takes the current function selected in the application. If the selected function is changed afterwards, the previously created tool typically does not change functions, but newly created tools will. (Some applications offer a menu entry to override the functions of existing tools.) This approach allows a user to create different tools performing different functions and map them to different buttons, to be able to use them at the same time. The typical process of mapping an application function to a wand button is thus:

# Select the desired application function. # Map a tool of the class matching the function to a free button. # When a different tool is needed, remove the tool by pressing its button inside the “release zone” and repeat from step 1.

H2 Shutting Down a VR Application

* The applications can be shut down from the control workstation, by either clicking on the “X” icon in the upper right-hand corner of the application's control window, or by moving the mouse cursor into the control window (the mouse cursor disappears when entering the window) and pressing the “Esc” key. * Right afterwards, the control window should disappear, and the CAVE screens should revert to the default image.

H2 Stopping the CAVE

# Turn off tracking devices.\\When not in use, the CAVE wand and the head tracker/stereo glasses should carefully be put on the workstation table. # Turn off all projectors.\\Double-click on the “Projectors OFF” icon. The projectors will turn off immediately. # Log off from control workstation.\\To log off, click on the red hat icon in the lower right-hand corner of the control workstation's screen, and select “Log out” from the popup menu. In the confirmation dialog box, highlight the “Log out” radio button, and click on the “OK” button. # Turn off control workstation screen.

H2 {anchor:troubleshooting}{anchor}Troubleshooting

This section lists some common problems when using the CAVE, and some steps to hopefully solve them.

H3 CAVE Applications Do Not Start

If a CAVE application does not start up when double-clicking its icon in the “CAVE Demo Programs” folder, this can be for a variety of reasons. To pinpoint the actual problem, one should go through the following diagnostic steps:

# Ensure that all cluster nodes are running.\\To show the current state of the cluster nodes, double-click on the “Cluster Monitor” icon on the desktop. This will open a window with CPU usage gauges for all cluster nodes (caveman with two CPUs, r1-r5 with one CPU each). Normally, all bars should be green (denoting idle CPUs), with an occasional flash of red or blue. Once the status of the cluster is affirmed, the monitor window can be closed again.\\If the cluster monitor window does not open, this means that at least one cluster node is not responding. At this point, it is best to call for help. # Open a terminal window and change to the CAVE applications directory.\\There should already be at least one terminal window on the desktop; otherwise, one can be opened by clicking on the terminal icon in the panel bar at the bottom of the desktop (third icon from the left). To type into the terminal window, the mouse cursor needs to be moved inside (typing works if the terminal window's title bar is dark blue).\\To change into the CAVE applications directory, type '~~cd ~/Demos~~'. The terminal prompt will now read '~~caveman:~/Demos~~'. # Check that no zombie processes are running.\\Sometimes when a CAVE application crashes, its child processes on the cluster nodes do not shut down. While these “zombie processes” are running, no other CAVE application can start. To check for zombie processes, open a terminal and type '~~ListAllNodes~~'. This script will print the names of all processes running on the cluster nodes (nodes r1-r5). If any zombie processes show up, they can be terminated by typing '~~KillAllNodes <process name>~~, where '~~<process name>~~' is the name of the zombie process reported by '~~ListAllNodes~~'.\\One caveat is that '~~ListAllNodes~~' only lists the first fifteen characters of the application name, but '~~KillAllNodes~~' needs the full name. It is easiest to type '~~KillAllNodes~~' followed by the (abbreviated) name reported by '~~ListAllNodes~~, and then hit the Tab key. The terminal will then automatically complete the application name, or print a list of alternatives if the prefix is not unique. One can then type further characters and hit Tab again to complete.\\Another caveat is that '~~ListAllNodes~~' only lists processes owned by the user running the '~~ListAllNodes~~' script. If '~~ListAllNodes~~' shows a clean cluster, but VR applications still do not work, a previous user might accidentally have left some zombies behind (this is why it is important to check for zombies before logging out). # Ensure that the input device driver is running.\\The driver for the input device hardware is a stand-alone process that is automatically started at boot up and should never terminate. Unfortunately, it sometimes does. To check whether the device driver is running, type '~~CheckDeviceDriver~~'. If the script prints anything but '~~VRDeviceDaemon (pid <some number>) is running…~~' the device driver terminated and needs to be restarted. To restart the device driver, type '~~sudo RestartDeviceDriver~~'. This will prompt for the user's password, and then attempt to shut down the device driver (which will fail), followed by starting it again. Afterwards, the driver's status can be checked again by typing '~~CheckDeviceDriver~~, and its log file can be inspected by typing '~~cat /var/log/VRDeviceDaemon.log~~'. The latter should print text that should end with a line stating '~~VRDeviceServer: Waiting for client connection~~'.\\If either of these two checks fails after the device driver has been restarted, there is something wrong with the input device hardware. At this point, it is best to call for help. # Ensure that the device driver is working properly.\\Even when the input device driver runs, it might still not react to connections. To run a test program, type '~~TestInputDevices~~'. If the input devices are functioning, this should print the (x, y, z) position of the wand device in inches relative to a spot in the center of the CAVE floor. If the wand is turned on, the numbers will be updated several times per second. To finish the test program, press '~~Enter~~'.\\If this test fails, and restarting the device driver does not help, there is something wrong with the input device hardware. At this point, it is best to call for help. # Ensure that all video cards are working properly.\\To check if the video cards in all cluster nodes are working, start the display control program by double-clicking on the “Video Card Control” icon on the desktop. Note: Improper use of the display control program can cause serious trouble. Please be careful to not make any accidental changes to any settings. If the program does not start, the cluster nodes' X servers might be down. At this point, it is best to call for help. Otherwise, the display control program should be closed again (it affects performance of CAVE applications if left running).

H3 A CAVE Application Has Crashed Or Is Frozen

Since most CAVE applications are prototypes and still under development, any of them might crash at any given time. Due to the CAVE computer system's cluster nature, the operating system can not always automatically clean up after a misbehaving application. Crashes come in three flavors:

# “Hangs” or “Locks”: The application seems to “freeze” and does not react to input anymore. The application's console window is still open, and the CAVE walls do not show the background image. # Console crashes: The application's console window is closed, but all CAVE walls still do not show the background image. # Wall crashes: One or more of the CAVE walls show the background image, but the application's console window is still open.

In all cases, the first thing to do is to remove all remnants of the misbehaving application. Each application runs as a separate process on the console computer and the render nodes, and if only some of those processes crash, the others are stuck, but still running. The following steps will remove all “hung” application processes:

# Check if the console computer is reacting to mouse movements and/or key presses. If the console computer appears frozen, something is seriously wrong, and it is best to call for help. # Open a terminal window. # Type '~~ListAllNodes~~'. This shows all relevant processes owned by the current user, running on the console computer (caveman) and the render nodes (render-1 to render-5). The process listing for caveman also includes any applications running on the desktop (such as web browsers), whereas the listings for the render nodes usually only contain VR application processes. # Identify the name of the stuck application. This is normally easy, since the render nodes can at most run a single VR application at a time. In some cases, one has to guess from the listed process name. # Type '~~KillAllNodes <full application name>~~'. This will kill all processes of the given name on all CAVE computers. The command prints a list of messages as it goes through the computers; any messages can normally be ignored. Note: The KillAllNodes command requires the full application name, whereas ListAllNodes unfortunately only lists the first 16 or so characters of the name. The easiest way to get the full name is to type '~~KillAllNodes~~' and the prefix of the application name reported by ListAllNodes, and then press '~~Tab~~'. Tab autocompletes a command name, and since all CAVE applications are in each user's path, this should work in all cases. # Check if the application's console window is gone, and if the CAVE walls show the background image again. If this is not the case, something is seriously wrong, and it is best to call for help. # Check if the VR device driver is still running, as described in the previous section. Sometimes a crashing application takes the device driver with it, sometimes a locked device driver causes an application to appear frozen. Try the TestInputDevices command first, and then the other steps if it fails.

H3 Stereo Is Not Synchronized Between CAVE Screens

If CAVE applications are working, but the display flickers badly when looking at a corner between two or three screens, or there is no depth perception at all, the CAVE screens might have lost synchronization. To run the display control program, double-click on the “Video Card Control” icon on the desktop.

Note: Improper use of the display control program can cause serious trouble. Please be careful to not make any accidental changes to any settings.

In the display control program's window, select the “FrameLock” section in the list on the left. This will change the right side of the window to show a list of all render nodes (r1-r4). If the list does not show up completely, enlarge the program's window vertically by pulling on the lower window border with the mouse. In the list of render nodes, all lines should show green bullets in the “Stereo Sync” and “Sync Ready” columns, and all but the master render node (r2) should show green bullets in the “Timing” column. If this is not the case, or the button on the lower right of the window reads “Enable FrameLock,” restart synchronization by clicking on the button on the lower right of the window once or twice, until it reads “Disable FrameLock”. After this, the bullets in the list will change color, and after a few seconds all of them should be green again. At this point, the display control program should be closed again (it affects performance of CAVE applications if left running).

keckcaves/caves_trouble_shooting_guide.txt · Last modified: 2008/07/17 21:26 (external edit)