User Tools

Site Tools


keckcaves:troubleshooting

Troubleshooting

This section lists some common problems when using the CAVE, and some steps to hopefully solve them.

In general, please don't restart the CAVE computers. It will mostly likely cause more problems than it will solve. 1)

Useful Commands

Commands have changed in the year 2011. Here are some useful one:

  • cave-psvrui: Show all Vrui processes running on all nodes.
  • cave-killvrui: Kill all Vrui processes running on all nodes. Use this if you are having a hard time starting Vrui applications, and cave-psvrui is showing another (probably dead) Vrui application still running.
  • cave-framelock: This will reset the framelock between all the render nodes. Try this if stereo doesn't seem to be working.

Problems and Fixes

Not All Projectors Turn On

See caves status to make sure that there isn't a known problem.

Right Wall Shows Unexpected Colors: see caves status

CAVE Applications Do Not Start

Application run via icons in the Demo directory should output any error messages on a console that pops up after they fail. Applications run from the command line should output messages to the command line window from which they were run.

Here are some error messages to look for, and how to respond to them.

Zombie Vrui

Error message:

MulticastPipeMultiplexer: Unable to bind socket to port number <some number> while initializing cluster communication

Cause: Another application is already running, or remnants from another application are still blocking some of the CAVE cluster nodes.

Solution:

  1. Open a terminal window.
  2. Type cave-killvrui
  3. You will get a list of all running Vrui processes on all nodes, then they should be terminated, and then you'll get a list all running Vrui processes again, which should be none.
  4. If the same problem persists, email keckcaves_help@ucdavis.edu

VR device daemon

Error message:

Ignoring input device adapter CAVEAdapter due to exception TCPSocket: Unable to connect to host caveman on port 8555

Cause: The VR device daemon, which receives tracking data from the tracking system, is not running.

This really shouldn't happen anymore, so if you find this you should e-mail KeckCAVES help via keckcaves_help@ucdavis.edu. Yet here you are, so give the following a try:

  1. Open a terminal window.
  2. Type sudo service VRDeviceDaemon stop
  3. Type cave-killVRdevicedaemon to end any lingering process.
  4. Type sudo service VRDeviceDaemon start
  5. Wait for about 15 seconds.
  6. Try starting your application again.
  7. If you notice navigation issues or see the green warning grid lit, turn on and bring all the tracked devices and sensors (wand, head tracker, etc.. (a total of four)) into the Cave and wait for a few seconds. Return the wireless wand to its charger turned on.
  8. If it still doesn't work, the InterSense tracking system may not be communicating with the head node. Now it is really time to email keckcaves_help@ucdavis.edu.

InterSense tracking system

Error message:

terminate called after throwing an instance of 'Comm::TCPSocket::PipeError'
what(): TCPSocket: Connection terminated by peer during read

Cause: The InterSense tracking system is no longer communicating with the head node.

Solution: The InterSense tracking system needs to be restarted. Unfortunately, this requires stepping behind the CAVE, and it would be preferable if people didn't make a habit of this.

  1. Carefully step behind the CAVE, where the stack of components are.
  2. At the top is the InterSense. It has a power button on the left. Press it. The InterSense should turn off.
  3. Wait ideally up to 30 seconds.
  4. Press that button again. The InterSense should beep and starting booting up.
  5. Once it is booted, you may need to restart the VR device daemon. Refer to the above.

Problems running against your own Vrui installation

You're doing what now? You may be kinda on your own here, pal.

However, if you get an error with the InputDeviceManager, it's probably a configuration file problem. Please see Configuration Files in the CAVES

Stereo in the CAVE does not work

Symptoms include:

  • Glasses flicker
  • Glasses flicker when looking at an edge or corner between two or three screens
  • Large amounts of “ghosting”, i.e., double images

Here are some possible causes and solutions:

Glasses out of power?

Your glasses batteries may be low on power.

Please try another pair of glasses. If that works, let keckcaves_help@ucdavis.edu know and set the glasses aside.

No frame lock?

Frame lock may have been lost between the render nodes. This can happen if the render nodes have been restarted, e.g. due to a power outage.

To try fixing this:

  1. Open a terminal window.
  2. Type cave-framelock
  3. You'll get lots of output, but hopefully no errors.
  4. If you got errors, you may need to do the frame lock manually. Instructions for this are below.

Ubuntu Screen Doesn't Show Log Off Menu

Sometimes the monitor graphics get jumbled and the log-off menu doesn't show up. To log off type this command in a terminal:

killall -9 gnome-session

This will log you off and show the main login screen.

Something else?

On any other error messages that do not come from the application itself (missing input data, wrong data formats, etc.), please email keckcaves_help@ucdavis.edu.

General advice

A CAVE Application Has Crashed Or Is Frozen

Since most CAVE applications are prototypes and still under development, any of them might crash at any given time. Due to the CAVE computer system's cluster nature, the operating system can not always automatically clean up after a misbehaving application. Crashes come in three flavors:

  1. “Hangs” or “Locks”: The application seems to “freeze”; and does not react to input anymore. The application's console window is still open, and the CAVE walls do not show the background image.
  2. Console crashes: The application's console window is closed, but all CAVE walls still do not show the background image.
  3. Wall crashes: One or more of the CAVE walls show the background image, but the application's console window is still open.

In all cases, the first thing to do is to remove all remnants of the misbehaving application. Each application runs as a separate process on the console computer and the render nodes, and if only some of those processes crash, the others are stuck, but still running. The following steps will remove all “hung” application processes:

  1. Check if the console computer is reacting to mouse movements and/or key presses. If the console computer appears frozen, something is seriously wrong, and it is best to call for help.
  2. Open a terminal window.
  3. Type cave-psvrui. This shows all VR application processes, running on the console computer (caveman) and the render nodes (render-1 to render-5).
  4. Type cave-killvrui. This will kill all VR application processes on all CAVE computers.
  5. Check if the application's console window is gone, and if the CAVE walls show the background image again. If this is not the case, something is seriously wrong, and it is best to call for help.
  6. Check if the VR device driver is still running, as described in the previous section. Sometimes a crashing application takes the device driver with it, sometimes a locked device driver causes an application to appear frozen. Try the DeviceTest localhost:8555 command first, and then the other steps if it fails.

Warped or Double Images

If the image on the screen appears “warped” or does not change when a user moves her/his head, the head tracker might not be able to communicate with the tracking rails (above the CAVE, with the blue “blinkenlights”). The head tracker has two tiny microphones on the top front corners. The tracker will not work properly if the microphones are obstructed, for example by a visor. (Although not recommended as a fashion statement, it is advisable to wear baseball caps backwards, or not at all.)

If the image on the screen appears “double” and there is no depth perception, the stereo glasses might not be able to communicate with the projectors. The stereo glasses are controlled via an infrared signal; the infrared receiver is located between the eye glasses. The line of sight between the receiver and the projectors (behind the screens) must not be obstructed (headgear advisories apply as above). Sometimes the glasses do not turn on properly; in that case, taking them off, folding, and unfolding them again usually fixes the problem.

Manual reset of CAVE frame lock

In case you don't trust running the automatic cave-framelock, or it doesn't seem to be working for you:

  1. Run cave-framelock --gui from the command line.
  2. Select “Frame Lock” in the list box on the left side of the dialog that appears.
  3. Add entries for r2:0.0, r1:0.0, r3:0.0, and r4:0.0 via the Add Devices… button.
  4. The list entry for r2:0 should have three black bullets on the left, and a check mark in the “Server” box. The other list entries should have three green bullets on the left, and check marks in the “Client” boxes. The “House” bullet should be red on all four entries.
  5. If there are any red bullets on the left side of any of the entries, wait a few seconds – they should turn green after a while. If they don't, press the “Disable Frame Lock” button in the lower-right hand corner, wait a few seconds, and press the button again (it's now labeled “Enable Frame Lock”). Then wait another few seconds until all lights turn green. If they don't, please email keckcaves_help@ucdavis.edu
  6. Close the dialog box.

Restarting After a Building Power Outage

For experts only!

  1. Check if tracking unit is running. If not, power on and wait until fully booted.
  2. Check if head node survived on UPS power; if not, power on and wait until fully booted.
  3. If head node survived, restart the VR device driver by running sudo service VRDeviceDaemon restart in terminal.
  4. Check if all render nodes r1-r5 are running. Power on any non-running nodes and wait until cave-psvrui lists (no) processes on all five nodes.
  5. Restart frame locking on render nodes r1-r4, either via cave-framelock or manually as described above.
  6. Test the CAVE:
    1. Turn projectors on via “CAVE On” icon.
    2. Run Earthquake Viewer in training mode.
    3. Turn on head tracker and wand.
    4. Take devices to the CAVE, ensure that head and wand are tracked properly, and that the wand's indicator cone follows the wand device closely.
    5. Test stereo synchronization by looking at all CAVE screens in turn, and checking for loss of stereo, stereo inversion, or any flicker (especially when looking from one screen to another).

Finding the command line of a Demo icon

  1. Right-click on the application icon in the CAVE Demo Programs folder and select “Properties” from the pop-up menu.
  2. In the dialog box that pops up, go to the “Basic” tab (should be visible by default), and highlight the entire contents of the “Command” text field (triple-click in the text field, or right-click on the text field and select “Select all” from the pop-up menu).
  3. Open a terminal window, by going to the “Applications” menu in the top-left corner of the desktop, then selecting the “Accessories” submenu, and finally the “Terminal” menu entry.
  4. Middle-click in the terminal window. This should paste the entire contents of the “Command” text field into the terminal window. If that somehow does not work, follow these alternative steps:
    1. Right-click on “Command” text field and select “Select all”.
    2. Right-click on “Command” text field and select “Copy”.
    3. Right-click on terminal window and select “Paste”.
  5. Press “Enter” in the terminal window. This will attempt to start the application, just as double-clicking on the icon would, but print diagnostic and/or error messages.
1) Yes, ideally everything would come up cleanly after a reboot. This is one of Braden's many goals.
keckcaves/troubleshooting.txt · Last modified: 2015/04/24 14:49 by yikilmaz