How to run the system standalone

From CSL Wiki

These directions assume that you have the nodes already flashed and ready to run the software. If not, see the instructions for flashing the nodes and loading the software. The current release tag for the emstar software is XXXX.

Contents 1 Boot Flags 2 Things to take to the field 3 Configuring your laptop 4 Using rbsh to control the nodes 5 Physically deploying the nodes 6 Arranging the nodes in deployment 7 Debugging a deployment 8 Running the System 9 Inside the Box

Boot Flags

The startup script is set up to boot into different modes depending on the existence of special files in /home/emstar. These flags include:

• RANGEEXP enables the localization application to run, sets sample rate to 48KHz
• RECORDEXP enables only the recording and time synchronization system, and sets the sample rate to 12KHz.
• NFSBOOT runs the localization system via NFS (assuming you have a properly set up server).
• NOBOOT starts up no additional software, so that you can manually start something using rbsh (single hop or via wired... flooding will not be running).

This page describes how to use the system in RANGEEXP mode. It's easiest to set the flag when they are connected up via wired ethernet (because rbsh can reach all nodes) or when they are all one hop away. After setting the flag, issue a reboot command to boot it into the correct mode.

For example, to switch from NOBOOT to RANGEEXP using rbsh via wired ethernet:

 $ rbsh -b eth0
 > rm /home/emstar/NOBOOT
 > touch /home/emstar/RANGEEXP
 > sleep 1
 > reboot
 >

If you have more than one boot flag set, one of them will be used, perhaps not the one you intended. To find out what boot flag is currently set:

$ rbsh -b eth0
> ls /home/emstar/[[:upper:]]*
>

Things to take to the field

• Laptop running linux, with emstar checked out and compiled. Be sure to include the libraries required to build emview and rbsh. (Run make config and check to see what messages appear with "**" in them.. these are warnings about stuff that won't build.)

• Rangefinder

• Laptop battery or power source

• (optional) Large 802.11 antenna.

• Serial cable and x-over ethernet cable for on-the-spot debugging

Required for each node:

• 4 AA batteries in a pack with '9V' connector. The open circuit voltage should be at least 5.3 volts, although they may work down to 4.5 or lower in a pinch.

• 12V Gel Cell. The nodes run for approximately 24 hours on a single charge.

• Microphone array and pair of cat-5 wires. Longer wires may be substituted if needed.

• Tripod or stake to hold the array. If using stakes, also include a pvc pipe and large hose clamp.

Configuring your laptop

In addition to the nodes, you will want a linux laptop running, with wireless configured to ESSID woodpckr and ip address 192.168.11.13/24. Be sure to check out emstar onto your laptop, along with all of the required packages to build this software (be sure to include libreadline5-dev). Then, run the following software on your laptop:

# runs some code to facilitate visualization of the network
# you may need to edit this file to set the interface eth1 to your wireless interface
emrun/emrun ../devel/loc/testtabs/ar-sink-floodonly.run &

# run the visualizer.  this should provide a picture of link connectivity.
# if you have an approximate laydown in mind, feeding that topology in with 
# -l <file> will enhance the readability of the graph.
emview -a localhost &

# open a browser, point to the node of your choice: http://192.168.11.XXX/cgi-bin/aensbox.cgi
firefox &

# open a shell for telnet debugging
telnet 192.168.11.XXX

# open a shell for rbsh operation
emproxy/rbsh -U flood

If you are connecting via wired, configure your wired interface as 192.168.10.13/24. The nodes are have static IP address, with 192.168.10.XXX being the wired address, and 192.168.11.XXX being the wireless address, where XXX is the node's ID (printed on the box).

If you are already using the wired interface on some other network, you can create a virtual interface on the 192.168 network (using linux):

 # if your current interface is called eth0
 ifconfig eth0:1 192.168.10.13 netmask 255.255.255.0

This will create a new interface called eth0:1 that is on the nodes' network, without interfering with your existing network. With wireless it is not so simple because you have to configure the wireless card to use the "woodpckr" ESSID. However, using this setup has the security advantage that the nodes can't be reached from outside your LAN.

Using rbsh to control the nodes

rbsh is a useful tool that provides a broadcast remote shell. That is, you can use it to execute a set of commands on all nodes at once and return results back to you. Since rbsh is udp-based, it is unreliable, although it sends each command 3 times for redundancy (only executing each command once). Replies are limited to 1K of data returned, and are not reliable.

You run rbsh like this

 # to connect via UDP via a network interface
 rbsh -b <ethernet_interface>
 # to connect via a link device
 rbsh -U <link_device>

Note that above we recommend rbsh -U flood to send messages via the flood adaptor so that they do multiple hops. When you run rbsh, it will create an interactive command line with the usual line editing features. You can enter multiple command lines, and when you press return twice (to create a blank line) the previous commands are sent out to the nodes in a batch, and the responses are then summarized as the return.

The rbsh command line reports in backets the number of nodes that have responded to the most recent command. There are also a few helpful special commands that rbsh itself interpreted, which you can see by typing "help":

$rbsh

Welcome to the rbsh remote broadcast interactive shell.
Enter shell commands, followed by a blank line to send.

rbsh 1> help
Prompt format: [X] rbsh Y>, where
  X is the number of nodes that replied to the last request
  Y is the sequence number of the next request
Local commands:
  help:    prints this message
  abort:   discards current command
  check:   prints current command
  delta:   prints out any missing/added nodes since last command
  sorted:  prints out sorted list of node IDs
  exit:    exits
rbsh 1>

Physically deploying the nodes

Plant a stake or tripod for each node. To connnect up the node:

• Place the array on the pvc pipe attached to the tripod or stake.

• Attempt to position the box and antenna away from the ground. Balance the box on end with antenna up to maximize antenna height, or place the box on higher ground. If you are using green stakes, you may be able to hang the box from the "bumps" on the stake, as shown in this picture: http://www.lecs.cs.ucla.edu/~girod/aensbox/

• Connect the CPU box to the array using the pair of cat-5 wires. These wires plug into the ports inside the holes drilled near the bottom of the box. Longer wires may be substituted if needed. The red cat5 cable is the microphone cable. It connects to the red marked port on the CPU box, and the port on the array with the red dot sticker. Do not connect the array to the black RJ45 port that sticks out of the box.. this is the ethernet port.

• Connect the AA battery pack to the battery connector on the array.

• Using the external 12V Gel Cell: Set the power switch to "off" (middle contact, neither off nor on), and connect the 12V Gel Cell to the large red and black power cable. Then, set the power switch to "adaptor" to turn on the node. The nodes run for approximately 24 hours on a single charge. Use the Battery Tenders 4-port charging stations to recharge the Gel Cells. A full charge takes about 8-12 hours.

• Using the internal Li+ Battery: They can also run for 4 or more hours on the internal battery, if it is charged and not damaged. Set the power switch to "battery" and set charging to "off" to turn on the battery. At the present time, some of the nodes have missing or damaged internal batteries. We have spares which can be used to replace damaged batteries in the spare parts collection. To recharge the internal batteries, plug the 16V Li+ charger into the port that it fits on the side of the box, and turn the charger switch to "on". Be sure the Li+ charger is set to 16.8 volts.

• Using a 12 volt adaptor. The 12 volt adaptor may also be plugged into a port on the side of the box. Be sure that the external 12V Gel Cell is disconnected -- these are internally wired to the same place!!

Arranging the nodes in deployment

The system should be fairly resilient and has very long range, so the requirements on deployment are not too stringent. However, there are a few rules of thumb:

• The signals do not penetrate solid ground. If you have a solid ground or hillside, try to position a few nodes surrounding the hill that more or less have line of sight.

• Synchronization places topology requirements that all nodes be part of a triangle. In general it is safer to have more connectivity than less.

• The more line of sight the better; however obstructing foliage is not generally a major problem.

Debugging a deployment

After deploying the nodes physically, turn them on if they are not already on. You will now want to check the deployment to make sure it's all working properly. The emview laptop can be used to assess connectivity and also to check for faults. Cat "/dev/mhsync/data/faults-44-2" on the laptop to see any "faults" reported by the nodes. Two kinds of faults are reported:

• VXPX channel X dead. This means that something is wrong with that node's microphone array. Possible causes:
  • Check batteries and voltage. Broken battery wires are potentially very difficult to fix in the field, as the array is quite difficult to disassemble.
  • Check that the CPU-array cat-5 wires are securely inserted.
  • If a single channel is reporting dead, it may be that that mic's wiring is loose. Try reinserting the RJ45 connector for that channel (note, the numbering on the circuitboard is 1-based while the faults numbering is 0-based.)
• SelfRange failure. This has a variety of causes, and sometimes can be a temporary glitch.
  • Did you notice whether that node chirped or failed to chirp? If so, this could be caused by loose wiring in the array for the speakers. The speakers are all wired in parallel with a sort of rats-nest of wire, back to a red and black wire that clips in with a white molex connector.
  • If the node seems to have chirped, then the problem may be transient. If it is persistent this is a strange issue.. try rebooting that node. If that doesn't work then it could be some kind of more complex hardware or software issue.

You can also check the device "/dev/mhsync/data/range-20-9" (is that correct?); each node should appear once in this list.

In addition to faults, the emview display may show different sorts of problems.

• Missing nodes. This can be caused by poor topology, hardware problems, broken nodes, and possibly by software problems.
  • If "rbsh" can't reach them, the best debugging strategy is to walk out with a laptop and see if you can ping them over wireless.
  • If you can ping it, telnet in and see if the software is running. Debugging beyond that is difficult.
  • If the node is not responsive, try rebooting it; unfortunately the serial console is on the inside of the box and difficult to access. You might also try connectiving via the ethernet crossover.
• Toplogy issues. If you are not seeing great topology, the easiest fix is to raise up the nodes higher if possible. You can also add better antennas. You can also add more nodes (even ones that do not have microphones) as these will still aid in synchronization and network connectivity.

Running the System

Once the deployment is complete, you should be able to run the system. Start by picking a node that is near to your laptop, to act as the "master". Point your web browser at that node, to cgi-bin/aensbox.cgi. This will be your means of controlling the system.

You may also want to telnet to that node, in case you want to take a look at logs or other local information.

The first web page you see will report any faults that the nodes may have (leave this page open, and check it after you click the localize button). Click the 'Automatic Localization' link to open the control page in a new browser window. Once you have the browser up, select the button marked "Enable Master Mode". This will refresh the page with a more complicated form that you can use to operate the system.

First, you should wait a bit (at least 5 minutes) after firing everything up to make sure that the nodes have achieved synchronization. You can get an idea of this by issuing the following command via rbsh:

 cat /dev/sync/params/status | wc

This will give you an idea of the number of sync relations present for each node.

Once it appears that you have enough data (for 10 well connected nodes,about 500, for 5 well connected nodes about 200-300, for 10 not so well connected nodes, about 300) you can try issuing a ranging command, by clicking on the "refine" button on the website. This should trigger all of the nodes to chirp. By counting the chirps you can tell whether some chirps were missing. Missing chirps can be caused by packet loss or by lack of synchronization.

After clicking refine, wait for the data to be processed and to to arrive back. It will take some time (approx. 20 seconds * the number of nodes) for the data to arrive back. When it does, the web page should display the number of ranges reported by each node, which should indicate how many nodes heard each other. Run through a few refines to improve your coverage (3-5 trials).

Then, try clicking "Start Computation"; this will begin running the multilateration on the collected data. This can take a while. If you want to watch the progress, you can telnet to the "master node" and cat /dev/emlog/ml/all-f. This will spew out a bunch of log messages.

After computation the web page will update with a lot of additional information, including a coordinate system, a list of residuals, some information about errors, etc. This information is somewhat technical but one thing that might be worth looking at is the residuals output. This tells you whether any of the data is particularly inconsistent with the rest of the system. If you are seeing very large residuals that is a sign that you may want to enable data pruning. The second column of the residuals table lists the "studentized" residual value; you can set a threshold which will drop any data that has a studentized residual over a certain number. In my experience, 3 has been a good choice for that parameter. However, this may depend on conditions. If you set that number, you must click the compute button to try again and drop that data.

If you are running the system in a data-saving mode (which is currently the default), it will be recording lots of data to the sd-card in the /mnt/sd_card/data directory. This data can be used offline to refine the results or debug the algorithms. If you want to save this data, be sure to move the data directory after you do a set of refines, and before you move the nodes to new positions. Otherwise it will be difficult to analyse the data later.

You can do that using rbsh:

 # move the data
 cd /mnt/sd_card
 mv data experiment1
 mkdir data

You can later retrieve the data via NFS.

Missing features that would help a lot

These are not terribly difficult changes to make.

• Fix the problem with cpd that needs the "hack" patch right now.
• Raise a fault if a chirp did not fire because of timesync.
• Raise a fault if at some node, no sync exists to any other node.
• display faults in the main aensbox page.

Inside the Box

The CPU box can be opened by unscrewing the brass thumbscrews and carefully lifting the aluminum plate. Use care when opening it as the cpu card with its cables does not quite fit past the plastic edges of the box. With the hinge away from you, you typically want to first lift the back edge of the aluminum up, and push the plate back, then lift up the front edge up. This is because the ethernet connector and power connector for the CPU board stick out a little bit and bump into the front edge of the plastic box. Then, swivel up the aluminum panel and rest it against the hinged top of the box.

Inside the box you will find various wiring and boards. All cabling should be stuck down with adhesive cable pads. If anything is loose inside the box, try to tie it down; otherwise it will rattle around and potentially break.

The CPU board

• Two slots contain: the SMC ethernet card and the VX Pocket sampling board.
• The antenna connector should connect to the SMC card above the C in "SMC".
• The dongle with the microphone connectors should connect to the VX Pocket board.
• There is an SD-card inserted in a socket underneath the PCMCIA cards; you may need to remove the microphone dongle to extract it. Only cards up to 1GB are supported.
• The console serial port plugs in on the underside of the CPU board and has two DB9 connectors wired to it. Plug in using a standard serial cable, (no null modem), to the female DB9 coming off the CPU board. Use 115200N1 modem settings, no flow control.
• The power connector for the CPU board should have 9-18 V on it. It is wired directly to the power switches, and through those switches either to the adaptor plug / 12V battery terminals, or to the internal battery.

Connectors on the Box

• The external power connectors are:
  • TOP: (4 prongs) AC adaptor, 12V
  • BOT: (3 prongs) Charger (16.8V)
• The RJ45 receptacle that sticks out of the box is ethernet
• The two RJ45 connectors in holes near the bottom of the box are the connectors to the array. The red marked one is the microphone connector, and that matches the red dot connector on the array.
• The red and black cord with quick-connect lugs on the end is intended to hook up to a 12V gel cell. Note that this is wired in parallel with the adaptor port: do not connect both at the same time!

Audio volume and output control

• You can change the volume of the chirps. Inside the CPU box, there is an autio amp board with a potentiometer, which you can adjust. You can also use alsamixer to adjust the output volume.
• The audio amplifier is gated by a software-controlled switch. This switch is in a small heat-shrinked board that is ziptied onto the power plug for the audio amplifier board. Make sure that the fine bluewires are indeed connected to the CPU board; they should connect to the hole marked '19' and the ground point.
• You can set this switch using /dev/gpio/19:

 # to turn off the amplifier
 echo 1 > /dev/gpio/19  
 # to turn on the amplifier
 echo 0 > /dev/gpio/19

• You can test the audio using the /home/emstar/devel/loc/soundtest program. This will emit a long continuous chirp. You can calibrate the amplifiers by running this program and adjusting the volume to achieve a particular sound level using a sound pressure meter a fixed distance away from the speaker.