Feedback design

One positive aspect of having run the tests in Kalamazoo last week is that it permitted us to begin to see which kinds of feedback are going to be relatively easy to acheive (once we are able to take measurements during travel as expected) and which ones may be very difficult to do. In this posting, I am going to set down some thoughts about feedback, partially based on what we observed last week, and also based on what we have been talking about as a group. I am hoping that this will stimulate a discussion among group members as we enter this next crucial phase in the project.

I will start by listing the kinds of events that will or could occur during a wiiCane session (there may be others that I have not considered):

1. Tap-this is the result of striking the cane's tip on the groud at point of maximum x displacement. this is a necessary component of two-point touch (and our system may not work if the user does not execute a tap at the completion of each swing). Tap events can also occur in mid-swing and are highly recognizeable amid potentially noisy data. In the test in Kalamazoo, we were able to synchronize the Optotrak and WiiCane data streams temporally by having the user produce three rapid taps in succession when he had finished walking. Taps are detected using accelerometer values exclusively, cane orientation is not relevant to tap detection. Due to system latency, I don't think that we can issue any feedback on taps (because the user would probably notice a lag between tapping the cane and perceiving the feedback). Also, I don't think it serves an informational purpose, since the user knows he has tapped the cane (assuming we can filter out non-tap acceleration events). But detecting and recording tap events is a crucial first step in being able to interpret other aspects of the user's technique.

2. Mid-swing. This is probably defined as the point where the cane's azimuth is co-linear with the travel path, although it could also be thought of as the point at which the cane's azimuth is co-linear with the midpoint of a line connecting the right and left tap points of a single swing (when the traveler is swinging the cane but not walking). In some cases, it may be desireable to provide vibration feedback as the cane passes the swing midpoint. This form of feedback would provide an easy-to-follow tactile path; while this kind of guidance is probably not useful in mobility training (because the real world is not equipped with this feature), it may be useful for preliminary exercises (like training wheels on a bike). In practice, creating a vibratory feedback that coincides with the point of mid-swing may be difficult (due to system latency: see discussion of Tap events, above), although there may be ways to predict that the cane is about to pass the arc midpoint, and send the feedback slightly before the event occurs.

3. Arc width. This is defined as the either the maximum angular displacement of the cane as it moves through a single sweep, or as the perpendicular displacement of the cane (from the walking path) at each tap event. Accurate measurement of arc width is essential to the system as envisioned in the proposal. In our discussions so far, there appears to be some concensus that feedback on cane arc should be provided intermittently and not after every tap. In a preliminary version, we provided verbal feedback consisting of the phrases, "too narrow", "too wide" or "good". Probably our administration utility needs to have a means of adjusting the frequency of this kind of advice. My guess is that, for adult users, spoken feedback should be provided remedially, when arc width is observed to deviate from the ideal width by some software-configurable percentage. For kids, it might be better to use a positive-sounding earcon to signal that arc width is correct, and nothing when it is incorrect. I can imagine an exercise where the child is simply asked to walk or stand still and swing the cane to produce the praise sound.

4. Hand at midpoint of body. It will be challenging to detect when the cane hand is not held at midpoint of the body. We are considering a variety of methods for tracking this; it may be, however, that the best we can do is to ask them to start at the beginning of the course with their hand held at midpoint, and we should be able to watch them execute a few stationary (non-walking) swings to make sure that we recognize the gesteral pattern associated with type of movement.

5. Wrist isolation. Similarly, it will be hard to check for wrist isolation during travel. Classic two-point touch technique calls for wrist flexion and forearm immobilization while swinging the cane. We will probably be able to detect good form (and provide confirming or admonitory feedback) during stationary swings, but when motion associated with walking is added, it may not be feasible to track wrist position using a relatively simple set up of lights and cameras like the one we are considering here.

6. Wrist rotation. One big problem we ran into during the verification exercise at Western was that we were not able to correct for wrist rotation. When the user rotated his wrist, all of the lights we were tracking moved significantly, even for a very minor rotation. While this can be compensated for in our measurement algorithms, the problem becomes much more difficult. It is not clear (to me) whether classic two-point touch teaches the traveler to avoid rotating the wrist of the cane hand. However, we need to find a sensing method that is not affected by this kind of movement, or we have to train the user to avoid wrist rotation, something that might be very hard to do with kids. We may find that this problem disappears when we switch to a different arrangement of lights. This issues is an imporant one, and I think that the accuracy of our system might be negatively affected by failure to account for this kind of motion.

8. Cane/step synchronization. While it was not one of our original measurement and feedback objectives, it would be very helpful to know whether the user's cane swings are properly synchronized with their steps. According to Zach, accelerometers mounted to the person could determine gait, but our current plan calls for mounting the wii device on the cane, not on the person, so we cannot assume that periodic motion in the z-axis (as reported by the wii's accelerometers) is related to stepping movements. It is possible to add another wii device and attach that one to the user's belt (for example), but we are trying to avoid adding additional hardware if that's possible.

9. Arc centered-ness. Another feature of classic two point touch technique is that the arc in the x-y plane that is described by the cane as it swings back and forth should be centered. That is, the line that bisects the arc should align with the travel path, and, conversely, the angular displacement of the cane to the right and left sides should be equal. it appears that this will be extremely difficult to determine conclusively, and we need to think carefully about how important this measurement is to the overall usefulness of the eventual wiiCane training system.

6. Veering. The literature indicates that everyone tends to veer in the absence of usable visual, tactile, or auditory information about the environment. Current research is considering ways to train blind pedestrians to overcome or correct for this tendency, because some mobility tasks, such as crossing wide streets, can be dangerous for blind travelers, and experts believe that if individuals could be trained to walk straight, accidents could be reduced and independence improved. One goal of the WiiCane project is to particate in this ongoing inquiry, and to produce a very practical tool for measuring veering.

Veering occurs (in my definition) when the wiiCane user's path deviates from a straight line in respect to a fixed perpendiular surface (the surface against which the user "squared-off" before starting to walk). Veering will be measured by the wii's camera, which can be trained to observe the position of infrared lights that we will array in the environment. Lights can be mounted on a variety of surfaces (including floor, ceiling, walls, the cane and the user's body), and one big job we are facing now is to study a variety of configurations to determine what works best. This is a challenging problem, because we have to be able to detect veering and to correct the user's performance before he or she has walked off to a position where they can no longer be "seen" by the wii's camera. So, our plan for providing feedback in this case is of the utmost importance if we expect to avoid the frustration of having to repeatedly return the user to the starting point because our system has lost track of their location and cannot recover. My guess is that, as with cane width, it will work best to provide spoken corrective messages when veering occurs with adults, and positive earcons when a child is successfully walking straight.

7. Distance. One important measurement that should be relatively easy to take will be distance to the goal. In the event that lights are positioned along a horizontal bar at the goal end of the course, we believe that the wii will be able to use a multi-step process, including a range-finding step (where we determine which are the widest apart lights that can be seen in a typical arc to set the gross distance) and a fine-tuning step where we look at the perceived distance between the lights. This measurement, however, can only take place when the user is more or less pointing the cane in the direction of the goal. Distance measurement will be an important component in constructing a graphical display of cane movement over time. There also may feedback opportunities associated with the student's position along the course, for example, we might want the system to speak "ten feet", "you are halfway there", or "turn left" in the event we construct a game which includes more complex route configurations.

9. Goal attained/Point awarded. Another form of feedback that we need to consider is associated with events that are not, strictly speaking, driven by cane motion. In developing a curriculum for building cane skills using wiiCane, we may want to consider various interactive activities involving game scenarios similar to SAL, which is our braille literacy system. That has a number of simple games that students and teachers find incentivizing, especially when a score for an activity is announced, and the student can practice to get a better result. We need to create a lexicon of feedback options that are stimulating and helpful for all of the populations for whom we are designing.

11. Improper usage. We know that some behaviors with the wiiCane will not be trackable. In the event the user veers significantly, there may be no way to recover location awareness, and the instructor will have to intervene. If this experience is repeated often enough, students may get discouraged and lose interest. In general, we want to promote a feeling of independence and control, and having to go back to the beginning may disincentivize further training. Therefore, we need to consider how we will use feedback to head off extreme veering before it's too late. Other researchers have set up courses that were designed to measure the onset and degree of veering; but in our case, we want to prevent veering to the point that we no longer can sense the user's position. I think feedbacks can be designed that accomplish this, but that are not overly intrusive or interfering with the student's efforts to practice walking straight.

12. Dangerous usage. In extreme cases, it will be necessary to let the user know that their movements with the cane are potentially dangerous. We have to observe a lot of people using the system before we will know all of the potentially dangerous things that could happen, and we have to be careful to carefully supervise subjects while they are using the product at all times. There are also product liablility issues that must be considered in any product where people are mobile, especially if our system calls for occluding hearing or vision. Also, in designing any curriculum for the system, we have to make sure to consider possible injuries, and discuss necessary warnings and precautions with someone knowledgeable about this (like a lawyer).

13. System alerts. Finally, we must consider that there may be a number of system warnings that we may want to include in feedback schedules. For sure, there should be some kind of "low battery" warning.

Data logging and graphical representations

In our proposal, we talked specifically about the need for maintaining a record of student performance as they carry our various training excercises, games or other activities with the WiiCane. Depending on the applications to which our system is put, we need to capture movement along the course in formats that will be of interest and of use to trainers and teachers. So, we need to begin considering how this will be accomplished in our system, and who will be responsibile for its implementation.

We looked at the output from RoboCane, a project carried out by Bruce Blasch, et. al. as a starting place for thinking about this.

Robocane was a computer simulator for cane travel. Researchers could enter parameters like cane length and traveler dimensions, and

the system produced a graphical representation of theoretical traveler matching those characteristics walking a straight course was produced. The purpose of this software was to allow the developers see the impact of making changes to various paramters on efficiency and safety for the theoretical traveler. It was expected that this information could be interpreted in the context of training actual students, to help the traveler optimize his or her performance by adjusting individual attributes of his or her technique.

Graphical record of cane travel from RoboCane

from: Blasch, Weiner & Welsh, Foundations of Orientation and Mobility, second edition

I am hoping that our system will be capable of generating graphical output like the sketch in this post. This shows a record of cane movement over a course: slanted lines show the top view of the cane at each time increment, and the line that represents the widest sweep (coinciding with the observed cane tap event) is darkened to make it stand out and to emphasize the period of the classic sine wave pattern produced during perfect execution of two-point touch technique. A side view will help illustrate the movement of the cane in the z (vertical) direction; a different sine wave will show the movement of the cane as the tip reaches its apex (height above the floor) at mid-swing.

It should be relatively straightforward to generate a visualization of wiiCane data like I am describing here. I am hoping that we can set up a process to do this as part of the software running on the computer that is being used to communictate with the wii. It would be helpful if a visual display on the computer showed the diagram being produced in real time. In the Optotrak system that we used in the verification experiment at WMU, an image can be produced that shows the movement of the cane in all three axes, however this is shown as three separate lines, and the there is no real-time display of the data. I will ask Rob to post an image of the data that Optotrak produced when we ran some subjects last week.

wiiCane fixture design concept

Okay, here is my idea for how to fix the wii to the cane so that all of my conditions in the previous post are satisfied. This is a two-part plastic assembly. The wii is slid into the pivoting cradle (labled "pivoting section"). There is a matching slot on the device to accept the little track, so you can slide the device in until a click is heard. This occurs when the part labeled "detente" drops into a matching recess on the device. Once the device is in place, it should not tend to slide back out, regardless of the angle that it is held, although this will have to be tested. It's very important that the device be held motionless in relation to the cane. The pivoting section, then, is placed within the fixed section, and 2 pairs of screws with matching threaded sleeves are used to hold this assembly together. When the screws are loose, it is possible to freely rotate the wii (held by the pivoting section) 360 degrees, with minimal blockage of the wii's camera, even when the angle of rotations is almost 0 (in respect to the cane). The camera will be blocked, and the wiiCane unusable, when the device is exactly aligned with the camera, but it seems to me that this does not limit the kinds of light configurations that we will be able to observe and respond to during travel exercises.

If it turns out that everynoe likes this approach, I will have 3D prints made so that we can begin evaluating the parts quickly.

Designing the WiiCane

In response to general agreement among team members that I have spoken to, I am going forward with some preliminary sketches for the fitting that we will use to mount the WiiMote device to the cane. My operating assumptions are as follows:

1. The fixture needs to be able to hold the wii so that it can point in any direction along a vertical plane that passes through the idealized travel path. There may be different ways to use the device that call for positioning lights in ways that are hard to predict now. This fixture will let us experiment with various configurations and courses. For example, with this design, we could place lights on the floor, at the goal, on the user's body or shoes, or on the ceiling. This appears to give us enough flexibiliy so that we will be able to determine the best way forward experimentally.

2. The fixture needs to distribute the weight of the device evenly and symetrically regardless of which way the device is pointing. In the current design, as long as the cane is held so that the flat portion of the grip is sideways, the cane will remain balanced. Since the direction that the flat of the grip points is determined by the hand it is held in, it may be necessary to remove the wii from the fixture and reverse it depending on whether the user is a rightly or a lefty.

3. The fixture needs to be lightweight, so that the experience of using WiiCane is not very different from using a regular cane.

4. The fixture has to be strong enough to withstand reasonable abuse. Since we are replacing an 8" section of the cane with the plastic fixture, significant forces may be imposed on this part. While the design will take into account this requirement, we may not be able to acheive the same strength from the 3D print as we would expect from a metal casting or mass produced plastic part.

5. In this approach, where we modify a standard cane with our plastic fixture, it will probably be most practical to use an adjustable length cane like the one we borrowed from Rob in K-z00, rather than multiple fixed-length canes as in our prior approach.

6. there needs to be a very simple way to insert and remove the device in the fixture. I am thinking about a thumb screw or hand-tightened knob on the side that will permit us to loosen the knob or screw, then swivel the unit to the desired angle and then tighten it.

7. There should be some form or pointer or indicator to register the angle of the unit's tilt (in some suitable units), so that teachers can set the angle with precision.

WiiCane Kalamazoo Verification Exercise Report June 23, 2009

Click here to download the pdf version of this report.

Click here to download the text version of this report.

WiiCane System - Technical Description

Apologies for the long post, but Steven has asked me to present the tech side of the WiiCane system as it stands now. Lots of new stuff here - I hope some of it is of interest!

The system tested at Western Michigan University was made up of a strip of discretely-controllable infrared emitters (940 nm wavelength) fixed in a wooden strip on 12" centers. This light strip, composed of 32 individual emitters, together with a controller device attached to the host computer, provided reference lights detectable by the Wii Remote infrared sensor.

As the Wii Remote's infrared sensor is sensitive to light in the 940 nm spectrum, we are able to derive positional and orientation information of the Remote relative to the light strip by analyzing the perceived locations of lights. The sensor itself provides Cartesian coordinates and perceived brightness values for up to four lights in the sensor's field of view. In the presence of more than four lights, the sensor reports information on the four it perceives to be brightest. Generally these are the four lights closest to the camera, but the perceived brightness may be affected by other factors such as occlusion of lights or off-axis orientation of the Remote relative to the emitter's emission angle. The top picture in this post shows the host application view of four infrared emitters in a double strip.

The software used for the trial attempts to determine arc width and to measure the amount of veering. It does this by analyzing raw accelerometer and infrared sensor data provided by resources in the Remote. Accelerometer data is normalized and used to detect discrete "tap" events marking the beginning and end of an arc. During an arc, data from the infrared sensor is used to develop a model of the traveler's movements in order to determine arc width and amount of veering.

To analyze movement, the reported infrared sensor data is plotted and examined from frame to frame. In order to calculate arc width, the position of all visible lights is noted at the beginning of an arc (defined as the first "tap") and the changing position of these marker lights is then followed as the traveler completes the arc. The final position of the lights at the end of the arc or second "tap" is noted and the total movement is calculated based on the distance between the initial and final positions. A scaling factor (experimentally determined based on cane length) is then applied to the calculated distance value to produce a reading in mm.

Veering detection follows a similar method. Once an arc is completed, the arc time is first calculated based on the time between the first and second taps. The infrared sensor data collected for the arc is then examined - it is scanned to find the time when the Remote was most closely aligned with the light strip (determined by examining the light locations which outline the strip itself). Since the light strip makes up the desired course of travel and since the arc should cross the light strip approximately halfway through the arc when the traveler is not veering, it is possible, based on the timing figures just calculated, to determine how much veering has occurred and in which direction. The highlighted area in the bottom picture in this post shows typical results for the analysis of three cane arcs.

In addition to the analysis tasks performed by the host software, it also serves to log raw accelerometer and infrared sensor data received from the remote for later analysis. A simple display area is also provided to aid visual analysis and debugging.

I just got done meeting with Zach, and we came up with an idea that I think will solve several problems at once. This image to the left shows a customized cane that is designed to hold a Wii remote device so that it can be angled such that the camera is pointing either down towards lights in a strip on the floor, or level, towards lights mounted at the goal end of the course. This allows us to continue to experiment with both versions, and also could end up becoming part of the final product. To make this cane, we propose starting with an adjustable length cane, like the one that Rob Wall Emerson showed us when we visited him last week in Kalamazoo. We would cut out a 6" section of the shaft of the cane near the grip, and replace that with a plastic fork that holds the wii device such that it can pivot. A tightening screw would allow us to fix the angle as needed to account for different configurations of sensing lights, and also to compensate for diffences due to the height of the user and cane length. One nice thing about this is that it should not noticably affect the balance of the cane.

latest results on feedback and veering

After 38 treatments composed of feedback about left and right veering, you can see the trend toward less veeering from an intended path. Guth recommends about 160 treatments to see signifcant and lasting changes to veering behavior. The x axis is the number of treatments; the y axis is the number of paces walked forward before the subject veered into the wall on the left or right.