Observer Error in Identifying Species Using Indirect Signs: Analysis of a River Otter Track Survey Technique

1. What they did

Texas Parks and Wildlife Department (TPWD) biologists have searched for river otter tracks under road bridges in east Texas since 1977. The share of bridges with otter tracks was used as an index of how many otters there were, and that index supports the state’s reports on otter harvest for international trade. A 2004 review questioned whether the population was really stable, partly because nobody had measured how well the observers identified tracks.

The author used two approaches to test whether observers affected the results:

• Reanalysing the survey data. Four standardized surveys (1996, 1997, 2000 and 2003), each of about 250 bridges in 27 counties, from the 13 observers who had searched more than 25 bridges. He tested whether observer, watershed, water type, vegetation type and bridge area predicted whether otters were reported, and whether observers were linked to these habitat factors.
• Testing observer skill. At two workshops (October to November 2005 and January 2006), 23 TPWD staff, including 7 experienced otter observers, took a CyberTracker track and sign evaluation run by evaluator Mark Elbroch. Only their answers about otter tracks were analysed. From these the author calculated, for each experienced observer, how often otter tracks were missed (false negatives) and how often tracks of “otter-like” species were called otter (false positives).

2. What they found

The bridge survey

• The share of bridges where each observer reported otters ranged from 7% to 59% (mean 34%). Observer was strongly linked to whether otters were reported.
• Observers were not assigned to bridges at random but by county, so they were also linked to all four habitat factors. Watershed and bridge area also predicted reported otters, much more weakly than observer did. Water type did not.

The skill evaluation

• Tracks of 12 species were called otter at least once; 8 of these were called otter more than once and counted as “otter-like”. Swamp rabbit tracks were called otter in 9 of 20 answers (45%), raccoon 24 of 193 (12%) and opossum 11 of 109 (10%).
• Experienced observers called 6% to 19% of otter-like tracks otter (mean 13%).
• They misidentified on average 44% of otter tracks, ranging from 0% to 100% between observers. These figures rest on only 4 or 9 otter questions per person.
• Experienced observers scored higher overall than the other staff (69% versus 59%). Of the 7, one reached level 2, four reached level 1 and two did not reach a level; of the other 16, three reached level 1.

3. What they concluded

• Temporal changes in the Texas otter population cannot be inferred from these survey data.
• The survey probably both over-reported otters (otter-like tracks called otter, especially raccoon and opossum, which are common under bridges) and under-reported them (otter tracks missed).
• Correct identification of indirect signs is challenging, and it cannot be assumed that even experienced field observers are accurate. Few species leave tracks so distinctive they are never confused, so the problem is likely to be widespread, not limited to otters in Texas.
• Studies using indirect signs can only be judged reliable if the signs are unmistakable even to a novice, or observer skill is measured and reported.
• Managers have three options: train and evaluate current observers; select observers already evaluated at an adequate level; or record tracks (track plates, plaster casts, photographs) and have an evaluated track and sign specialist confirm the identification.
• Appendix C sets out recommendations for TPWD, including training and evaluating staff regularly, a 90% skill threshold where possible, a photo library of otter and otter-like tracks, measuring detection probability each survey, and redesigning the survey by watershed with randomly chosen bridges.

4. How tracking was or was not used

The whole study is about tracks and signs identification. In the bridge survey, observers working mostly alone searched all suitable ground under each bridge for footprints of otters and other furbearers; scats and other signs were not recorded. The skill evaluation used tracks and signs of many species in the field, chosen to test broad natural history knowledge rather than otters alone. No trailing was involved.

5. Tracker or observer certification

The survey observers had a one-day training in 1996, and new observers were later trained in the field by experienced co-workers. Their skill had never been tested before this study. The CyberTracker evaluation, developed by Louis Liebenberg, weights each question by difficulty and awards level 1 (70 to 79%), level 2 (80 to 89%), level 3 (90 to 99%) and track and sign specialist (100%). It was run by Mark Elbroch, a certified senior tracker evaluator in South Africa and North America. The author notes that in South Africa, trackers are generally only hired for research if they reach level 3 or higher.

6. Other technologies, skills or innovations

• The CyberTracker evaluation, used here both to check the validity of data already collected and as a training tool. A companion study (Wharton 2006) found scores rose from a mean of 61% to 79% three months later.
• Geographic Information System (GIS) layers to link bridge locations with watershed, water type, vegetation type and bridge area.
• An appendix of photographs of otter tracks and otter-like tracks (raccoon, opossum, bobcat, nutria), and a list of resources for improving tracking skills.

7. Implications for conservation (from the paper)

• Reliable otter data are needed for informed management, especially as high pelt prices may increase trapping pressure. Bridge surveys that do not account for observer skill could lead to misinformed management decisions.
• Even 100% accurate track identification would not fix all the survey’s problems: detection probability, non-random bridge selection and survey intervals also need attention.
• Standardized evaluations of observer skill, combined with occupancy estimation, would allow sound inferences about populations. In the United States, such evaluations could help validate the skills of citizen scientists and data collectors without formal wildlife training.