Are We Gathering Reliable Data? The Need for Measuring Observer Skill in Wildlife Monitoring

1. What they did

Wildlife surveys often rely on tracks, scats and other signs, so their results depend on how well observers identify those signs. Yet observer skill is rarely measured. The author reviewed the literature on observer error and looked at one way of measuring skill: CyberTracker Conservation’s Track and Sign Interpretation Evaluation, part of the Tracker Evaluation System created by Louis Liebenberg in 1994.

As an intern with the Texas Parks and Wildlife Department (TPWD), the author helped organise two evaluation workshops for department staff in east Texas. The department monitors northern river otters with track surveys under bridges, and chose the evaluation as a pilot to measure observer reliability and to train staff.

• The evaluation. Over two or more days in the field, an evaluator points out tracks and signs and participants answer questions on their own, such as: What species made this? Which foot is this? How is this animal moving? What was it doing? Answers are then discussed as a group. Questions cover all species in the area, not only otters, and are weighted by difficulty (easy 1 point, difficult 2, very difficult 3). Wrong answers on easy questions cost the most. At least 35 questions are asked.
• The workshops. Workshop I was held in October 2005 and Workshop II three months later, in January 2006, at bridges, wetlands and uplands in Jasper and Newton Counties. Both were run by Mark Elbroch, a certified senior tracker evaluator. Nineteen employees attended both.
• Feedback. Participants filled in anonymous questionnaires before and after each workshop, rating their confidence, how well the process measured their skill, and their satisfaction.

2. What they found

Scores

• Mean scores rose from 62% (standard deviation 9%) at Workshop I to 79% (standard deviation 8%) at Workshop II, a mean gain of 17 percentage points.
• The top score rose from 74% to 90%. One participant went from 69% to 90%.
• Scores also rose within Workshop I: an average of 50% correct on the first day and 67% on the second.
• Certificates: six Level 1 certificates at Workshop I (13 of the 19 scored below 70%). At Workshop II, one Level 3, ten Level 2 and six Level 1 (two scored below 70%).

Participant feedback (means on a scale of 1 to 5, Workshop I then Workshop II)

• Confidence in identifying river otter tracks: 3.3, then 3.8.
• Confidence in identifying other tracks and signs: 3.0, then 3.5.
• How well the process measured their skill: 4.2, then 4.4.
• Satisfaction: 4.6, then 4.9. Would recommend it to other districts: 4.7, then 4.9.
• Most wanted evaluations every year or every two years. The most common suggestion after Workshop I was smaller groups (11 people).
• Before Workshop I, 17 of 19 expected a certificate, but only 6 received one (Appendix C).
• Between workshops, participants practised track identification three to four times on average, so not all of the improvement can be credited to the workshops.

What made the workshops work

The author suggests four reasons, based on informal discussions: group discussion and tips after participants had given their own answers; the evaluator’s expertise and communication; participants’ own motivation; and the short three months between workshops. One participant compared learning to see a track to seeing the hidden image in a 3-D poster.

3. What they concluded

• The Track and Sign Interpretation Evaluation can do two jobs at once: measure observer reliability and train observers.
• Experience does not always match skill, and even experienced biologists can vary greatly. It is often assumed that professional biologists or experienced observers will all score highly, and this should not be taken for granted.
• The evaluation could be used to screen applicants for monitoring work, to find skilled trackers outside wildlife degree holders, and as quality control for volunteer and citizen science projects.
• As the system grows, CyberTracker Conservation faces the challenge of keeping the standard consistent across evaluators and regions. Safeguards described include evaluators needing a 100% score and assistant evaluator experience, and external evaluators overseeing local ones in Africa.
• Broader use of this or similar assessments could help researchers tell whether they are seeing real changes in wildlife populations or the effects of observer skill.

4. How tracking was or was not used

Tracking is the subject of the study. Participants were tested on tracks and signs identification and interpretation (species, which foot, how the animal moved, what it was doing), including tracks, trails and scat of mammals, birds and reptiles. The evaluation was the Track and Sign Interpretation type, which does not include trailing. In one Workshop II question about a trail of dots, only one participant followed the trail down a slope and found that it was made by a rolling sweet gum seed ball.

5. Tracker or observer certification

Both workshops were run by Mark Elbroch, described as a certified senior tracker evaluator with CyberTracker Conservation. Participants’ skill was measured with the standard CyberTracker Track and Sign Interpretation Evaluation, and certificates (Levels 1 to 3) were awarded by score. Participants’ earlier experience varied: not all had worked on bridge surveys, and they reported learning from field guides, self-teaching, on-the-job training, other biologists and otter surveys.

6. Other technologies, skills or innovations

• A weighted scoring system in which question value depends on the species, the condition of the track or sign, and whether similar species are present; wrong answers on easy questions are penalised most.
• Certificate levels: Level 1 (70 to 79%), Level 2 (80 to 89%), Level 3 (90 to 99%); those scoring 100% can take a senior evaluation of at least 50 very difficult questions for a Track and Sign Specialist Certificate.
• Pre- and post-workshop questionnaires used to measure confidence and satisfaction alongside test scores.

7. Implications for conservation (from the paper)

• Many population studies rely on tracks and scats, so their results depend on surveyors identifying signs correctly. Poor management decisions based on untested assumptions about observers are costly.
• Observer skill should be assessed and reported, because observers are a major source of variation and bias even when highly trained.
• A network of certified trackers could help researchers monitor wildlife more accurately and managers make better decisions.