# Are We Gathering Reliable Data? The Need for Measuring Observer Skill in Wildlife Monitoring

> How good are wildlife survey crews at reading tracks? When 19 Texas Parks and Wildlife staff took the CyberTracker Track and Sign Interpretation Evaluation, their average score was 62%. Three months later it was 79%, and most had expected to do far better the first time. This thesis argues that measuring observer skill is both a check on data quality and a powerful way to train.

Summary by Kersey Lawrence, Original Wisdom. Updated 9 November 2026. Page: https://www.originalwisdom.com/tracking-literature/2006-wharton-we-gathering-reliable-data/

## Citation and access

- **Authors:** Ciel Amy Wharton
- **Year published:** 2006
- **Journal or source:** Master of Science professional paper, Wildlife and Fisheries Sciences Department, Texas A&M University
- **Full citation:** Wharton, C. A. 2006. Are we gathering reliable data? The need for measuring observer skill in wildlife monitoring. Master of Science professional paper, Wildlife and Fisheries Sciences Department, College of Agriculture and Life Sciences, Texas A&M University, Texas. 43 pp.
- **Access status:** Hosted (freely available elsewhere)
- **PDF:** https://www.originalwisdom.com/wp-content/uploads/2026/11/2006_Wharton_are-we-gathering-reliable-data-the-need-for-measuring-observer-skill-in-wildlife-monitoring.pdf

## Tags

- **Species:** Reptiles > American alligator; Mammals > Coyote; Birds > Great egret; Humans; Mammals > North American river otter; Mammals > Nutria; Mammals > Raccoon; Reptiles > Turtles; Mammals > Woodrats
- **Region:** North America > United States
- **Tracking:** Scat; Tracking; Tracks; Tracks and signs identification
- **Methods and technology:** CyberTracker; CyberTracker Tracker Certification; Descriptive statistics; Observer reliability; Survey instruments
- **Data type:** New field data
- **Observer qualification:** Certified trackers
- **Discipline:** Wildlife tracking
- **Year published:** 2006

## Detailed keywords

- **Species:** Studied: humans (19 Texas Parks and Wildlife Department employees tested at both evaluation workshops). Target species of the department’s bridge track surveys: northern river otter (Lontra canadensis), along with other furbearers. Tracks and signs used as sample evaluation questions (Appendix A): alligator trail, turtle trail, raccoon tracks, river otter tracks and scat, nutria tracks, coyote track, woodrat tracks and great egret track; a “trail of dots” that turned out to be made by the seed ball of a sweet gum tree. Mentioned from the literature: Ju/’Hoan trackers tested by Stander et al. (1997); grey foxes (Urocyon cinereoargenteus), coyotes (Canis latrans), cougars (Puma concolor) and Eurasian otters (Lutra lutra) as examples of species surveyed by tracking.
- **Location and habitat:** Bridges, wetland and upland areas in Jasper and Newton Counties, Piney Woods ecoregion, east Texas (Texas Parks and Wildlife Department District 6), United States. The department’s otter bridge surveys cover about 250 bridges in 27 east Texas counties. The Tracker Evaluation System is described as also used in South Africa and Botswana.
- **Topics:** observer reliability and observer error in wildlife monitoring; field sign (track and scat) surveys; CyberTracker Track and Sign Interpretation Evaluation as a measure of observer skill and as a training tool; weighted scoring and certificate levels; improvement between workshops; participant confidence and satisfaction; self-predicted versus achieved certificate levels; river otter monitoring for the Convention on International Trade in Endangered Species (CITES); screening observers; quality control for citizen science and volunteers; maintaining an international tracking standard.
- **Statistics or models:** Descriptive statistics only: means, standard deviations, ranges and box plots of scores; counts and means of Likert-scale (1 to 5) questionnaire responses. No hypothesis testing. Scores were calculated as correct points divided by correct plus incorrect points, using CyberTracker’s weighted system (Table 1). Inconsistency in the paper: the questionnaires in Appendix B for Workshop I are dated 31 October and 1 November 2006, but the text says Workshop I was held in October 2005 (the paper is dated May 2006).
- **Data type:** New field data: evaluation scores and anonymous pre- and post-workshop questionnaires from two workshops (Workshop I, October 2005; Workshop II, January 2006), plus participant observation and informal conversations. Includes a discussion of the literature on observer reliability. A Master of Science professional paper (not peer-reviewed).

## Synopsis

### 1. What they did

Wildlife surveys often rely on tracks, scats and other signs, so their results depend on how well observers identify those signs. Yet observer skill is rarely measured. The author reviewed the literature on observer error and looked at one way of measuring skill: CyberTracker Conservation’s Track and Sign Interpretation Evaluation, part of the Tracker Evaluation System created by Louis Liebenberg in 1994.

As an intern with the Texas Parks and Wildlife Department (TPWD), the author helped organise two evaluation workshops for department staff in east Texas. The department monitors northern river otters with track surveys under bridges, and chose the evaluation as a pilot to measure observer reliability and to train staff.

- The evaluation. Over two or more days in the field, an evaluator points out tracks and signs and participants answer questions on their own, such as: What species made this? Which foot is this? How is this animal moving? What was it doing? Answers are then discussed as a group. Questions cover all species in the area, not only otters, and are weighted by difficulty (easy 1 point, difficult 2, very difficult 3). Wrong answers on easy questions cost the most. At least 35 questions are asked.

- The workshops. Workshop I was held in October 2005 and Workshop II three months later, in January 2006, at bridges, wetlands and uplands in Jasper and Newton Counties. Both were run by Mark Elbroch, a certified senior tracker evaluator. Nineteen employees attended both.

- Feedback. Participants filled in anonymous questionnaires before and after each workshop, rating their confidence, how well the process measured their skill, and their satisfaction.

### 2. What they found

Scores

- Mean scores rose from 62% (standard deviation 9%) at Workshop I to 79% (standard deviation 8%) at Workshop II, a mean gain of 17 percentage points.

- The top score rose from 74% to 90%. One participant went from 69% to 90%.

- Scores also rose within Workshop I: an average of 50% correct on the first day and 67% on the second.

- Certificates: six Level 1 certificates at Workshop I (13 of the 19 scored below 70%). At Workshop II, one Level 3, ten Level 2 and six Level 1 (two scored below 70%).

Participant feedback (means on a scale of 1 to 5, Workshop I then Workshop II)

- Confidence in identifying river otter tracks: 3.3, then 3.8.

- Confidence in identifying other tracks and signs: 3.0, then 3.5.

- How well the process measured their skill: 4.2, then 4.4.

- Satisfaction: 4.6, then 4.9. Would recommend it to other districts: 4.7, then 4.9.

- Most wanted evaluations every year or every two years. The most common suggestion after Workshop I was smaller groups (11 people).

- Before Workshop I, 17 of 19 expected a certificate, but only 6 received one (Appendix C).

- Between workshops, participants practised track identification three to four times on average, so not all of the improvement can be credited to the workshops.

What made the workshops work

The author suggests four reasons, based on informal discussions: group discussion and tips after participants had given their own answers; the evaluator’s expertise and communication; participants’ own motivation; and the short three months between workshops. One participant compared learning to see a track to seeing the hidden image in a 3-D poster.

### 3. What they concluded

- The Track and Sign Interpretation Evaluation can do two jobs at once: measure observer reliability and train observers.

- Experience does not always match skill, and even experienced biologists can vary greatly. It is often assumed that professional biologists or experienced observers will all score highly, and this should not be taken for granted.

- The evaluation could be used to screen applicants for monitoring work, to find skilled trackers outside wildlife degree holders, and as quality control for volunteer and citizen science projects.

- As the system grows, CyberTracker Conservation faces the challenge of keeping the standard consistent across evaluators and regions. Safeguards described include evaluators needing a 100% score and assistant evaluator experience, and external evaluators overseeing local ones in Africa.

- Broader use of this or similar assessments could help researchers tell whether they are seeing real changes in wildlife populations or the effects of observer skill.

### 4. How tracking was or was not used

Tracking is the subject of the study. Participants were tested on tracks and signs identification and interpretation (species, which foot, how the animal moved, what it was doing), including tracks, trails and scat of mammals, birds and reptiles. The evaluation was the Track and Sign Interpretation type, which does not include trailing. In one Workshop II question about a trail of dots, only one participant followed the trail down a slope and found that it was made by a rolling sweet gum seed ball.

### 5. Tracker or observer certification

Both workshops were run by Mark Elbroch, described as a certified senior tracker evaluator with CyberTracker Conservation. Participants’ skill was measured with the standard CyberTracker Track and Sign Interpretation Evaluation, and certificates (Levels 1 to 3) were awarded by score. Participants’ earlier experience varied: not all had worked on bridge surveys, and they reported learning from field guides, self-teaching, on-the-job training, other biologists and otter surveys.

### 6. Other technologies, skills or innovations

- A weighted scoring system in which question value depends on the species, the condition of the track or sign, and whether similar species are present; wrong answers on easy questions are penalised most.

- Certificate levels: Level 1 (70 to 79%), Level 2 (80 to 89%), Level 3 (90 to 99%); those scoring 100% can take a senior evaluation of at least 50 very difficult questions for a Track and Sign Specialist Certificate.

- Pre- and post-workshop questionnaires used to measure confidence and satisfaction alongside test scores.

### 7. Implications for conservation (from the paper)

- Many population studies rely on tracks and scats, so their results depend on surveyors identifying signs correctly. Poor management decisions based on untested assumptions about observers are costly.

- Observer skill should be assessed and reported, because observers are a major source of variation and bias even when highly trained.

- A network of certified trackers could help researchers monitor wildlife more accurately and managers make better decisions.

### 8. Implications for tracking (Original Wisdom's observations)

_These are Original Wisdom's observations, not the authors'._

- Before Workshop I, most participants expected to earn a certificate, but most scored below 70%. Self-rated confidence was a poor guide to actual skill, which supports measuring skill directly.

- Testing on all species in the area, not only the target species, helps observers rule out look-alike tracks, which matters for single-species surveys such as otter bridge counts.

## Cite this summary

Lawrence, K. 2026. Summary of: Are We Gathering Reliable Data? The Need for Measuring Observer Skill in Wildlife Monitoring (Wharton, 2006). Original Wisdom, Tracking Literature. https://www.originalwisdom.com/tracking-literature/2006-wharton-we-gathering-reliable-data/