The curse of observer experience: Error in noninvasive genetic sampling

1. What they did

Noninvasive genetic sampling (NGS) means collecting DNA from things animals leave behind, such as scat and hair, so that rare or elusive animals can be studied without being caught. Its success depends on observers finding the right scats, identifying them correctly and sampling them well. The authors asked whether observers with previous experience on the project do this better than newcomers.

Over seven summers (2008 to 2014), paid field crews searched for gray wolf scats in central Idaho, Yellowstone National Park and southwestern Alberta, Canada:

• The observers. All were students or recent graduates of university natural resources programs, with prior wildlife fieldwork but not necessarily NGS experience. Anyone new to the project counted as “new”; anyone who had worked on it in an earlier year counted as “experienced”.
• Training. Everyone had three days of training: one day of pictures of wolf sign (tracks, scats, trails), measuring plaster casts of wolf tracks and looking at wolf scats in the lab, then two days in the field looking at tracks and scats of wolves, coyotes and other carnivores, and sampling wolf scats under supervision.
• Surveys. At dawn and dusk, crews drove and walked roads and hiked off trail looking for tracks and scat around predicted wolf rendezvous sites (places where a pack gathers and feeds pups). They howled to get a response from wolves, then searched for the activity centre. Observers were not assigned sites by experience.
• Sampling. Observers collected canid scats over 2.5 cm in diameter as “wolf”, took a small sample from the outside of each scat and stored it for DNA analysis.
• Checking the results. A laboratory tested each sample’s species with DNA and tried to get a genotype. Scats from active rendezvous sites (almost all wolf) were used to measure DNA amplification success. Scats a single observer found more than 500 m from an active rendezvous site (“incidentals”), which could be from other species, were used to measure how often observers correctly identified wolf scat.

2. What they found

• 34 people took part, giving 58 observer-years (33 new, 25 experienced). They collected 4,836 scat samples; 1,899 (39%) gave no usable DNA.
• Finding scats: new observers collected more samples at active rendezvous sites than experienced observers (96.4 versus 66.2 on average), and more that successfully amplified (59.5 versus 40.2).
• Sample quality: the proportion of scats that amplified did not differ (0.60 for new versus 0.62 for experienced observers).
• Identifying wolf scat: both groups collected similar numbers of incidental scats (15.1 versus 16.0). New observers were right more often that a scat was from a wolf (probability 0.66 versus 0.50), but this difference was weaker than the others and not significant (p = 0.08).
• Individuals varied widely: amplification rates ranged from 0.29 to 0.89, and correct species identification from 0.20 to 0.90.
• Incidental scats were 24% of all samples; the other 76% came from active rendezvous sites.

3. What they concluded

• Contrary to their hypothesis, previous experience did not give better results. New observers collected more samples and identified the right species slightly more often.
• They offer two possible explanations: experienced observers may be less cautious and collect more “questionable” scats, which brings in more scats of other species; or they may be tired or bored. Fatigue and boredom were not measured in this study.
• New observers need rigorous training, but experienced observers may also need attention and retraining.
• Without the genetic check, the experienced observers’ misidentifications would have produced false detections of wolves and biased occupancy estimates upwards. Projects that rely only on field identification of scat may face the same problem.
• Spending more than two days on field training may bring little extra benefit. Observer experience should be weighed against project goals, and previous experience collecting fecal DNA does not guarantee greater success.

4. How tracking was or was not used

Finding and identifying scat was the heart of the study. Observers searched roads, trails and off-trail areas for wolf tracks and scats to locate rendezvous sites, and identified wolf scats in the field by size (canid scats over 2.5 cm across) and by the training they had received in telling wolf tracks and scats from those of coyotes and other carnivores. DNA testing was then used to check whether each incidental scat really was from a wolf, which turned the survey into a test of each observer’s scat identification. No trailing is reported. The authors note that tracking skills and knowledge of the target species’ habits are important, but that the temperament of observers also plays a role.

5. Tracker or observer certification

No certification is reported. All observers had prior wildlife fieldwork experience and three days of project training in wolf sign identification. “Experienced” here means having worked on this project in an earlier year, not a tested tracking skill. Their scat identification was, however, checked in this study against DNA species tests, and new observers did at least as well as experienced ones. One experienced observer was one of the authors (the paper does not say which).

6. Other technologies, skills or innovations

• A habitat model predicting rendezvous sites (wet meadows and standing water) to target survey effort.
• Howling surveys to detect wolves and locate activity centres.
• Global Positioning System (GPS) collars on wolves in Idaho and Yellowstone from 2011 to 2014, which let crews walk straight into active rendezvous sites; scats from those areas and years were left out of the species identification analysis.
• DNA species identification and individual genotyping of scats (mitochondrial DNA test, nuclear microsatellites, GENALEX, STRUCTURE, RELIOTYPE).

7. Implications for conservation (from the paper)

• Observer error in scat surveys can lead to false detections and biased occupancy estimates; for rare or imperilled species such errors can have grave consequences.
• Projects using several observers should expect large differences between individuals and work to limit observer bias.
• Project managers might put more weight on tracking ability when hiring and training, since DNA amplification rates did not differ between groups.