Evaluating the accuracy of wolverine identification from photographs of snow tracks by expert observers in North America

1. What they did

The Cascades Wolverine Project (CWP) runs a community science program in the Cascade Mountains of Washington State, where wolverines appear to be returning to range they lost early in the 20th century. Backcountry travellers send in photographs of possible wolverine tracks through an online portal: skiers, snowshoers and snowmobilers in winter, and hikers and mountaineers in summer. Two experts review each submission independently. The authors wanted to know whether identifying wolverine from such photographs is reliable.

They built a test of 99 “observations”. Each observation was a set of 1 to 8 photographs of the same species from the same event:

• 48 were of wolverine. The other 51 were of species whose tracks members of the public had mistaken for wolverine. Lynx and fisher were included in the largest numbers, because the authors expected them to be the hardest to tell apart.
• The species that made every set of tracks was independently verified: by direct observation of the animal (82), which included seeing the animal and then photographing its tracks, following tracks until the animal was seen, or photographing tracks of an animal caught in or released from a live trap; by a remote camera photograph (1); by genetic samples (hair or scat) from the tracks that matched the field identification (15); or by a combination of methods (1).
• 89 observations had an object of known size in the photo, such as a hand, trekking pole or notebook.
• Each observation was given a photographic snow track quality (PSTQ) rating from 1 to 4. The rating, amended from an earlier method, describes whether the individual track details and the track pattern are clear or obscure.

They tested 29 observers (8 women and 21 men), all known colleagues of at least one author:

• They chose observers for two kinds of expertise: high skill in identifying wildlife tracks, and academic training with field experience of wolverines. Some observers had both.
• They recorded each observer’s academic degree, CyberTracker certification level, self-rated expertise, how often they had seen possible wolverine tracks in the field, and any past experience reviewing track photographs systematically.
• Everyone received the same instructional video, including how to use a ruled scale, plus suggested reference materials. Observers could use books, notes and photographs but could not discuss the test with anyone. They were given no training on wolverine tracks specifically and were not told which species were included.
• Everyone saw the same 99 observations in the same order online, randomly mixed by species and quality, with no time limit.
• Observers placed each observation in one of four categories: definitively wolverine, likely wolverine, identity unknown, or definitively not wolverine. “Definitively wolverine” required at least two observable characteristics that rule out every other species.

2. What they found

Overall

• Observers made 2,851 reviews in total. Each observer reviewed 92 to 99 observations, skipping any they had seen before.
• False positives (another species called “definitively wolverine”): only 12 out of 1,473 opportunities, a mean error rate of 0.9%. Twenty-one of the 29 observers made none. Counting “likely wolverine” as well, the mean false-positive rate rose to 3.5%, and 8 observers made none.
• True positives: observers called real wolverine tracks “definitively wolverine” 42.6% of the time across all observations. Including “likely wolverine”, the figure was 65.6%. About a quarter of all observations (25.2%) were called “unknown”.

Photograph quality

PSTQ rating

Definitively wolverine (true positives)

Definitively or likely wolverine

Called “unknown”

1: track details and pattern both clear

74.5%

88.0%

7.2%

2: details clear, pattern obscure

40.2%

85.5%

5.9%

3: details obscure, pattern clear

33.2%

63.5%

30.2%

4: details and pattern both obscure

3.8%

14.1%

53.5%

• False-positive rates stayed low at every quality level, at 1.8% or less for “definitively wolverine”. The highest rate, 6.0%, was for PSTQ 3 when “likely” answers were included. Poor photographs mostly led to “unknown” answers, not to more errors.
• The number of photographs in an observation was not related to true-positive or false-positive rates.

Observer background

• False-positive rates were similar across education levels, field experience with wolverine tracks and past experience reviewing track photos.
• Specialist-certified observers (7) were the only group that made no false-positive errors.
• Observers with Level 3 or Specialist certification found more wolverines than those with no certification or Level 2. For “definitively wolverine” the rates were 48.2% versus 34.7%; including “likely”, they were 70.4% versus 58.9%. The confidence intervals of the two groups did not overlap.
• Certification level was the only clear background factor that predicted skill.

Species confused with wolverine

• Of the 12 false positives, 6 were fisher, 5 were lynx and 1 was river otter.
• Lynx tracks were called wolverine 0.8% of the time, the same as the overall error rate. Fisher tracks were called wolverine 1.6% of the time.
• Every fisher error came from tracks too small for any published wolverine size range given to the observers.
• Only two observations caused more than one error: a lynx observation (two errors) and a fisher observation (four errors).

3. What they concluded

• A wide variety of observers can distinguish wolverine from other species with similar tracks, using photographs of snow tracks, and are especially good at avoiding false identifications. The ability to avoid false positives is a foundation of a reliable dataset.
• Detection rates were comparable with those reported for genetic samples collected remotely.
• Members of the public and less skilled field staff can make the first observations, with experts confirming identifications later. This can greatly increase the effort, spread and lifespan of a study.
• Wildlife tracking skill is independent of scientific training, and studies using track evidence should ensure that the people interpreting it have the right expertise.
• Observers’ confidence matched their accuracy: “likely” answers carried more errors than “definitive” ones, and poor photos were mostly called “unknown”. The authors suggest this shows observers can judge their own reliability.
• Avoiding errors: fisher errors could be avoided by requiring every definitive wolverine identification to show tracks within the expected size range. CWP uses the most cautious category when its two reviewers disagree; projects can take a more liberal approach if their goal allows. The error and detection rates measured here can also be used to account for uncertainty in models.
• Limitations: some real wolverine observations will never reach “definitively wolverine”, depending on observer skill and photo quality. The design was narrow, made to mimic community science submissions. Observers knew wolverine was the target species, and the species were not evenly represented.
• The published assumption that track identification produces unreliable data “appears to be false in this application”. The authors suggest that the growing preference for genetic evidence for simple species identification may reflect a lack of confidence among survey designers, not a real lack of reliability in tracks.

4. How tracking was or was not used

• The test was entirely tracks and signs identification from photographs of tracks in snow. Observers used both details of individual tracks (the shape, size and arrangement of toes and palm pad, and the length and direction of claws) and track patterns (gaits), for example the lope used by wolverine compared with the bound used by lynx at speed, and walk or trot patterns that several species share.
• A scale of known size was central to the identifications.
• Trailing was used to verify some observations: people followed tracks until they saw the animal. The paper does not say how many observations were verified this way.

5. Tracker or observer certification

• CyberTracker certification was the main objective measure of tracking skill. The paper describes it as an international system that awards certification based on performance in a rigorous field test.
• Of the 29 observers, 9 had no certification, 3 had Track and Sign Level 2, 10 had Level 3 and 7 had Specialist.
• As described in the paper: Level 2 shows 80–90% accuracy across tracks and signs of a wide range of difficulty; Level 3 shows 90% or higher; Specialist shows near 100% accuracy on very difficult tracks and signs in the field.
• Self-rated expertise was recorded separately: 5 amateur, 15 professional and 9 expert.
• The paper states a conflict of interest: the lead author is on the Board of Directors of Tracker Certification North America, which oversees CyberTracker Conservation certifications in North America, and works as an independent contractor running field evaluations for this system.

6. Other technologies, skills or innovations

• Smartphones: the paper notes that smartphones carried by the public can take high-quality, georeferenced photographs, which makes them excellent tools for collecting snow track data.
• PSTQ rating: the photographic snow track quality system is a way to sort observations by how much detail they hold.
• Verification and test design: track identities were verified by genetic samples (hair and scat) and, in one case, a remote camera. The test ran on an online platform, and random identification numbers allowed blind analysis.

7. Implications for conservation (from the paper)

• Wolverines are listed as threatened in the contiguous United States under the federal Endangered Species Act and as Special Concern by the Committee on the Status of Endangered Wildlife in Canada (COSEWIC). They live in remote, rugged terrain that is hard to survey, and the impact of winter recreation on them is a growing concern.
• Photographic track identification lets members of the public take part. It needs no specialized equipment and no storage, transport or analysis of fragile samples, so it is very cost effective. In suitable field conditions, the paper notes, track identification is a more consistent way to detect carnivores than remote cameras. Compared with genetic sampling, it is more straightforward to interpret, leaves less room for contamination, and allows a more transparent and repeatable outside review of the original evidence.
• Combined with community science, snow track evidence can greatly increase the power of monitoring. It also helps with community engagement, transparency, long-term monitoring and understanding how wolverine land use relates to recreation.
• The method could be repeated for other wide-ranging species of conservation concern.