An analysis of Perplexity's search models found that 34.7% of 1,826 citations linked to figures did not lead to accessible pages or contained no relevant figures. The study involved asking Perplexity’s models 310 factual questions about 210 technology companies and checking the cited sources for accuracy. Out of the 1,826 citations, 14.4% of 872 claims failed when evaluated per claim rather than citation. The majority of failures were not due to dead links; only 1.3% of cited URLs were dead. Instead, the issues stemmed from pages that were either inaccessible or did not contain the cited figures.
The analysis involved fetching all unique cited URLs, totaling 2,915 for the sonar model, and classifying them as dead, gated, empty, unreachable, or live. The results showed that 84.2% of sonar’s answers cited at least one URL that an ordinary reader could not access, and 10.6% cited at least one dead link. Specific examples of dead links were noted, including URLs that returned errors when checked.
The study also highlighted that 16.1% of the readable pages contained none of the figures from the claims they were cited for. For instance, a citation for Vercel’s pricing did not match the information on the cited page. The analysis concluded that a significant portion of the citations in Perplexity's responses could not be verified, raising concerns about the reliability of the information provided by the models. The findings were published under a Creative Commons license, and the full dataset is available online.