Package {rdomains}

rdomains: Classify Domains by their Content

Description

Want to know what kind of content is carried on a domain? Get the results quickly using rdomains. The package provides access to virustotal API, shalla, aws, OpenAI GPT models, Anthropic Claude models, and validated ML model based off shallalist data to predict content of a domain.

Details

To learn how to use rdomains, see this vignette: ../doc/rdomains.html.

Author(s)

Gaurav Sood

Probability that Domain Hosts Adult Content Based on features of Domain Name and Suffix alone.

Description

Uses a validated ML model that uses keywords in the domain name and suffix to predict probability that the domain hosts adult content. For more information see https://github.com/themains/keyword_porn

Usage

adult_ml1_cat(domains = NULL)

Arguments

Value

data.frame with original list and content category of the domains

Examples

## Not run: 
adult_ml1_cat("http://www.google.com")

## End(Not run)

Get Category from Anthropic Claude

Description

Fetches category of content hosted by a domain using Anthropic's Claude API. The function uses Claude models to classify domains into specified categories.

Usage

claude_cat(
  domains = NULL,
  api_key = NULL,
  categories = NULL,
  model = "claude-3-haiku-20240307",
  rate_limit = 0.5
)

Arguments

Value

data.frame with original list and content category of the domain

Examples

## Not run: 
claude_cat("google.com")
claude_cat(c("google.com", "facebook.com"))
claude_cat("google.com", categories = c("search", "social", "ecommerce", "news", "other"))

## End(Not run)

Fetch homepage HTML and text for domains

Description

Every requested domain comes back, in order, whether or not it was reachable. Failures carry a code from [fetch_error_codes()] rather than a bare 'NA'.

Usage

collect_content(
  domains = NULL,
  delay = 1,
  timeout = 10,
  max_bytes = 2 * 1024^2,
  obey_robots = TRUE,
  max_crawl_delay = 30,
  max_redirects = 5,
  user_agent = rdomains_user_agent()
)

Arguments

Details

The crawler identifies itself as 'rdomains/<version>', obeys 'robots.txt' including 'Crawl-delay', spaces requests to the same host, caps the response body, and refuses to fetch hosts that resolve to private or link-local addresses.

Value

A tibble with one row per input: 'domain_name', 'status', 'stage', 'error_code', 'retryable', 'http_status', 'final_url', 'fetched_at', 'content_bytes', 'title', 'description', 'lang', 'text', 'n_tokens', 'page_state', 'block_vendor', 'robots_allowed', 'source_last_published'.

See Also

[fetch_report()] to summarise the run, [page_signals()] for what the page states are, [source_vintage()] for how a live fetch compares with the static lists.

Examples

## Not run: 
res <- collect_content(c("example.com", "wikipedia.org"))
fetch_report(res)

# retry only what is worth retrying
again <- collect_content(res$domain_name[res$retryable])

## End(Not run)

Get Category from DMOZ

Description

Fetches category (or categories) of content hosted by a domain according to DMOZ. The function checks if path to the DMOZ file is provided by the user. If not, it looks for dmoz_domain_cateory.csv in the working directory. It also returns results for prominent subdomains.

Usage

dmoz_cat(domains = NULL, use_file = NULL)

Arguments

Details

DMOZ closed in March 2017 and the snapshot shipped here is from 2015, so these labels describe the domain as it was a decade ago. The returned source_last_published column carries that vintage, and a warning is issued once per session. See source_vintage.

Value

data.frame with the original list, the content category of the domain, and the vintage of the list that supplied it

See Also

source_vintage for the provenance of every category source

Examples

## Not run: 
dmoz_cat(domains = "http://www.google.com")
dmoz_cat(domains = c("http://www.google.com", "http://plus.google.com"))

## End(Not run)

Reasons a domain can fail to get a category

Description

The closed set of 'error_code' values that [collect_content()] and [pie_cat()] can return, with the pipeline stage each belongs to and whether it is worth retrying.

Usage

fetch_error_codes()

Details

These strings are shared verbatim with the Python package 'piedomains', so results from the two can be aggregated together.

Value

A tibble with columns 'code', 'stage', 'retryable' and 'description'.

See Also

[fetch_report()] to summarise an actual run, [source_vintage()] for the provenance of the static label sources.

Examples

fetch_error_codes()

# which failures are worth trying again?
subset(fetch_error_codes(), retryable)

Summarise the outcome of a fetch or classification run

Description

A pure function over the returned tibble, deliberately **not** a stored summary. A snapshot goes stale the moment you filter the rows; recomputing means 'fetch_report(subset(x, retryable))' is always correct.

Usage

fetch_report(x)

Arguments

Value

A one-row tibble: 'n', 'ok', 'failed', 'retryable', and list-columns 'by_stage', 'by_reason' and 'missing' (the domains that produced nothing).

Examples

rows <- data.frame(
  domain_name = c("a.com", "b.com", "c.com"),
  status = c("ok", "failed", "failed"),
  stage = c("infer", "fetch", "process"),
  error_code = c(NA, "timeout", "thin_content"),
  retryable = c(FALSE, TRUE, FALSE)
)
fetch_report(rows)

Get DMOZ Data

Description

Downloads archived DMOZ (Open Directory Project) data. DMOZ was discontinued in March 2017. This function downloads our preserved copy of the final DMOZ dataset. For more details, check: https://github.com/themains/rdomains/tree/master/data-raw/dmoz/

Usage

get_dmoz_data(outdir = ".", overwrite = FALSE)

Arguments

References

https://archive.org/details/dmoz-rdf-20150327

Examples

## Not run: 
get_dmoz_data()

## End(Not run)

Get Shalla Data

Description

Shallalist service was discontinued in January 2022. This function downloads the last archived copy (from 1/14/22) that we have preserved on GitHub. The original service at shallalist.de is no longer available. Downloads, unzips and saves the final version of shallalist data. By default, saves shalla data as shalla_domain_category.csv.

Usage

get_shalla_data(outdir = "./", overwrite = FALSE)

Arguments

References

https://web.archive.org/web/20210502020725/http://www.shallalist.de/

Examples

## Not run: 
get_shalla_data()

## End(Not run)

Get Steven Black's Host List Data

Description

Downloads the latest version of Steven Black's unified hosts file. The hosts file contains domains known for serving ads, malware, and tracking.

Usage

get_stevenblack_data(outdir = "./", variant = "base", overwrite = FALSE)

Arguments

References

https://github.com/StevenBlack/hosts

Examples

## Not run: 
get_stevenblack_data()
get_stevenblack_data(variant = "all")

## End(Not run)

ML Model

Description

ML Model

Usage

glm_shalla

Format

A list

Author(s)

Gaurav Sood

Source

ML model based on shallalist using keywords and domain suffixes,

Extract text, title, description and language from HTML

Description

Extract text, title, description and language from HTML

Usage

html_text_content(html)

Arguments

Value

A list with 'text' (cleaned, lowercased), 'title', 'description' and 'lang'. Missing elements are 'NA_character_'; 'text' is '""' when nothing could be extracted.

See Also

[page_signals()], which uses this to decide whether a page is classifiable.

Examples

html <- "<html lang='en'><head><title>Example</title>
  <meta name='description' content='A demo page'>
  <script>ignored()</script></head>
  <body><p>Hello   World</p><style>p{}</style></body></html>"
html_text_content(html)

Classify News and Non-News Based on keywords in the URL

Description

Based on a slightly amended version of the regular expression used to classify news, and non-news in: “Exposure to ideologically diverse news and opinion on Facebook” by Bakshy, Messing, and Adamic. Science. 2015.

Usage

not_news(url_list = NULL)

Arguments

Details

Amendment: sport rather than sports

URL containing any of the following words is classified as soft news: "sport|entertainment|arts|fashion|style|lifestyle|leisure|celeb|movie|music|gossip|food|travel|horoscope|weather|gadget"

URL containing any of following words is classified as hard news: "politi|usnews|world|national|state|elect|vote|govern|campaign|war|polic|econ|unemploy|racis|energy|abortion|educa|healthcare|immigration"

Note that it is based on patterns existing in a small set of domains. See paper for details.

Value

data.frame with 3 columns: url, not_news, news

References

https://www.science.org/doi/10.1126/science.aaa1160

Examples

## Not run: 
not_news("http://www.bbc.com/sport")
not_news(c("http://www.bbc.com/sport", "http://www.washingtontimes.com/news/politics/"))

## End(Not run)

Get Category from OpenAI

Description

Fetches category of content hosted by a domain using OpenAI's chat completion API. The function uses GPT models to classify domains into specified categories.

Usage

openai_cat(
  domains = NULL,
  api_key = NULL,
  categories = NULL,
  model = "gpt-4o-mini",
  rate_limit = 0.5
)

Arguments

Value

data.frame with original list and content category of the domain

Examples

## Not run: 
openai_cat("google.com")
openai_cat(c("google.com", "facebook.com"))
openai_cat("google.com", categories = c("search", "social", "ecommerce", "news", "other"))

## End(Not run)

What kind of page is this?

Description

Inspects already-fetched HTML for the three things that are not a classifiable site: an anti-bot interstitial, a domain-parking placeholder, and a server's "nothing here" page. No network access.

Usage

page_signals(html, text = NULL, domain = "", status = NULL)

Arguments

Details

'parked' and 'unavailable' are **answers, not failures** – they are facts about the domain, plainly stated in the page, that a caller can act on. They are also free: no classification service needs to be consulted.

Value

A one-row tibble: 'page_state' (one of '"content"', '"blocked"', '"parked"', '"unavailable"', '"thin"'), 'blocked', 'block_vendor', 'block_reason', 'parked', 'unavailable', 'thin', 'n_tokens'.

See Also

[fetch_error_codes()] for how these map onto run outcomes.

Examples

# A Cloudflare challenge, not a website
page_signals("<html><title>Just a moment...</title><body>cf_chl_opt</body></html>")

# A parking page
page_signals("<html><body>This domain is for sale. Inquire now.</body></html>")

# A real page that merely embeds reCAPTCHA is not blocked
page_signals(paste0("<html><title>Reddit</title><body>",
                    paste(rep("real discussion content", 200), collapse = " "),
                    "<script src='recaptcha/api.js'></script></body></html>"))

Get Category from Shallalist

Description

Fetches category of content hosted by a domain according to Shalla. The function checks if path to the shalla file is provided by the user. If not, it looks for shalla_domain_category.csv in the working directory.

Usage

shalla_cat(domains = NULL, use_file = NULL)

Arguments

Details

Shallalist stopped publishing in 2022, so these labels describe the domain as it was then or earlier, not as it is now. The returned source_last_published column carries that vintage, and a warning is issued once per session. See source_vintage.

Value

data.frame with the original list, the content category of the domain, and the vintage of the list that supplied it

See Also

source_vintage for the provenance of every category source

Examples

## Not run: 
shalla_cat(domains = "http://www.google.com")

## End(Not run)

Report what each category source is and when it was last published

Description

Every lookup in this package answers from a static list, and those lists have very different vintages. Two of them are no longer published at all. Use this to see which answers can be trusted to describe a domain as it is today.

Usage

source_vintage(source = NULL)

Arguments

Value

A tibble with one row per source: key, source, last_published, status, successor and note.

Examples

source_vintage()
source_vintage("dmoz")

Get Category from Steven Black's Host List

Description

Classifies domains based on Steven Black's unified host list which blocks ads, malware, and tracking domains. The function checks if a domain appears in the blocklist and categorizes it accordingly.

Usage

stevenblack_cat(domain = NULL, use_file = NULL)

Arguments

Details

Steven Black's host list is a consolidated list from multiple sources including adaway.org, mvps.org, malwaredomainlist.com, and someonewhocares.org.

Unlike the Shallalist and DMOZ lookups, this list is actively maintained, so the returned source_last_published is the fetched file's own modification date rather than a constant. See source_vintage.

Value

data.frame with the original domain name, the category, and the date of the list that supplied it

References

https://github.com/StevenBlack/hosts

See Also

source_vintage for the provenance of every category source

Examples

## Not run: 
stevenblack_cat("doubleclick.net")
stevenblack_cat(c("google.com", "googleadservices.com", "malware-example.com"))

## End(Not run)

Get Category from University Domain List

Description

Fetches university domain json from: https://raw.githubusercontent.com/Hipo/university-domains-list/master/world_universities_and_domains.json

Usage

uni_cat(domains = NULL)

Arguments

Value

data.frame with original list and all the other columns from the university json

Examples

## Not run: 
uni_cat(domains = "http://www.google.com")

## End(Not run)

Get Category from VirusTotal

Description

Returns category of content from multiple security vendors using the VirusTotal API v3. The function retrieves domain analysis results including categories from various security services. Not all services will have categories for all domains.

Usage

virustotal_cat(domains = NULL, apikey = NULL)

Arguments

Details

Get the API Access Key from https://www.virustotal.com/. Either pass the API Key to the function or set the environmental variable: VirustotalToken. Environment variables persist within a R session.

Value

data.frame with domain and VirusTotal analysis results

References

https://docs.virustotal.com/reference/domains

Examples

## Not run: 
virustotal_cat("http://www.google.com")
virustotal_cat(c("google.com", "facebook.com"))

## End(Not run)