DuckDB HTS File Reader Extension for R

Description

The Rduckhts package provides an interface to the DuckDB HTS (High Throughput Sequencing) file reader extension from within R. It enables reading common bioinformatics file formats such as VCF/BCF, SAM/BAM/CRAM, FASTA, FASTQ, GFF, GTF, and tabix-indexed files directly from R using SQL queries via DuckDB.

Author(s)

DuckHTS Contributors

References

https://github.com/RGenomicsETL/duckhts

See Also

Useful links:

• https://github.com/RGenomicsETL/duckhts

• https://rgenomicsetl.r-universe.dev/Rduckhts

• Report bugs at https://github.com/RGenomicsETL/duckhts/issues

Detect Complex Types in DuckDB Table

Description

Identifies columns in a DuckDB table that contain complex types (ARRAY or MAP) that will be returned as R lists.

Usage

detect_complex_types(con, table_name)

Arguments

Value

A data frame with columns that have complex types, showing column_name, column_type, and a description of R type.

Examples

library(DBI)
library(duckdb)

con <- dbConnect(duckdb::duckdb(config = list(allow_unsigned_extensions = "true")))
rduckhts_load(con)
bcf_path <- system.file("extdata", "vcf_file.bcf", package = "Rduckhts")
rduckhts_bcf(con, "variants", bcf_path, overwrite = TRUE)
complex_cols <- detect_complex_types(con, "variants")
print(complex_cols)
dbDisconnect(con, shutdown = TRUE)

DuckDB to R Type Mappings

Description

The mapping covers the most common data types used in HTS file processing:

• BIGINT <-> double (not integer due to 64-bit overflow protection)

• DOUBLE <-> numeric/double

• VARCHAR <-> character/string

• BOOLEAN <-> logical

• ARRAY types (e.g., VARCHAR[], BIGINT[]) <-> list

• MAP types (e.g., MAP(VARCHAR, VARCHAR)) <-> data.frame

Important notes:

• 64-bit integers (BIGINT, UBIGINT) become double to prevent overflow

• DATE/TIME values return as Unix epoch numbers (double)

• MAP types become data frames with 'key' and 'value' columns

• ARRAY types become vectors (which are lists in R terminology)

Usage

duckdb_type_mappings()

Details

Returns a named list mapping between DuckDB and R data types. This is useful for understanding type conversions when reading HTS files or when specifying column types in tabix functions.

Value

A named list with two elements:

duckdb_to_r

Named character vector mapping DuckDB types to R types

r_to_duckdb

Named character vector mapping R types to DuckDB types

Examples

mappings <- duckdb_type_mappings()
mappings$duckdb_to_r["BIGINT"]
mappings$r_to_duckdb["integer"]

Append DuckDB extension metadata trailer to a shared library

Description

Append DuckDB extension metadata trailer to a shared library

Usage

duckhts_append_metadata(ext_file, verbose = FALSE)

Bootstrap the duckhts extension sources into the R package

Description

Copies extension source files from the parent duckhts repository into inst/duckhts_extension/ so the R package becomes self-contained. Run this before R CMD build to prepare the source tarball.

Usage

duckhts_bootstrap(repo_root = NULL)

Arguments

Value

Invisibly returns the destination directory.

Build the duckhts DuckDB extension

Description

Compiles htslib and the duckhts extension from the sources bundled in the installed R package. The built .duckdb_extension file is placed in the extension directory.

Usage

duckhts_build(build_dir = NULL, make = NULL, force = FALSE, verbose = TRUE)

Arguments

Value

Path to the built duckhts.duckdb_extension file.

Detect DuckDB platform string

Description

Detect DuckDB platform string

Usage

duckhts_detect_platform()

Load the duckhts extension into a DuckDB connection

Description

Load the duckhts extension into a DuckDB connection

Usage

duckhts_load(con = NULL, extension_path = NULL)

Arguments

Value

The DuckDB connection (invisibly).

Extract Array Elements Safely

Description

Helper function to safely extract elements from DuckDB arrays (returned as R lists) with proper error handling.

Usage

extract_array_element(array_col, index = NULL, default = NA)

Arguments

Value

The array element at the specified index, or full array if index is NULL

Examples

library(DBI)
library(duckdb)

con <- dbConnect(duckdb::duckdb(config = list(allow_unsigned_extensions = "true")))
rduckhts_load(con)
bcf_path <- system.file("extdata", "vcf_file.bcf", package = "Rduckhts")
rduckhts_bcf(con, "variants", bcf_path, overwrite = TRUE)
data <- dbGetQuery(con, "SELECT ALT FROM variants LIMIT 5")
first_alt <- extract_array_element(data$ALT, 1)
all_alts <- extract_array_element(data$ALT)
dbDisconnect(con, shutdown = TRUE)

Extract MAP Keys and Values

Description

Helper function to work with DuckDB MAP data (returned as data frames). Can extract keys, values, or search for specific key-value pairs.

Usage

extract_map_data(map_col, operation = "keys", default = NA)

Arguments

Value

Extracted data based on the operation

Examples

library(DBI)
library(duckdb)

con <- dbConnect(duckdb::duckdb(config = list(allow_unsigned_extensions = "true")))
rduckhts_load(con)
gff_path <- system.file("extdata", "gff_file.gff.gz", package = "Rduckhts")
rduckhts_gff(con, "annotations", gff_path, attributes_map = TRUE, overwrite = TRUE)
data <- dbGetQuery(con, "SELECT attributes FROM annotations LIMIT 5")
keys <- extract_map_data(data$attributes, "keys")
name_values <- extract_map_data(data$attributes, "Name")
dbDisconnect(con, shutdown = TRUE)

Normalize R Data Types to DuckDB Types for Tabix

Description

Normalizes R data type names to their corresponding DuckDB types for use with tabix readers. This function handles common R type name variations and maps them to appropriate DuckDB column types.

Usage

normalize_tabix_types(types)

Arguments

Details

The function performs the following normalizations:

• Integer types (integer, int, int32, int64) -> BIGINT

• Numeric types (numeric, double, float) -> DOUBLE

• Character types (character, string, chr) -> VARCHAR

• Logical types (logical, bool, boolean) -> BOOLEAN

• Other types -> Converted to uppercase as-is

If an empty vector is provided, it returns the empty vector unchanged.

Value

A character vector of normalized DuckDB type names suitable for tabix columns.

See Also

rduckhts_tabix for using normalized types with tabix readers, duckdb_type_mappings for the complete type mapping table.

Examples

normalize_tabix_types(c("integer", "character", "numeric"))
normalize_tabix_types(c("int", "string", "float"))

Create SAM/BAM/CRAM Table

Description

Creates a DuckDB table from SAM, BAM, or CRAM files using the DuckHTS extension.

Usage

rduckhts_bam(
  con,
  table_name,
  path,
  region = NULL,
  index_path = NULL,
  reference = NULL,
  standard_tags = FALSE,
  auxiliary_tags = FALSE,
  sequence_encoding = NULL,
  quality_representation = NULL,
  overwrite = FALSE
)

Arguments

Value

Invisible TRUE on success

Examples

library(DBI)
library(duckdb)

con <- dbConnect(duckdb::duckdb(config = list(allow_unsigned_extensions = "true")))
rduckhts_load(con)
bam_path <- system.file("extdata", "range.bam", package = "Rduckhts")
rduckhts_bam(con, "reads", bam_path, overwrite = TRUE)
dbGetQuery(con, "SELECT COUNT(*) FROM reads WHERE FLAG & 4 = 0")
dbDisconnect(con, shutdown = TRUE)

Build BAM or CRAM Index

Description

Builds a BAM or CRAM index using the DuckHTS extension.

Usage

rduckhts_bam_index(con, path, index_path = NULL, min_shift = 0, threads = 4)

Arguments

Value

A data frame with 'success', 'index_path', and 'index_format'

Create VCF/BCF Table

Description

Creates a DuckDB table from a VCF or BCF file using the DuckHTS extension. This follows the RBCFTools pattern of creating a table that can be queried.

Usage

rduckhts_bcf(
  con,
  table_name,
  path,
  region = NULL,
  index_path = NULL,
  tidy_format = FALSE,
  additional_csq_column_types = NULL,
  overwrite = FALSE
)

Arguments

Value

Invisible TRUE on success

Examples

library(DBI)
library(duckdb)

con <- dbConnect(duckdb::duckdb(config = list(allow_unsigned_extensions = "true")))
rduckhts_load(con)
bcf_path <- system.file("extdata", "vcf_file.bcf", package = "Rduckhts")
rduckhts_bcf(con, "variants", bcf_path, overwrite = TRUE)
dbGetQuery(con, "SELECT * FROM variants LIMIT 2")
dbDisconnect(con, shutdown = TRUE)

Build VCF or BCF Index

Description

Builds a TBI or CSI index for a VCF/BCF file using the DuckHTS extension.

Usage

rduckhts_bcf_index(con, path, index_path = NULL, min_shift = NULL, threads = 4)

Arguments

Value

A data frame with 'success', 'index_path', and 'index_format'

Create BED Table

Description

Creates a DuckDB table from a BED file using the DuckHTS extension.

Usage

rduckhts_bed(
  con,
  table_name,
  path,
  region = NULL,
  index_path = NULL,
  overwrite = FALSE
)

Arguments

Value

Invisible TRUE on success

BGZF Decompress a File

Description

Decompresses a BGZF file using the DuckHTS extension.

Usage

rduckhts_bgunzip(
  con,
  path,
  output_path = NULL,
  threads = 4,
  keep = TRUE,
  overwrite = FALSE
)

Arguments

Value

A data frame describing the created output file

BGZF Compress a File

Description

Compresses a plain file to BGZF using the DuckHTS extension.

Usage

rduckhts_bgzip(
  con,
  path,
  output_path = NULL,
  threads = 4,
  level = -1,
  keep = TRUE,
  overwrite = FALSE
)

Arguments

Value

A data frame describing the created BGZF file

Detect FASTQ Quality Encoding

Description

Inspects a FASTQ file's observed quality ASCII range and reports compatible legacy encodings with a heuristic guessed encoding.

Usage

rduckhts_detect_quality_encoding(con, path, max_records = 10000)

Arguments

Value

A data frame with the detected quality encoding summary

Create FASTA Table

Description

Creates a DuckDB table from FASTA files using the DuckHTS extension.

Usage

rduckhts_fasta(
  con,
  table_name,
  path,
  region = NULL,
  index_path = NULL,
  sequence_encoding = NULL,
  overwrite = FALSE
)

Arguments

Value

Invisible TRUE on success

Build FASTA Index

Description

Builds a FASTA index (.fai) using the DuckHTS extension.

Usage

rduckhts_fasta_index(con, path, index_path = NULL)

Arguments

Value

A data frame with columns 'success' and 'index_path'

Compute FASTA Interval Nucleotide Composition

Description

Computes bedtools nuc-style nucleotide composition over either a BED file or generated fixed-width bins.

Usage

rduckhts_fasta_nuc(
  con,
  path,
  bed_path = NULL,
  bin_width = NULL,
  region = NULL,
  index_path = NULL,
  bed_index_path = NULL,
  include_seq = FALSE
)

Arguments

Value

A data frame with interval composition statistics

Create FASTQ Table

Description

Creates a DuckDB table from FASTQ files using the DuckHTS extension.

Usage

rduckhts_fastq(
  con,
  table_name,
  path,
  mate_path = NULL,
  interleaved = FALSE,
  sequence_encoding = NULL,
  quality_representation = NULL,
  input_quality_encoding = NULL,
  overwrite = FALSE
)

Arguments

Value

Invisible TRUE on success

List DuckHTS Extension Functions

Description

Returns the package-bundled function catalog generated from the top-level functions.yaml manifest in the duckhts repository.

Usage

rduckhts_functions(category = NULL, kind = NULL)

Arguments

Value

A data frame describing the extension functions, including the DuckDB function name, kind, category, signature, return type, optional R helper wrapper, short description, and example SQL.

Examples

catalog <- rduckhts_functions()
subset(catalog, category == "Sequence UDFs", select = c("name", "description"))
subset(rduckhts_functions(kind = "table"), select = c("name", "r_wrapper"))

Create GFF3 Table

Description

Creates a DuckDB table from GFF3 files using the DuckHTS extension.

Usage

rduckhts_gff(
  con,
  table_name,
  path,
  region = NULL,
  index_path = NULL,
  header = NULL,
  header_names = NULL,
  auto_detect = NULL,
  column_types = NULL,
  attributes_map = FALSE,
  overwrite = FALSE
)

Arguments

Value

Invisible TRUE on success

Create GTF Table

Description

Creates a DuckDB table from GTF files using the DuckHTS extension.

Usage

rduckhts_gtf(
  con,
  table_name,
  path,
  region = NULL,
  index_path = NULL,
  header = NULL,
  header_names = NULL,
  auto_detect = NULL,
  column_types = NULL,
  attributes_map = FALSE,
  overwrite = FALSE
)

Arguments

Value

Invisible TRUE on success

Read HTS Header Metadata

Description

Reads file header records from HTS-supported formats using the DuckHTS extension.

Usage

rduckhts_hts_header(con, path, format = NULL, mode = NULL)

Arguments

Value

A data frame with parsed header metadata.

Read HTS Index Metadata

Description

Reads index metadata from HTS-supported index files via DuckHTS.

Usage

rduckhts_hts_index(con, path, format = NULL, index_path = NULL)

Arguments

Value

A data frame with index metadata.

Read Raw HTS Index Blob

Description

Returns raw index metadata blob data for a file index.

Usage

rduckhts_hts_index_raw(con, path, format = NULL, index_path = NULL)

Arguments

Value

A data frame with raw index blob metadata.

Read HTS Index Spans

Description

Returns index span-oriented metadata for planning range workloads.

Usage

rduckhts_hts_index_spans(con, path, format = NULL, index_path = NULL)

Arguments

Value

A data frame with span-oriented index metadata.

Lift Over Variant Coordinates Against a Query

Description

Applies the DuckHTS 'duckdb_liftover(...)' table macro to rows from a SQL query or table expression with chromosome and position columns, plus optional reference and alternate alleles.

Usage

rduckhts_liftover(
  con,
  query,
  chain_path,
  dst_fasta_ref,
  chrom_col = "chrom",
  pos_col = "pos",
  ref_col = NULL,
  alt_col = NULL,
  src_fasta_ref = NULL,
  max_snp_gap = 1,
  max_indel_inc = 250,
  lift_mt = FALSE,
  end_pos_col = NULL,
  no_left_align = FALSE
)

Arguments

Value

A data frame with source columns, lifted coordinates/alleles, and warnings.

Load DuckHTS Extension

Description

Loads the DuckHTS extension into a DuckDB connection. This must be called before using any of the HTS reader functions.

Usage

rduckhts_load(con, extension_path = NULL)

Arguments

Details

The DuckDB connection must be created with allow_unsigned_extensions = "true".

Value

TRUE if the extension was loaded successfully

Examples

library(DBI)
library(duckdb)

con <- dbConnect(duckdb::duckdb(config = list(allow_unsigned_extensions = "true")))
rduckhts_load(con)
dbDisconnect(con, shutdown = TRUE)

Munge Summary Statistics Rows

Description

Applies the DuckHTS 'duckdb_munge(...)' table macro to rows from a SQL query or table expression, using either an upstream-style preset, a named column map, or a two-column mapping file. When no mapping mode is provided, the bundled 'colheaders.tsv' alias file is used by default.

Usage

rduckhts_munge(
  con,
  query,
  fasta_ref = NULL,
  preset = NULL,
  column_map = NULL,
  column_map_file = NULL,
  iffy_tag = "IFFY",
  mismatch_tag = "REF_MISMATCH",
  ns = NULL,
  nc = NULL,
  ne = NULL
)

Arguments

Value

A data frame with normalized GWAS-VCF-style variant/effect columns.

Compute Polygenic Scores

Description

Calls the DuckHTS 'bcftools_score(...)' table function to compute sample-level polygenic scores from one genotype VCF/BCF file and one summary-statistics file.

Usage

rduckhts_score(
  con,
  bcf_path,
  summary_path,
  use = NULL,
  columns = "PLINK",
  columns_file = NULL,
  q_score_thr = NULL,
  use_variant_id = FALSE,
  counts = FALSE,
  samples = NULL,
  force_samples = FALSE,
  regions = NULL,
  regions_file = NULL,
  regions_overlap = 1,
  targets = NULL,
  targets_file = NULL,
  targets_overlap = 0,
  apply_filters = NULL,
  include = NULL,
  exclude = NULL
)

Arguments

Value

A data frame with one row per sample and score/count columns.

Create Tabix-Indexed File Table

Description

Creates a DuckDB table from any tabix-indexed file using the DuckHTS extension.

Usage

rduckhts_tabix(
  con,
  table_name,
  path,
  region = NULL,
  index_path = NULL,
  header = NULL,
  header_names = NULL,
  auto_detect = NULL,
  column_types = NULL,
  overwrite = FALSE
)

Arguments

Value

Invisible TRUE on success

Build Tabix Index

Description

Builds a tabix index for a BGZF-compressed text file using the DuckHTS extension.

Usage

rduckhts_tabix_index(
  con,
  path,
  preset = "vcf",
  index_path = NULL,
  min_shift = 0,
  threads = 4,
  seq_col = NULL,
  start_col = NULL,
  end_col = NULL,
  comment_char = NULL,
  skip_lines = NULL
)

Arguments

Value

A data frame with 'success', 'index_path', and 'index_format'

Setup HTSlib Environment

Description

Sets the 'HTS_PATH' environment variable to point to the bundled htslib plugins directory. This enables remote file access via libcurl plugins (e.g., s3://, gs://, http://) when plugins are available.

Usage

setup_hts_env(plugins_dir = NULL)

Arguments

Details

Call this before querying remote URLs to allow htslib to locate its plugins.

Value

Invisibly returns the previous value of 'HTS_PATH' (or 'NA' if unset).

Examples

## Not run: 
setup_hts_env()

plugins_path <- tempfile("hts_plugins_")
dir.create(plugins_path)
setup_hts_env(plugins_dir = plugins_path)
unlink(plugins_path, recursive = TRUE)

## End(Not run)