---
title: "Getting Started with smartcor"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Getting Started with smartcor}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```

## Why smartcor?

Researchers routinely compute correlations between variables, but the "right" correlation method depends on what kinds of variables are being compared.
Pearson correlation is the default in most software, yet it is only optimal for continuous-continuous pairs.
When variables are binary, ordinal, or categorical, specialised methods (point-biserial, phi, polychoric, polyserial, Cramer's V, Kendall's tau, and others) are more appropriate.

The scale of the problem is visible in published research.
For the accompanying paper (Harshvardhan and Ranjan, 2026), we traced correlation usage through the replication supplements of published economics articles: across 197 papers issuing at least one correlation command, 92.3% of the calls were Pearson, and 88% of the papers used Pearson and nothing else, whatever the variable types involved.
Recomputing 3,322 variable pairs from those supplements shows when that default is harmless and when it is not: the type-appropriate method diverges from the authors' choice in 8.1% of pairs where no variable is ordinal, but in every single one of the 732 pairs involving an ordinal variable.
Where the methods diverge the median change in the coefficient is small ($|\Delta r| \approx 0.017$), but the distribution has a long tail (reaching $0.47$), and in 36 pairs the divergence moves the result across the conventional $p < 0.05$ threshold.

**smartcor** automates this process: it detects variable types, selects the best method, and explains its reasoning.

## Installation

```{r install, eval = FALSE}
# from a local source:
install.packages("smartcor", repos = NULL, type = "source")

# or use devtools from the package directory:
devtools::install("path/to/smartcor")
```

## Quick Start

```{r library}
library(smartcor)
```

### Pairwise Correlation

The main function is `smart_cor()`.
Pass two vectors and it will figure out the rest:
```{r pairwise}
# continuous + continuous → Pearson
smart_cor(mtcars$mpg, mtcars$wt)
```

```{r pairwise-binary}
# continuous + binary → point-biserial (= Pearson)
smart_cor(mtcars$mpg, mtcars$vs)
```

### Ordinal Variables

When both variables have a small number of unique values, smartcor detects them as ordinal:

```{r ordinal}
# numeric variables with ≤10 unique values are treated as ordinal by default;
# assume_latent_normal controls whether polychoric or Kendall is used
smart_cor(mtcars$gear, mtcars$carb, assume_latent_normal = FALSE)
```

### Categorical Variables

```{r categorical}
# some categorical data
colour = factor(sample(c("red", "blue", "green"), 100, replace = TRUE))
shape  = factor(sample(c("circle", "square", "triangle"), 100, replace = TRUE))
smart_cor(colour, shape)
```

### Override Type Detection

If the automatic detection gets it wrong, you can override:

```{r override}
# force cyl to be treated as continuous
smart_cor(mtcars$mpg, mtcars$cyl, y_type = "continuous")
```

### Force a Specific Method

```{r force-method}
smart_cor(mtcars$mpg, mtcars$wt, method = "spearman")
```

## Correlation Matrix

`smart_cormat()` computes all pairwise correlations for a data frame, using the best method for each pair:

```{r cormat}
mat = smart_cormat(
  mtcars[, c("mpg", "cyl", "vs", "gear")],
  assume_latent_normal = FALSE
)
mat
```

## Tidy Output

Both `smart_cor()` and `smart_cormat()` support `tidy()` for integration with the tidyverse:

```{r tidy}
res = smart_cor(mtcars$mpg, mtcars$wt, verbose = FALSE)
tidy(res)
```

```{r tidy-mat}
tidy(mat)
```

## Available Methods

Use `available_methods()` to see what methods are available for a given variable-type pair:

```{r available}
available_methods("ordinal", "ordinal")
available_methods("continuous", "binary")
available_methods("categorical", "categorical")
```

## Method Selection Logic

The method selection is based on the following decision matrix, drawn from simulation studies comparing correlation methods (Harshvardhan and Ranjan, 2026):

| Variable types            | Default method           | Alternative                                        |
|:--------------------------|:-------------------------|:---------------------------------------------------|
| *Correlation methods (signed, from -1 to +1)* |      |                                                    |
| Continuous + continuous   | Pearson                  | Spearman (suggested, especially if nonlinear)      |
| Continuous + binary       | Pearson = point-biserial | ---                                                |
| Binary + binary           | Pearson = phi            | Tetrachoric (preferred, if latent normality holds) |
| Continuous + ordinal      | Spearman/Kendall         | Polyserial (preferred, if latent normality holds)  |
| Ordinal + ordinal         | Kendall's tau            | Polychoric (preferred, if latent normality holds)  |
| Binary + ordinal          | Rank-biserial            | Spearman                                           |
| *Association measures (unsigned, from 0 to 1)* |     |                                                    |
| Continuous + categorical  | Cramer's V (binned)      | ---                                                |
| Binary + categorical      | Cramer's V               | ---                                                |
| Ordinal + categorical     | Cramer's V               | ---                                                |
| Categorical + categorical | Cramer's V               | Theil's U, Tschuprow's T                           |

Count variables are treated as numeric continuous variables, so each count combination follows the corresponding continuous row.

Key findings from the simulation studies:

1. **Pearson = Point-Biserial = Phi** for 0/1-coded binary variables (exact mathematical equivalence).
2. **Polychoric/polyserial** are unbiased when the latent normality assumption holds, but can be biased when it is violated.
3. **Kendall's tau** is the most robust measure across different latent distributions, making it a safe default when normality is uncertain.
4. **More ordinal categories** = less attenuation, with the biggest improvement from 2 to 3 categories.

## The Latent-Normality Decision

When the decision matters (ordinal or binary variables), the default `assume_latent_normal = "auto"` runs a likelihood-ratio chi-square test on the pair via `test_bivariate_normality()` and lets the data decide:

```{r auto, eval = FALSE}
# runs the LR test and picks polychoric or Kendall accordingly:
smart_cor(mtcars$gear, mtcars$carb)
```

A binary-binary pair is the exception: its 2 x 2 table is saturated (df = 0), the test cannot run, so smartcor uses phi and leaves tetrachoric to an explicit `assume_latent_normal = TRUE` or `method = "tetrachoric"`.
Set `assume_latent_normal = TRUE` or `FALSE` to skip the test, or `NULL` to be asked interactively (non-interactive sessions then default to `TRUE`).

## Dependencies

The latent-variable methods rely on **polycor** (polychoric, polyserial, and tetrachoric estimates, with their standard errors).
It is listed under `Imports`, so it installs with smartcor.

When a latent-variable estimator fails numerically on a particular dataset, such as a singular Hessian or a sparse contingency table, smartcor does not simply error.
For the polyserial correlation it first retries the fit from a safe starting value, since polycor's default start for the correlation can land outside $(-1, 1)$ on strongly correlated pairs and abort the optimiser before it takes a step; this rescues most failures.
Only if no fit can be obtained does smartcor fall back to Kendall's tau (for polychoric) or Spearman (for polyserial); the returned `method`, `method_label`, and `rationale` then name the method actually used, so the substitution is visible in the output.