Welcome to Ruodu Wang's Homepage

Lhasa, China, 2019

Working Paper Series

Axiomatic theory of risk measures Risk sharing, allocation, and equilibria
Joint mixability Robust risk aggregation
Risk management with risk measures E-values: Theory and methods

E-values: Theory and methods

The series of working papers contains recent advances on the theory and methods of e-values. The recently introduced notion of e-values have many advantages over p-values. In particular, they are convenient for optional stopping in data collection and experiment design, arbitrary dependence in multiple testing, and high-dimensional composite hypotheses. E-values are intimately linked to martingales, likelihood ratios, and betting scores (Glenn Shafer); see details in WP01. This working paper series has an overlap with Hypothesis testing with e-values maintained by Vladimir Vovk. See also the work of Peter Grünwald and Aaditya Ramdas. Some basic terminology and their references are listed below.

  • The null (hypothesis): A collection of probability measures to test (WP01)
  • E-values: Realizations of e-variables (WP01)
  • E-variables: Non-negative random variables with mean no larger than 1 under the null (WP01)
  • E-processes: Stochastic processes yielding an e-variable at any stopping time under the null (WP06)
  • P-values: Realizations of p-variables (WP05)
  • P-variables: Random variables stochastically dominating a standard uniform under the null (WP05)
  • Merging methods: Methods to combine several p-variables or e-variables into one (WP01)
  • E-confidence intervals: Confidence intervals (CIs) for a parameter generated by a class of e-variables associated with the parameter (WP02)
  • False discovery rate: The expected ratio of #(false discoveries) to #(total discoveries) (WP06)
  • False coverage rate: The expected ratio of #(mis-coverages) to #(total selected CIs) (WP09)
  • A brief description of each working paper is provided below to explain its main results and the logical structure across papers.

    WP12 E-backtesting
    (by Qiuqi Wang, Ruodu Wang, Johanna Ziegel)

    One of the most challenging tasks in risk modeling practice is to backtest ES (see the working paper series on Axiomatic theory of risk measures for a background on risk measures) forecasts provided by financial institutions. We use e-values and e-processes to construct a model-free backtesting procedure for ES using a concept of universal e-statistics, which can be naturally generalized to many other risk measures and statistical quantities.

    WP11 Efficiency of nonparametric e-tests
    (by Vladimir Vovk, Ruodu Wang)

    In this short note, we introduce a simple analogue for e-values of Pitman’s asymptotic relative efficiency and apply it to three popular nonparametric tests.

    WP10 E-values as unnormalized weights in multiple testing
    (by Ruodu Wang, Aaditya Ramdas)

    We study procedures based on both e-values and p-values for each hypothesis to show the advantages of using e-values as random weights. We also collect several other results, such as a tiny but uniform improvement of e-BH, a soft-rank permutation e-value, and the use of e-values as masks in interactive multiple testing.

    WP09 Post-selection inference for e-value based confidence intervals
    (by Ziyu Xu, Ruodu Wang, Aaditya Ramdas)

    This is a follow-up of WP06. Inspired by the Benjamini-Yekutieli (BY) procedure for reporting confidence intervals, we design the e-BY procedure which uses e-CIs instead of p-CIs. The e-BY procedure has a desirable false coverage rate control under arbitrary dependence and stopping rules, in contrast to the BY procedure.

    WP08 A unified framework for bandit multiple testing
    (by Ziyu Xu, Ruodu Wang, Aaditya Ramdas)

    This is a follow-up of WP06. We apply e-values and the e-BH procedure to multi-armed bandit testing problems. We propose a unified, modular framework for FDR control that emphasizes the decoupling of exploration and summarization of evidence. We utilize the powerful e-processes to ensure FDR control for arbitrary composite nulls, exploration rules and stopping times in generic problem settings. The conference version is published in Advances in Neural Information Processing Systems (NeurIPS 2021).

    WP07 Testing with p*-values: Between p-values, mid p-values, and e-values
    (by Ruodu Wang)

    We introduce the notion of p*-values (p*-variables), which generalizes p-values (p-variables) in several senses. The new notion has four natural interpretations: operational, probabilistic, Bayesian, and frequentist. A main example of a p*-value is a mid p-value. A unified stochastic representation (Theorem 3.1) for p-values, mid p-values, and p*-values is obtained. The notion of p*-values becomes useful in many situations even if one is only interested in p-values, mid p-values, or e-value; it is connected to e-values in a simple way: 1/(2e) is a p*-value for an e-value e.

    WP06 False discovery rate control with e-values
    (by Ruodu Wang, Aaditya Ramdas)

    We design a natural analog of the Benjamini-Hochberg (BH) procedure for false discovery control (FDR) control that utilizes e-values (e-BH) and compare it with the standard procedure for p-values. The e-BH procedure includes the BH procedure as a special case through calibration between p-values and e-values. The journal version is published in Journal of the Royal Statistical Society Series B (2022).

    WP05 Admissible ways of merging p-values under arbitrary dependence
    (by Vladimir Vovk, Bin Wang, Ruodu Wang)

    This paper is a follow-up of WP12 of Joint mixability. We study the admissibility in Wald's sense of p-merging functions and their domination structure under arbitrary dependence. As a technical tool e-values become essential in the main results (Theorems 5.1 and 5.2). Another result (Theorem 6.2) showing the admissibility of some p-merging methods uses results on Joint mixability. The journal version is published in Annals of Statistics (2022).

    WP04 Merging sequential e-values via martingales
    (by Vladimir Vovk, Ruodu Wang)

    This is a follow-up of WP01. We study the problem of merging sequential or independent e-values into one e-value for statistical decision making. We describe a class of e-value merging functions via martingale merging functions, and show that all merging methods for sequential e-values are dominated by such a class. In case of merging independent e-values, the situation becomes much more sophisticated, and we provide a general class of such merging functions based on reordered test martingales.

    WP03 True and false discoveries with independent e-values
    (by Vladimir Vovk, Ruodu Wang)

    This is a follow-up of WP02. We use e-discovery matrices in the context of multiple hypothesis testing assuming that the base tests produce independent e-values, and compare those with procedures based on p-values.

    WP02 Confidence and discoveries with e-values
    (by Vladimir Vovk, Ruodu Wang)

    We discuss systematically two versions of confidence regions: those based on p-values and those based on e-values, the latter are e-confidence intervals (e-CIs). Both versions are applied to multiple hypothesis testing to construct procedures that control the number of false discoveries under arbitrary dependence between the base p- or e-values. We introduce the e-discovery matrices and show that it is efficient both computationally and statistically using simulated and real-world datasets.

    WP01 E-values: Calibration, combination, and applications
    (by Vladimir Vovk, Ruodu Wang)

    We introduce e-values in their pure form to the statistical community. In particular, we demonstrate that e-values are often mathematically more tractable and develop procedures using e-values for multiple testing of a single hypothesis and testing multiple hypotheses. Related concepts introduced in this paper are e-variables, sequential e-values, and e-merging functions. The journal version is published in Annals of Statistics (2021).