Novel Markov Chain Monte Carlo (MCMC) methods have enabled the generation of large ensembles of redistricting plans modeled as a graph partitioning problem. However, existing algorithms such as Reversible Recombination (RevReCom) and Metropolized Forest Recombination (MFR) have strong preferences for distributions related to the spanning tree measure. In this paper we introduce the Marked Edge Walk (MEW), a novel Markov chain proposal for sampling from the space of graph partitions. The walk operates on the space of spanning trees with marked edges, allowing for calculable transition probabilities for use in the Metropolis-Hastings algorithm. Empirical results on real-world dual graphs show convergence under a broad class of target distributions less constrained by spanning tree counts, including policy-based distributions, such as competitiveness on New Hampshire that are independent of spanning trees, and compactness and partisan symmetry distributions on New Hampshire and Texas that, while related to spanning trees, can now be properly targeted with a smaller degree of spanning tree bias, which represents an advancement in flexible ensemble generation.
Bates et al. (2015) described the evaluation of the profiled log-likelihood of a linear mixed-effects model by updating a sparse, symmetric positive-definite matrix and computing its Cholesky factor, as implemented in the lme4 package for R. Here we present enhancements to the derivation and theoretical presentation of the result and to its implementation using a blocked Cholesky factorization in the MixedModels.jl package for Julia (Bezanson et al., 2017). The gain in computational efficiency is primarily due to three factors: (1) the new derivation allows us to compute the penalized sum of squared residuals without computing the conditional estimates of the fixed-effects parameters and the conditional modes of the random effects at each optimization step, (2) the blocked Cholesky representation and careful ordering of the random effects terms reduces the amount of “fill-in” that occurs during the Cholesky factorization, and (3) the multiple dispatch feature of the Julia language allows us to use specialized algorithms for different kinds of matrices instead of relying on generic algorithms during the Cholesky factorization. To show the effectiveness of the blocked Cholesky approach we use it to fit a linear mixed model to over 32 million ratings of movies in the MovieLens ml-32m (Harper and Konstan, 2016) data set. The model incorporates random effects for over 200,000 movies and over 80,000 participants. Further enhancements to these computational methods are suggested.
Distribution shift between training and deployment is a pervasive challenge for modern AI systems. In many cases, the target marginals of covariates and response are known or specified through population-level observations, boundary conditions, properties of simulator configurations, or alignment-time distributional constraints. Such knowledge may provide valuable side information for regression estimation. We study this problem in the multivariate linear regression setting with a stable conditional mean $\mathbb{E}[Y\mid X]$ across source and target, and identify the hybrid-loss estimator, which jointly incorporates both target marginals, as a benchmark target-aware estimator. Its direct computation, however, requires solving a coupled nonlinear optimization that is expensive at scale. Our main contribution is to develop and evaluate two computationally tractable alternatives: a constrained moment-matching estimator and a two-stage estimator that augments ordinary least squares with a calibration step. For all three estimators, we derive and compare closed-form asymptotic mean squared errors, yielding conditions under which the tractable alternatives match or closely approximate the hybrid benchmark, and regimes in which they do not. Monte Carlo experiments across three controlled shift regimes validate the theoretical results, investigate the accuracy–runtime tradeoffs among the three estimators, and translate into guidance on estimator choice. In particular, the two-stage estimator nearly matches the hybrid benchmark in the high signal-to-noise regime at essentially no additional cost, providing theoretical grounding for empirical observations in nonlinear settings.
Density deconvolution is a fundamental problem in measurement error settings, arising when the goal is to recover the distribution of unobserved latent variables based on their error-contaminated proxies. An important application arises in nutritional epidemiology, where the objective is to estimate the long-term average intake of nutritional components using 24-hour dietary recall data. This task involves complex multivariate models that account for conditionally heteroscedastic measurement errors, zero-inflated proxies for episodically consumed dietary components, diverse marginal distribution shapes across components, etc. While flexible Bayesian hierarchical methods, coupled with Markov chain Monte Carlo techniques, have been increasingly successful in recent years to enable deconvolution under such intricate but realistic scenarios, the development of rapid and user-friendly software implementations has lagged behind. The $\mathsf{R}$ package BayesDecon has been developed to address this challenge, providing a fast and accessible implementation of flexible Bayesian deconvolution methods for practitioners working with measurement errors. In the process, we have introduced substantial improvements to several aspects of the original models and algorithms, and have also added options to fit simpler parametric models. Although illustrated on challenges in nutritional epidemiology, the package addresses general deconvolution problems and hence is broadly applicable to other domains as well.
Feature selection methods in Rough Set Theory and Logical Combinatorial Pattern Recognition commonly rely on simplified binary discernibility matrices to represent the condition attributes that distinguish objects from different decision classes. However, the original discernibility matrix involves a large number of redundant object comparisons, which severely limits computational efficiency and scalability. Although several algorithms have been proposed to compute the simplified matrix, most of them first construct the full discernibility matrix and then simplify it, making these approaches impractical for large-scale datasets. In this paper, we introduce ISBMA, an incremental algorithm based on a deterministic finite automaton that directly constructs the simplified binary discernibility and discernibility/similarity matrices without explicitly generating the full matrix. Experimental comparisons with the fastest reported method demonstrate that ISBMA significantly reduces redundant computations, requires substantially less memory, and achieves superior runtime performance, remaining feasible on large decision tables where existing approaches fail.
Addressing short-term recidivism presents a significant challenge within the criminal justice system, necessitating robust predictive models that can accurately identify risk factors and predict outcomes. This paper introduces application of Bayesian cure rate models to predict short-term recidivism using data from individuals released from Iowa prisons in 2018. The mixture cure rate models effectively capture the recidivism risk by distinguishing between cured individuals, those unlikely to re-offend, and those susceptible to relapse. By implementing accelerated failure time models combined with logistic regression and various link functions, including logit, skewed logit, reversed power logit, and flexible generalized logit, we enhance the model’s flexibility and interpretability. Through a comparative analysis of these models, key predictors of recidivism are identified and discussed in detail.
Pediatric vaccine efficacy studies face enrollment challenges, often resulting in underpowered studies to confirm vaccine efficacy. To address the challenge, this paper proposes a novel Two-Tier Bayesian dynamic borrowing framework that selectively incorporates two heterogeneous sources of historical data of adult populations into pediatric studies. The method employs similarity metrics based on intermediate outcomes, such as immunological markers, to dynamically determine borrowing weights via overlap coefficients, ensuring that only comparable data are utilized. Power priors are constructed with weights reflecting this similarity, while a cap on the effective sample size helps maintain control of the maximum information to borrow. Simulation studies demonstrate that the proposed approach improves power, reduces estimation bias and maintains type I error control, thus offering a robust and practical strategy for pediatric vaccine study design.
Traditionally z-scores specified from the WHO population growth curves have been used to describe a child’s growth in relation to his age- and sex-matched population distribution. We propose a new regression approach that offers a straightforward interpretation of the relative growth in terms of the original anthropometric variable. We create a hybrid data set consisting of the observations from the study of interest and counterpart pseudo-population observations imputed from the WHO population growth curves matched to each study participant. We then fit linear and quantile regression models to the hybrid data incorporating demographic variables (usually age and biological sex) corresponding to the growth curves of demographically-similar individuals, a study versus population indicator, and its interactions with demographic variables. We further control for confounding variables from the study by adding their interactions with the study indicator variable. The interaction terms between the study indicator and the demographic variables age and biologic sex can be interpreted as relative growth parameters that depict the differences in means (or quantiles) between the study participants and their pseudo-population counterparts of the original anthropometric variables, rather than the associated z-scores. We use anthropometric growth data from a prospective birth cohort study conducted in Uganda for illustration.
The self-organizing map (SOM) is an unsupervised, competitive learning neural network that projects high-dimensional data onto a low-dimensional grid, effectively showcasing the topological relationships within the original dataset. However, the conventional SOM training algorithm is restricted to numeric data. Categorical data typically needs to be converted into binary format before SOM training, which can lead to the loss of crucial similarity information between categorical values. As a result, the trained SOM may not accurately reflect the true topological order. While a training data splitting method (TDSM) can help identify perfect representative neurons and enhance clustering outcomes, the training data itself often lacks sufficient information, such as data distribution, and can be uncertain and ambiguous. Even when perfect neurons are identified, further improvements in clustering results become challenging. This paper investigates the possibility of improving the performance of supervised TDSM SOM clustering by utilizing unsupervised self-organization granule encoding for discrete data. This approach to unsupervised learning is advantageous for uncovering uncertain and ambiguous information within discrete data, leading to a more effective topological representation of the training data.
Joint models can describe the relationship between recurrent and terminal events. Typically, recurrent events are modeled using the total time scale, assuming constant covariate effects on each recurrent event. However, modeling the gap time between recurrent events could allow varying covariate effects and offer greater flexibility and accuracy. For instance, in HIV-infected patients, the intervals between the first occurrence of opportunistic infections (OIs) may follow a different distribution compared to later OIs. However, limited research has focused on mediation analysis using joint modeling of gap times and survival time. In this work, we propose a novel joint modeling approach that studies the mediation effect of recurrent events on survival outcomes by modeling the recurrent events by gap time. This allows us to handle cases where the first occurrence of a recurrent event behaves differently from subsequent events. Additionally, we use a relaxed “sequential ignorability” assumption to address unmeasured confounding. Simulation studies demonstrate that our model performs well in estimating both model parameters and mediation effects. We apply our method to an AIDS study to evaluate the comparative effectiveness of two treatments and the effect of baseline CD4 counts on overall survival, mediated by recurrent opportunistic infections modeled through gap times.