Journal of Data Science

Distribution shift between training and deployment is a pervasive challenge for modern AI systems. In many cases, the target marginals of covariates and response are known or specified through population-level observations, boundary conditions, properties of simulator configurations, or alignment-time distributional constraints. Such knowledge may provide valuable side information for regression estimation. We study this problem in the multivariate linear regression setting with a stable conditional mean $\mathbb{E}[Y\mid X]$ across source and target, and identify the hybrid-loss estimator, which jointly incorporates both target marginals, as a benchmark target-aware estimator. Its direct computation, however, requires solving a coupled nonlinear optimization that is expensive at scale. Our main contribution is to develop and evaluate two computationally tractable alternatives: a constrained moment-matching estimator and a two-stage estimator that augments ordinary least squares with a calibration step. For all three estimators, we derive and compare closed-form asymptotic mean squared errors, yielding conditions under which the tractable alternatives match or closely approximate the hybrid benchmark, and regimes in which they do not. Monte Carlo experiments across three controlled shift regimes validate the theoretical results, investigate the accuracy–runtime tradeoffs among the three estimators, and translate into guidance on estimator choice. In particular, the two-stage estimator nearly matches the hybrid benchmark in the high signal-to-noise regime at essentially no additional cost, providing theoretical grounding for empirical observations in nonlinear settings.

Density deconvolution is a fundamental problem in measurement error settings, arising when the goal is to recover the distribution of unobserved latent variables based on their error-contaminated proxies. An important application arises in nutritional epidemiology, where the objective is to estimate the long-term average intake of nutritional components using 24-hour dietary recall data. This task involves complex multivariate models that account for conditionally heteroscedastic measurement errors, zero-inflated proxies for episodically consumed dietary components, diverse marginal distribution shapes across components, etc. While flexible Bayesian hierarchical methods, coupled with Markov chain Monte Carlo techniques, have been increasingly successful in recent years to enable deconvolution under such intricate but realistic scenarios, the development of rapid and user-friendly software implementations has lagged behind. The $\mathsf{R}$ package BayesDecon has been developed to address this challenge, providing a fast and accessible implementation of flexible Bayesian deconvolution methods for practitioners working with measurement errors. In the process, we have introduced substantial improvements to several aspects of the original models and algorithms, and have also added options to fit simpler parametric models. Although illustrated on challenges in nutritional epidemiology, the package addresses general deconvolution problems and hence is broadly applicable to other domains as well.

Feature selection methods in Rough Set Theory and Logical Combinatorial Pattern Recognition commonly rely on simplified binary discernibility matrices to represent the condition attributes that distinguish objects from different decision classes. However, the original discernibility matrix involves a large number of redundant object comparisons, which severely limits computational efficiency and scalability. Although several algorithms have been proposed to compute the simplified matrix, most of them first construct the full discernibility matrix and then simplify it, making these approaches impractical for large-scale datasets. In this paper, we introduce ISBMA, an incremental algorithm based on a deterministic finite automaton that directly constructs the simplified binary discernibility and discernibility/similarity matrices without explicitly generating the full matrix. Experimental comparisons with the fastest reported method demonstrate that ISBMA significantly reduces redundant computations, requires substantially less memory, and achieves superior runtime performance, remaining feasible on large decision tables where existing approaches fail.

Addressing short-term recidivism presents a significant challenge within the criminal justice system, necessitating robust predictive models that can accurately identify risk factors and predict outcomes. This paper introduces application of Bayesian cure rate models to predict short-term recidivism using data from individuals released from Iowa prisons in 2018. The mixture cure rate models effectively capture the recidivism risk by distinguishing between cured individuals, those unlikely to re-offend, and those susceptible to relapse. By implementing accelerated failure time models combined with logistic regression and various link functions, including logit, skewed logit, reversed power logit, and flexible generalized logit, we enhance the model’s flexibility and interpretability. Through a comparative analysis of these models, key predictors of recidivism are identified and discussed in detail.

Pediatric vaccine efficacy studies face enrollment challenges, often resulting in underpowered studies to confirm vaccine efficacy. To address the challenge, this paper proposes a novel Two-Tier Bayesian dynamic borrowing framework that selectively incorporates two heterogeneous sources of historical data of adult populations into pediatric studies. The method employs similarity metrics based on intermediate outcomes, such as immunological markers, to dynamically determine borrowing weights via overlap coefficients, ensuring that only comparable data are utilized. Power priors are constructed with weights reflecting this similarity, while a cap on the effective sample size helps maintain control of the maximum information to borrow. Simulation studies demonstrate that the proposed approach improves power, reduces estimation bias and maintains type I error control, thus offering a robust and practical strategy for pediatric vaccine study design.

Traditionally z-scores specified from the WHO population growth curves have been used to describe a child’s growth in relation to his age- and sex-matched population distribution. We propose a new regression approach that offers a straightforward interpretation of the relative growth in terms of the original anthropometric variable. We create a hybrid data set consisting of the observations from the study of interest and counterpart pseudo-population observations imputed from the WHO population growth curves matched to each study participant. We then fit linear and quantile regression models to the hybrid data incorporating demographic variables (usually age and biological sex) corresponding to the growth curves of demographically-similar individuals, a study versus population indicator, and its interactions with demographic variables. We further control for confounding variables from the study by adding their interactions with the study indicator variable. The interaction terms between the study indicator and the demographic variables age and biologic sex can be interpreted as relative growth parameters that depict the differences in means (or quantiles) between the study participants and their pseudo-population counterparts of the original anthropometric variables, rather than the associated z-scores. We use anthropometric growth data from a prospective birth cohort study conducted in Uganda for illustration.

The self-organizing map (SOM) is an unsupervised, competitive learning neural network that projects high-dimensional data onto a low-dimensional grid, effectively showcasing the topological relationships within the original dataset. However, the conventional SOM training algorithm is restricted to numeric data. Categorical data typically needs to be converted into binary format before SOM training, which can lead to the loss of crucial similarity information between categorical values. As a result, the trained SOM may not accurately reflect the true topological order. While a training data splitting method (TDSM) can help identify perfect representative neurons and enhance clustering outcomes, the training data itself often lacks sufficient information, such as data distribution, and can be uncertain and ambiguous. Even when perfect neurons are identified, further improvements in clustering results become challenging. This paper investigates the possibility of improving the performance of supervised TDSM SOM clustering by utilizing unsupervised self-organization granule encoding for discrete data. This approach to unsupervised learning is advantageous for uncovering uncertain and ambiguous information within discrete data, leading to a more effective topological representation of the training data.

Joint models can describe the relationship between recurrent and terminal events. Typically, recurrent events are modeled using the total time scale, assuming constant covariate effects on each recurrent event. However, modeling the gap time between recurrent events could allow varying covariate effects and offer greater flexibility and accuracy. For instance, in HIV-infected patients, the intervals between the first occurrence of opportunistic infections (OIs) may follow a different distribution compared to later OIs. However, limited research has focused on mediation analysis using joint modeling of gap times and survival time. In this work, we propose a novel joint modeling approach that studies the mediation effect of recurrent events on survival outcomes by modeling the recurrent events by gap time. This allows us to handle cases where the first occurrence of a recurrent event behaves differently from subsequent events. Additionally, we use a relaxed “sequential ignorability” assumption to address unmeasured confounding. Simulation studies demonstrate that our model performs well in estimating both model parameters and mediation effects. We apply our method to an AIDS study to evaluate the comparative effectiveness of two treatments and the effect of baseline CD4 counts on overall survival, mediated by recurrent opportunistic infections modeled through gap times.

Forecasting business cycles and macroeconomic trends is inherently challenging due to their complex and non-linear relationships with volatile and noisy economic factors. However, the growing availability of large-scale economic data, coupled with advances in computational power, creates new opportunities to extract meaningful information and develop robust predictive models. Deep learning methods are well suited for handling noisy and complex data; nevertheless, their application to business cycle prediction remains at an early stage. To address this gap, this study develops an end-to-end computational framework that implements state-of-art deep learning architectures for identifying business cycle phases. The key contributions of this work include systematic input feature selection across a broad range of economic sectors, advanced preprocessing techniques for noisy data, the development of a customizable and reproducible computational framework, a data-driven approach to hyperparameter tuning, and the use of robust model selection strategies. Twenty two models based on two architectures—Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU)—were implemented and demonstrated strong predictive performance, highlighting the effectiveness of the proposed approach for forecasting macroeconomic trends. Among these, the GRU model with 512 neurons achieved the best results, with an accuracy of 86.42%, precision of 92.03%, recall of 93.32%, and an F1-score of 92.34%. Overall, the findings provide valuable insights that can support informed decision-making by policymakers and other stakeholders.

In data analysis, unexpected results often prompt researchers to revisit their procedures to identify potential issues. While some researchers may struggle to identify the root causes, experienced researchers can often quickly diagnose problems by checking a few key assumptions. These checked assumptions, or expectations, are typically informal, difficult to trace, and rarely discussed in publications. In this paper, we introduce the term analysis validation checks to formalize and externalize these informal assumptions. We then introduce a procedure to identify a subset of checks that best predict the occurrence of unexpected outcomes, based on simulations of the original data. The checks are evaluated in terms of accuracy, determined by binary classification metrics, and independence, which measures the shared information among checks. We demonstrate this approach with a toy example using step count data and a generalized linear model example examining the effect of particulate matter air pollution on daily mortality.

The receiver operating characteristics (ROC) curve has been widely used to evaluate the discrimination performance of biomarkers, but it has been criticized for overlooking their underlying distributions. In this paper, we propose a continuous version of the ROC curve that can assess not only the discrimination performance of biomarkers but also their continuity performance. Our method summarizes the biomarker values as conditional tail expectations at varying thresholds and compare them with true positive and false positive rates. The proposed method is particularly useful for an early phase of biomarker study that enrolls heterogeneous disease populations. Analysis of data from an ovarian cancer biomarker study illustrates the practical utility of the proposed method over the standard ROC curve analysis. The proposed methods are implemented in the R package varoc.