Modeling Spatially Correlated Failure-time Data Under Two Distance Functions with an Application to Titan GPU Data

Spatially correlated survival analysis has a substantial literature, including spatial frailty Cox models (Henderson, Shimakura, and Gorst 2002), CAR-based AFT models (Zhang and Lawson 2011), and general spatial survival frameworks for arbitrarily censored data (Zhou and Hanson 2018). Most methods, however, assume a single source of spatial dependence driven by physical distance. HPC systems such as Titan violate this simplification because geometry and network topology are both relevant, making dual-structure modeling necessary.

The core problem is to model censored failure times when dependence arises from two incompatible notions of proximity: cabinet layout in physical space and folded-torus connectivity in logical space. A valid model must estimate regression effects, capture both correlation sources, remain positive definite, and scale to tens of thousands of GPUs. The challenge is mathematical, statistical, and computational at once.

The first innovation is the two-random-effect AFT representation y=Xβ+Z_vv+Z_ww+ϵ, which isolates physical and logical dependence rather than merging them into one frailty term. The second is a torus-aware powered exponential correlation for the logical network, using min(|a-b|, n-|a-b|) distances in rows and columns. The third is the positive-definiteness proof via a Kronecker decomposition B⊗A, which turns a topology-specific construction into a valid covariance model for Bayesian inference.

• �� Data model: observed times t_i with censoring indicators δ_i; responses are analyzed on the log scale as y=log(\tilde t).
• �� Fixed effects: X includes an intercept plus cage, slot, and node indicators; Titan uses 12 non-baseline columns with cage 3, slot 8, and node 4 as references.
• �� Physical structure: cabinets are indexed on an 8-row × 25-column grid; physical distances use absolute row/column differences d^P_r, d^P_c.
• �� Logical structure: cabinets are relabeled according to folded-torus connectivity; logical distances use wrap-around minima d^L_r, d^L_c.
• �� Correlation: powered exponential kernels define ρ_P and ρ_L with shape parameters κ_v (0<κ_v≤2) and κ_w (0<κ_w≤1).
• �� Covariance assembly: Σ=Z_vΣ_vZ_v^T+Z_wΣ_wZ_w^T+Σ_ϵ, assuming v and w are independent.
• �� Inference: Bayesian estimation is recommended; Stan produces MCMC draws for parameters and uncertainty quantification.
• �� Theory: on the torus, the logical correlation matrix R_w is proven positive definite by sorting points, forming circle-wise matrices A and B, then using the eigenvalues of B⊗A.

The paper uses two experimental modes. First, simulation studies assess whether the dual-distance model can recover parameters and deliver accurate inference under controlled settings; the abstract states that the simulation demonstrates statistical accuracy. Second, the Titan GPU dataset provides a large real-world benchmark: more than 30,000 GPUs, arranged in an 8×25 cabinet grid, observed over 7 years, with 200 unique spatial locations. The model incorporates cabinet-level cage/slot/node indicators and is estimated with Stan-based MCMC. The excerpt does not report the full numerical tables, but it clearly frames both validation and application.

The strongest result is theoretical: the logical torus covariance is valid under 0<κ_w≤1, and the separable form R_w=B⊗A ensures positive definiteness by standard Kronecker eigenvalue arguments. Empirically, the simulation study confirms that the framework can perform accurate statistical inference for spatially correlated failure-time data. In the Titan application, the method is able to operate on a massive censored dataset with 30,000+ GPUs and 200 locations, demonstrating that the model is not only mathematically sound but also practically scalable.

Immediate use cases include supercomputer reliability analysis, thermal-risk mapping in datacenters, and failure prediction for any hardware system with both physical placement and communication topology. Practitioners need event times, censoring status, physical coordinates, and network layout. The output can guide replacement schedules, cooling design, rack placement, and topology-aware job scheduling.

A key limitation is the independence assumption between v and w; in real systems, thermal effects and network workload may interact. Another limitation is topology specificity: the torus-distance construction is tailored to folded-torus architectures and may need redesign for irregular or dynamically reconfigured networks. Finally, the excerpt lacks reported numerical coefficients and predictive benchmarks for Titan, making direct performance comparison impossible from the provided text alone.