Correctly estimating future failures in aging ship structures is a significant challenge. This manuscript explores a model updating approach based on fusing different visually observable physical records of structural degradation: permanent set of plating and fatigue crack initiation. A probabilistic S-N fatigue model is coupled to a permanent set model via a Bayesian network. Observations of permanent set and fatigue cracks are entered as evidence into this network. Through Bayesian inference, the network updates underlying loading and fatigue capacity models. These updated models are then used to forecast future failures. The proposed model is tested against Monte Carlo simulated service history data on five vessels from a larger fleet. The fusion of permanent set and fatigue together produces a more accurate estimate of future failures than using either failure mode alone. The benefit of fusing multiple visually observable measurements to update underlying structural models to provide fleet-wide prognosis appears promising, with further improvements possible with additional refinement.