Abstract: Estimating the fiber orientation density function (fODF) is fundamental for resolving crossing fibers in the human brain white matter and for performing fiber tractography. Conventional spherical deconvolution methods based on least-squares fitting and Gaussian noise assumptions are inadequate for handling Rician noise in magnitude MRI images, especially at low signal-to-noise ratio (SNR), where they tend to introduce biases; moreover, the fODF must satisfy a nonnegative constraint. In this paper, we represent the fODF using high-order tensors, and combine sum-of-squares (SOS) polynomials with semidefinite programming (SDP) to convert the nonnegative constraint into a convex constraint. Meanwhile, we establish a Rician maximum likelihood estimation model. For the likelihood term that contains the modified Bessel function, we adopt the expectation-maximization (EM) algorithm to transform the original problem into a convex subproblem with semidefinite constraints in each iteration, and then introduce the alternating direction method of multipliers (ADMM) for efficient solution, thereby proposing the Rician high-order tensor spherical deconvolution (R-HOT-CSD) method. Experiments on synthetic data, phantom data, and in vivo human brain data demonstrate that the proposed method can effectively suppress spurious peaks and improve angular resolution and estimation accuracy under low-SNR conditions.

Key words: high-order tensor, fiber orientation density function, Rician noise, expectation-maximization algorithm, alternating direction method of multipliers