There's something very off about this question.
In spherical coordinates,
x² + y² + z² = ρ²
so that
f(x, y, z) = 6 - (x² + y² + z²)
transforms to
g(ρ, θ, φ) = 6 - ρ²
When transforming to spherical coordinates, we also introduce the Jacobian determinant, so that
dV = dx dy dz = ρ² sin(φ) dρ dθ dφ
Since we integrate over a sphere with radius 4, the domain of integration is the set
E = {(ρ, θ, φ) : 0 ≤ ρ ≤ 4 and 0 ≤ θ ≤ 2π and 0 ≤ φ ≤ π}
so that the integral is
[tex]\displaystyle \int_{\phi=0}^{\phi=\pi} \int_{\theta=0}^{\theta=2\pi} \int_{\rho=0}^{\rho=4} (6 - \rho^2) \rho^2 \sin(\phi) \, d\rho \, d\theta \, d\phi[/tex]
Computing the integral is simple enough.
[tex]\displaystyle = \int_{\phi=0}^{\phi=\pi} \int_{\theta=0}^{\theta=2\pi} \int_{\rho=0}^{\rho=4} (6 \rho^2 - \rho^4) \sin(\phi) \, d\rho \, d\theta \, d\phi[/tex]
[tex]\displaystyle = 2\pi \int_{\phi=0}^{\phi=\pi} \int_{\rho=0}^{\rho=4} (6 \rho^2 - \rho^4) \sin(\phi) \, d\rho \, d\phi[/tex]
[tex]\displaystyle = 2\pi \left(\int_{\phi=0}^{\phi=\pi} \sin(\phi) \, d\phi\right) \left(\int_{\rho=0}^{\rho=4} (6 \rho^2 - \rho^4) \, d\rho\right)[/tex]
[tex]\displaystyle = 2\pi \cdot 2 \cdot \left(-\frac{384}5\right) = \boxed{-\frac{1536\pi}5}[/tex]
but the mass can't be negative...
Chances are good that this question was recycled without carefully changing all the parameters. Going through the same steps as above, the mass of a spherical body with radius R and mass density given by
[tex]\delta(x, y, z) = k - (x^2 + y^2 + z^2)[/tex]
for some positive number k is
[tex]\dfrac{4\pi r^3}{15} \left(5k - 3r^2\right)[/tex]
so in order for the mass to be positive, we must have
5k - 3r² ≥ 0 ⇒ k ≥ 3r²/5
In this case, k = 6 and r = 4, but 3•4²/5 = 9.6.
