If we use the substitution [tex]u = \tan x[/tex], then [tex]du = \sec^2 {x}\ dx[/tex]. If you try substituting just [tex]u[/tex] and [tex]du[/tex] into the integrand, though, you'll notice that there's a [tex]\sec^2x[/tex] left over that we have to deal with.
To get rid of this problem, use the identity [tex]\tan^2 x + 1 = \sec^2 x[/tex] and substitute in the left side of the identity for the extra [tex]\sec^2x[/tex], as shown:
[tex] \int\limits^{\pi/6}_0 {tan^2 x \ sec^4 x} \, dx [/tex]
[tex] \int\limits^{\pi/6}_0 {tan^2 x \ (tan^2 x + 1) \ sec^2 x} \, dx [/tex]
From there, we can substitute in [tex]u[/tex] and [tex]du[/tex], and then evaluate:
[tex] \int\limits^{\pi/6}_0 {tan^2 x \ (tan^2 x + 1) \ sec^2 x} \, dx [/tex]
[tex] \int\limits^{\frac{1}{\sqrt{3}}}_0 {u^2(u^2 + 1)} \, du [/tex]
[tex] \int\limits^{\frac{1}{\sqrt{3}}}_0 {u^4 + u^2} \, du [/tex]
[tex]= \left.\frac{u^5}{5} + \frac{u^3}{3}\right|_0^\frac{1}{\sqrt{3}}[/tex]
[tex]= (\frac{(\frac{1}{\sqrt{3}})^5}{5} + \frac{(\frac{1}{\sqrt{3}})^3}{3}) - (\frac{(0)^5}{5} + \frac{(0)^3}{3})[/tex]
[tex]= \frac{1}{45\sqrt{3}} + \frac{1}{9\sqrt{3}} = \frac{6}{45\sqrt{3}} = \bf \frac{2}{15\sqrt{3}}[/tex]
