Let the circumference of the circle be x. Then the perimeter
of the triangle is (5-x).
The diameter of the circle is x/π, and its radius is x/(2π). Thus, its area
is:
Ac = π(x/2π)² = x²/(4π)
One side of the triangle is (5-x)/3. Its height is √3/2 times as side. Thus,
its area is:
At = (1/2)(5 - x)(√3/2)(7 - x)/3
At = (√3/12)(25- 10x + x²)
Now, to minimize total area, take the derivative and set to 0:
A = Ac + At
A' = x/(2π) + (√3/12)(2x - 10)
A' = x/(2π) + x√3/6 - 5√3/6
0 = x(√3/6 + 1/(2π)) - 5√3/6
x = (5√3/6) / (√3/6 + 1/(2π))
x ≈ 5.16
That makes the length of the triangle piece 5-x = 2.84
To maximize area, it must be at one of the endpoints, x=0 or x=5.
x=0: A = (1/2)(5)(√3/2)(5)/3 ≈ 3.61
x=5: A = 5²/(4π) ≈ 1.99
