Answer:
(3a - 5b) * (7a - 8b)
Step-by-step explanation:
By observing the expression we note that the coefficient for either a or b must be negative, since the middle term has a negative coefficient. Further since the squared terms have positive coefficients we know that the coefficients for both a-terms must have the same sign, and the same goes for the b-terms.
The a²-term has a coefficient of 21 which has only four possible integer factorizations, either 1*21, 3*7, (-1)*(-27) or (-3)*(-7). The b²-term however has more possible integer factorizations, 1*40, 2*20, 4*10, 5*8 and the respective pairs for negative numbers.
From here I would just try the possible combinations which would lead me to find that (3a - 5b) * (7a - 8b) = 21a² - 24ab - 35ab + 40b² = 21a² -59ab + 40b²
I'll be honest, I'm not at all happy with this solution/explanation as it is quite 'hand-wavey'. I'm trying to come up with some reason as to why this expression must have an integer-coefficients factorization other than it's a school assignment which usually has fairly 'nice' solutions. In real life that's not often the case.
