'Quadrare' ('to make square') and its various forms and cognates constitute the normal technical terminology for this sort of mathemtical relations in classical Latin, as at Pliny Sr., Hist. Nat., 36.13.19, 'pes quadrata' ('a square foot' or at Gellius, Noct. Att., 1.20.4, who speaks of a 'numerus quadratus' ('square number').

If you are seeking the substantive form of this concept rather than the verbal phrase which you're requesting, it will be 'circuli quadratura' ('a squaring of a circle'), found at Apuleius, 'de dogmate Platonis', 3.