It is not that math has been prior to the mind. It is exactly the other way around.
The sequential, serialized nature of perception gave rise the notion of ordering (sequential) - this then that, this before that. The mind, it's introspection and attention turned to the order of events within a perception gave us math. Not that the math existed somewhere gave us mind. The brain is a mechanical machine. As mechanical as Lego set.
No numbers exists outside the mind. There are only forms of energy - photons, fields we call atoms, and everything else built out of them.
Numbers, time, space are concepts of an observer. But there is no observer at the level of atoms or at the higher level of molecular biology. There is no math there either. Only "quantum" mechanics. Nu numbers, but physical shapes due to electro-chemical properties of proteins, which defines it's shapes.
Universe does not contain math. There is no one to observe that particles might be added or multiplied. Each one of them is independent and a part of the whole.
Math requres an observer.
And he was right - so called apriory knowledge do exist, but it is not math, but environmental conditioning encoded as a structure of specialized brain areas - it reflects so called apriory reality.
When people are studying philosophy but not physics they end up in the realms of pure ideas - disconnected from reality chimeras of abstract nonsense.
When physicists does not study philosophy they end up in the same Hegelian abstrac nonsense of disconnected from reality pure mathematical abstractions, such as higher dimensions, space-time, and other god-like fancies.