If it's any consolation, really sorting it out does get very tough indeed. I spent long enough with it, both as an undergraduate and then teaching it myself, that I eventually got a pretty good grip. But if you ever come across anyone who's got a clear explanation of the practical consequences of gauge invariance of the magnetic vector potential then I'd be keen to hear it. (I reckoned the finals examiners simply wouldn't ask about that, and sure enough they didn't.)
The water's very deep here https://en.wikipedia.org/wiki/Magnetic_potential#Magnetic_vector_potential.
The staggering thing is that we now have a couple of very useful mathematical tools - vector calculus and tensor calculus - to make this more tractable. Neither of these had been invented when Maxwell was working. He had to grind through the whole thing longhand ! Genius.