Exactly.
If charging was to improve you would need to upgrade your house to three phase at lots of ££. You can do that now and use a charger rated for more power for a quicker charge (e.g. the Tesla charger can be switched to 21 from 7 if you have 3 phase. It is supplied to be capable of both inputs).
If batteries got bigger you just charge longer. It’s the house supply that’s capping everything.