Much of what I am going to say is generalized and can change some depending on how you write the program however, programs do not work like that. You do not assign an amount of memory and all of it is used. Basically the program asks for memory as it needs it. For example when you create a variable of type integer (a number), you get an integer amount of memory assigned. Different memory types use different amounts of memory. As the game gets further along, more memory is needed because more data has been created and needs to be stored somewhere. But you cannot force it to use more memory, it uses what it needs similar to having an 8 core CPU running a single threaded app. Ignoring the operating system, only one thread and therefore only one core will be used even though 8 are available. Usually the problem is that you do not have as much as it needs and it is forced to use other resources like swap files which are much slower. Limits are based on operating system so another issue is that only so much data can be assigned to an app. For Windows 7 32 bit, a program can only have 2 GB of static, 2 GB of dynamic and 1 GB of stack memory. More than that cannot be accessed by the program even if the box has it. That is why having the minimal amount of RAM you need to run something is good but more than you need or can be used does not help you at all. I would think your 128GB would be rarely if ever used in entirety. And I would be somewhat surprised if your GPU was truly the limiting factor your observation of the RAM and GPU usage imply.
As to it slowing down, even a multi-threaded app using multiple cores can be slow. One thread may be waiting on the answer from another thread and that prevents the current thread from executing until the data is available. Multiple threads only do so much as they are still inherently linear as only one can be executed at any one time in a core. The main thing threads do is allow multiple things to be done in parallel (not simultaneously) allowing for a more responsive interface. However there is a cost to more threads as a context switch between them does take CPU time. And no matter how well you balance things, there is often a critical path that is the reason why things do not happen as fast as you would like. Everything else is fast but that critical path prevents you from getting the speed you want because everything else is waiting on it. And no matter what, the CPU has to be able to initiate everything, the GPU only does what it is told. It relieves the CPU by doing a lot of the work for graphics by doing them in parallel but the GPU can only do what the CPU tells it to.
I play the game with an I3-6100, 16GB RAM and an RX470 graphics card. I see the same delays you do in late large games but it is not significant and seemingly confined to ship movement for the most part. I would almost like to see a setting to turn off the ship movement and have the ship just leave one place and arrive at the end briefly showing any enemy ships it sees along the way but not showing the movement itself. I doubt this would speed things up much but seeing the ships hesitate is the annoying thing to me.