Use a Small ship and see if XBox Fast Travel improves reliability it may also improve load times for some locations. For me the answer is a resounding YES. Try it for your self.
This is Personal view and experience. I have only tested for my ships in my play through. I can NOT Verify my claims, its been simple trial by error. I am a Xbox Player. I have played Starfield since preview on Xbox. This video is made on my PC and it is very clear I am not familiar with the PC keys. Game pass and cloud sync enabled this video via the cross play.
Now for the summary: On XBox players report errors, (I can personally verify), after many hours of game play, long sessions cause strange bugs. Fast travel will randomly fail, then get progressively worse. It is improved with a small ship.
BUT why? My personal view is all the decoration in all the habs you now have in your all in one battle ship. Here is my theory. The decorations are loose Items. They have mass, collision boxes, the exist in a 3D location and can be rotated via 3 rotational axis. They are effected differently depending on gravity. My hunch is they are ready to be seen the instant your ship exists. You can see into your ship via your cockpit from the out side. . . But the items can move. More landings = more movement, over time, many landings I am not sure how many of the 350 decoration items play well with each other.
I can only visualize the "Bethesda item jank" of a spinning plusie octopus in low grav swiping a few books for a crazy cascade of decorations creating a CPU hell of vectors, rotation and collisions, of every item in every hab on your favourite ship. Try a small ship and see if the Helps your Xbox experience