The major consideration is you are essentially trading raw power for a cleaner, less messy implementation of drivers in your system. You'll be getting a weaker card, but your main benefits have to do with not dealing with Nvidia's approach to Linux.
The technical understanding at the low level is beyond my current knowledge of how drivers are integrated into Distros as a whole, but from my research this is the overall gist:
- Nvidia does not like having the community take charge of their drivers. When you install an Nvidia GPU on your system, you are installing a proprietary driver blob made by their company that cannot be easily modified or improved by anyone outside. This can work (on my own system that relies on CUDA for certain workloads, it's serviceable), but overall performance and efficiency gets left on the table because of it. There are community projects to create a full open source driver through reverse engineering (Nouveau), but it is very primitive and quite awful for gaming.
- AMD, on the other hand, embraced allowing the community to do what they wish with drivers they released. It has gotten to the point that they have far greater efficiency than their Nvidia counterparts on the platform (And probably was a decisive factor in Valve including their APUs on the Steam Deck). Because of the lack of red tape on who can touch them, most Distros include their drivers by default, and don't have any extra hassle (don't even need to touch the driver menu).
I'm not sure if you're currently using the Nouveau drivers because you weren't aware of this (explaining your extremely low FPS), but if installing the third party drivers doesn't work, feel free to grab an AMD (or Intel, since things are getting good on their side) card for a more "It just works" experience.
And if you do, feel free to have some catharsis looking at this famous video of Torvalds bashing Nvidia's stubbornness: https://youtu.be/iYWzMvlj2RQ