Thinking you can even prevent all unkowns is a foolish endeavor. Only work on solving the problem at hand. Why would I think about some future requirement and lose sleep over it? That's the PM and sales team headache! Or think of it another way... you haven't yet reached the bridge yet but you're already thinking you need rope for the imagined bridge collapse.

All you can do with the present is make it relatively easier for your future self. Avoid complexity (thanks grug brained programmer!), don't tie yourself into knots and back yourself in corners (keep the code readable, testable and simple. This means minimal external dependencies, lose coupling but good coherence, and avoiding reinventing solution to difficult to solve problem)

some of these goals seem contradictory but you have to apply them to a specific problem and the objective of solving the problem (why are you solving this problem?). For instance, for crypto the advice of avoiding reinventing the solution takes precedence to minimal dependencies one because your objective is security which is important to get right without tolerance for bugs.

Aside from the commonly stated - experience:

The key point to keep in mind is that, at the end of the day, your building an application to satisfy a customer. Or to be exact, a list of requirements that may or may not be constantly changing. In this case from what i'm reading, I am assuming its your hobby project, so the 'customer' is you.

In this case, over-engineering is when you add more functionalities/services than what you needed to hit the bare minimum requirement. Ideally you want to hit basic requirements first, then start designing/engineering on top of what you have when your customer wants more features, etc.

Your design providing more features than the bare minimum should be an 'accident' more than intentional ideally, unless you think the extra feature takes 0% effort to implement. (Though TBH safe rule of thumb is, never design for additional unrequested features)

With the above context in mind, you should be asking the questions:

• How much downtime is acceptable? (Usually measured in per year)
• How much $ cost in SaaS/API services is acceptable? (per month, or year, etc...)
• How much time/money spent in maintenance is acceptable? (Helps in determining the API/Service you are going to use)
• Other questions related to acceptable risk/costs, etc...

Yea, welcome to client engineering. Usually its handled by senior developers or project managers, unless your in startup.

TLDR; It's not about finding a design that is perfect, its about finding a design that is acceptable by the involved. What is acceptable? that is for the involved parties to decide.

Every project should start by having well-defined specifications.

And should have some Quality&Testing involved (ideally independent) that will check that everything you have developed answers to the specifications. But nothing more.

If you go beyond the specs, it's overengineering.

ETA: remember that normally you're not trying to create something perfect, you're just answering the needs of a customer. You're paid (hopefully) to fulfil these needs and nothing more

The only way you know before doing it is through experience. If you don't already know you can make a prototype, or just guess and go for it - you might have to change it later though.

I would recommend prototyping when the cost of correcting a wrong decision later is going to be really high.

The other option if you don't have experience is just to copy what other people have done.