From my experience, it's the inability to read the cues, especially body language, of autistic people.
Picture it from the other side for a moment. Neurotypical people constantly show some kind of body language. And they often don't even know it - it's instinctual for them. It shows the outside world whether they're happy, sad, think of themselves as better or worse than others, how much energy they have, etc. They also read everyone else's body language, and use it to interpret whether the people may pose a threat to them. This guy is relaxed, on his phone and has tuned out the outside world - not a threat. This guy is happy and quite joyful - not a threat. This guy is agressive and has his nose in the air - stay the fuck away. And again, all of this happens subconsciously. So then suddenly, there's a person who's body language your subconscious can't interpret. Your subconscious has no clue if this person is a threat or not, and that makes it uncomfortable. If in doubt, your subconscious says "rather safe than sorry". And if your conscious side doesn't know how autistic people can express themselves, you don't have anything to override it.
A theater teacher I had once explained how to make your body language 100% neutral. He then said: "this is not something you ever do in daily life. Normally, you're always communicating something through your body language, and others pick up on it subconsciously. If you want to have some fun, go to a crowded subway platform and do that neutral stance. It won't take long before people start taking a distance, it freaks people the fuck out."
It's possible to learn most of the body language of autistic people, but it's not something that comes naturally to most. I've learned it myself, since I grew up with a number of autistic relatives (although sometimes I suspect I may be ever so slightly on the spectrum, but I've never bothered to look into it too deeply, so maybe take me "learning" it with a grain of salt), and if I pay attention I can notice the moment where my brain switches how it interprets cues from other people.