Elden Ring

Im currently playing through the game again and since I enjoy some pvp now and then I have taunters tongue active while going through some of the areas. Now I noticed that aside from the usual pranksters, some reds are summoned and immediately use the finger severer, usually before even seeing me, to leave. I'm not doing any intentional griefing. Im just going through the area normally and waiting around in some cool areas to fight in. Does anyone know the reason for this? Could they for some reason think that I'm doing some ill-legit business? Would like to hear your opinions on this...