They distract us
When you are so busy doing something and something in the external environment distracts you then you are very likely to feel annoyed. Flies provide constant distraction as they keep coming back every now and then.
The buzzing sound is annoying
According to a Quora thread, the sound a fly makes is inconsistent and has a high pitch. The annoying buzzing sound makes the fly even more annoying to us.
The rise in pitch as it approaches your ear
Flies like to fly near people and to land on them. As the fly approaches your ears, the pitch of their Buzz will become high all of sudden and so you will feel annoyed.
Persisting over and over
We humans get annoyed by repetitive annoying behavior. The fly doesn’t just do its annoying behavior once but it keeps repeating it over and over. As the fly keeps landing on us and buzzing around us over and over, we feel annoyed.
They cause stress
According to psychologists, stress happens when a person is focused on something yet another thing distracts him. If you are doing your work and a fly keeps buzzing around you then you are very likely to get stressed. (See What are the causes of stress?)
Flies like to stand on people
Flies like to land on people. When a fly lands on a person the unpleasant feeling, in addition to the sensation that something is moving on his skin, can make him feel annoyed. (See Why do flies sit on humans?)
They land on your food
The fact that you sometimes have to constantly watch your food to prevent a fly from landing on it can be very annoying and irritating.
They don’t care about your attempts to kill them
Flies become more annoying when a person tries to kill them more than once yet they keep coming back. The fact that they are indifferent to these attempts makes them more annoying.
People who are concerned about health and hygiene are more likely to find flies annoying since they can transfer diseases.
One of the reasons people find insects disgusting is their unpredictability. A fly can fly into any direction all of a sudden, thus making itself unpredictable just like many other insects.