1 Theon realized his father despises him
When Theon went to meet his father he realized that he despises him because he doesn't live like real Ironborns. For example he bought his cloak instead of taking it from an enemy that he killed like Ironborns do.
2 Theon realized he won't be his father's Heir
Theon realized that his sister had a better chance of becoming his father's heir and so he wanted to do anything to be the rightful heir. See how Robert Baratheon became king.
3 He wanted his Father's Approval
Balon , Theon's father, didn't have any respect for him. Theon wanted to gain his father's respect and so he had to show him that he is on his side.
4 He felt like an outsider
Even though Theon lived among the Starks he felt like an outsider and didn't really believe he is one of them. See why the Lannisters hate the Starks.
5 He was convinced that he was a hostage
Theon was convinced that the Stark , who used to treat him well, were actually keeping him as a hostage. This gave him another reason to betray them.
6 The Starks and Ironborns had conflicting interests
Thoen believed he should be serving his family which had different interests than the Starks.
7 Theon had no real place among the Starks
Even though Theon was treated well he always felt he never had a place among the Starks and that he can't be the heir of any throne. See Why Jaime killed the mad King.
8 Theon's father told him that the Starks are enimies
Theon's father convinced him that the Starks are their enemies. He made his point stronger by convincing him that he was a hostage all that time.