This idiom or expression has to do with:

Staring in surprise.
Turning your head and giving a startled look, with usually only one eye being visible.

We were walking and talking when I made an unexpected remark. She turned around and gave me the fish eye, obviously surprised and not knowing why exactly I had said that.