Unfortunately I have to agree with you. I think the problem is with people with great broadband and people with slower connections. I notice the game compensates for fastballs and curves differently when you're not evenly matched speedwise.
It's almost like blind swinging when you're at bat with someone who has a slower connection.
I'm still willing to play though. I just wish it was better at compensating for different speeds.