tiny farm

I've fished about ten times and haven't caught anything. What am I doing wrong? 

I'm pressing X by the pond, then when it shows the exclamation point, I spam the X key. Then there's a little splash in the color of the exclamation point, but I still have no fish in my inventory. 

I've also been mining, but again, my inventory says I haven't gotten anything from it.