I don't think part of what you say is true, deer are known to destroy crops

For one, I wasn't talking exclusively about deer. There are a ton of animals, including fish.

For two, deer are very good at surviving, but if we allowed unlimited hunting, they would go extinct very quickly. Or at least be uncommon enough and not grow mature before being shot. They may destroy crops, but a deer genocide is not the answer.

The balance may be gone in your opinion. And, if you're right, then hunting limitations should be lessened, but not done away with entirely.