The idea is to keep your content off of "Hot" (the default view) by weighing down its vote before many people can see it. From there it will continue to sink deeper with the passage of time.

This is the main reason I don't contribute many original posts and spend all my time here commenting.

There is no way to know who is doing it exactly. I wish you could click the vote count and it would go to a breakdown showing how each account voted.