Those who vote agree to a political contest where the vote determines the outcome. Those who do not vote are saying that it doesn't matter who enters office but rather that whoever is in office is accountable to his oath. Those who voted for or against the one who made it into office have no way to hold the incumbents feet to the fire. They have agreed to a political contract. Those who have no voter registration have agreed to no such political contract and might be able to free themselves from mischief of the majority.
And THAT is the difference.