No, there is ALWAYS a victim, however, each case has different consequences.
For example, IF the cheating spouse brings home a STD that would cause the death of an unborn baby, the death of the spouse OR cause the spouse to have to live with disease that could end up costing him/her their life, they should be prosecuted. IF their adultery leads to death via a disease, they should be tried for attempted murder.
In most cases, neither spouse has agreed to put themselves at risk via their partner's adultery. They need to be held criminally accountable should they cause health issues or death.
In cases where no physical harm is caused, they should at the very least receive a substantial fine which would go to the victimized spouse (and any children from their union).