windows - Restrict WinRM communication to specific server

I am working in an environment where we use WinRM to execute scripts on remote servers. All of this works fine but I am running into security concerns. What I would like to achieve is that execution of commands on the remote server through WinRM is limited to only a single server (two actually, since it's in a DR setup). I've looked at the trustedhosts parameter but that only seems to allow authentication if no credentials are given. In my setup we will be using https. I've also seen the ability to restrict the addresses the service listens to. I'm pretty sure this can be achieved but my Google-fu is failing me. So the question is, how can I restrict WinRM access to a single server?