...
If you generate the private key on any other system, then that system must also be secure, and moreover. Indeed, every system the private key comes in contact with must be secure—at least as secure as the target system—or the private key must be encrypted at rest. Moreover, the private key must be encrypted while in transit to the secure system. That All in all, that is much more work (and error-prone), so the best advice is don't do it. Generate your private keys on the target IdP (or SP) in the first place.
That said, it is easy to generate a private key and a corresponding long-lived, self-signed certificate with OpenSSL:
...