Infinite loop in `BN_mod_sqrt()` reachable when parsing certificates

The BN_mod_sqrt() function, which computes a modular square root, contains
a bug that can cause it to loop forever for non-prime moduli.

Internally this function is used when parsing certificates that contain
elliptic curve public keys in compressed form or explicit elliptic curve
parameters with a base point encoded in compressed form.

It is possible to trigger the infinite loop by crafting a certificate that
has invalid explicit curve parameters.

Since certificate parsing happens prior to verification of the certificate
signature, any process that parses an externally supplied certificate may thus
be subject to a denial of service attack. The infinite loop can also be
reached when parsing crafted private keys as they can contain explicit
elliptic curve parameters.

Thus vulnerable situations include:

• TLS clients consuming server certificates
• TLS servers consuming client certificates
• Hosting providers taking certificates or private keys from customers
• Certificate authorities parsing certification requests from subscribers
• Anything else which parses ASN.1 elliptic curve parameters

Also any other applications that use the BN_mod_sqrt() where the attacker
can control the parameter values are vulnerable to this DoS issue.