When negotiating a new SSH session with a remote server, one of libssh2's functions for doing the key exchange (kex_agree_methods()) was naively reading data from the incoming packet and using it without doing sufficient range checks. The SSH_MSG_KEXINIT packet arrives to libssh2 with a set of strings, sent as a series of LENGTH + DATA pairs. libssh2 would go through the list and read the LENGTH field, read the string following the LENGTH and then advance the pointer LENGTH bytes in memory and expect to find the next LENGTH + DATA pair there. Then move on until seven subsequent strings are taken care of. It would naively assume that the (unsigned 32 bit) LENGTH fields were valid.
This packet arrives in the negotiating phase so the remote server has not yet been deemed to be a known or trusted party.