Unrelated patches to openssl-1.1
warnings.patches

The attachment warnings.patches​ fixes few warning which I found as part of work to port certmonger to openssl-1.1.

There is still one automake warning left:

configure.ac:8: warning: The 'AM_PROG_MKDIR_P' macro is deprecated, and its use is discouraged.
configure.ac:8: You should use the Autoconf-provided 'AC_PROG_MKDIR_P' macro instead,
configure.ac:8: and use '$(MKDIR_P)' instead of '$(mkdir_p)'in your Makefile.am files.

http://stackoverflow.com/questions/29862780/why-does-gettext-automake-macros-include-deprecated-am-prog-mkdir-p-macro

If you want to fix it then you need to increase version in AM_GNU_GETTEXT_VERSION to 0.18.3 (0.19) or decrease to 0.14.4.

Patches which works well with openssl-1.0.1+
openssl_good.patches

The attachment openssl_good.patches​ fixes many problems with openssl-1.1 and works well with openssl-1.0.1 (tested on debian stable) and openssl-1.0.2 (tested on fedora 25). All unit tests passed.

patches which broke unit tests on openssl 1.0.2
openssl_bad.patches

The attachment openssl_bad.patches​ fixes some compilation problems with openssl-1.1 but it breaks unitests on older versions of openssl. Some of them are known problems e.g. e22b6c5. And rest of patches was just my bad attempt to fix compilation issues. But maybe it will help to find right solution.

Even with bad patches applied there are still some parts which need to be fixed for openssl-1.1:

../../certmonger/src/pkcs7.c: In function ‘get_pstring_attribute’:
../../certmonger/src/pkcs7.c:859:8: error: dereferencing pointer to incomplete type ‘X509_ATTRIBUTE {aka struct x509_attributes_st}’
   if (a->single) {
        ^~

Problems with ->cert_info->subjectUID and ->cert_info->issuerUID

../../certmonger/src/submit-o.c: In function ‘cm_submit_o_sign’:
../../certmonger/src/submit-o.c:115:14: error: dereferencing pointer to incomplete type ‘X509 {aka struct x509_st}’
       (*cert)->cert_info->subjectUID = ASN1_BIT_STRING_new();
              ^~

and also problem with ->cert_info->extensions in src/submit-o.c

Replace deprecated RAND_pseudo_bytes
openssl_pseudo.patch

Additional fixes for OpenSSL-1.1.x API (should be also working with 1.0.x)
remaining_openssl.patch

openssl_pseudo.patch​ works well for me on openssl-1.0.1+

remaining_openssl.patch​ works as well openssl-1.0.1+. But I would split it into more patches. But rcrit
or nalin can have different preference.

But there is still a problem with accessing subjectUID and issuerUID

sh$ git grep -- "->cert_info->"
src/submit-o.c:     (*cert)->cert_info->subjectUID = ASN1_BIT_STRING_new();
src/submit-o.c:     if ((*cert)->cert_info->subjectUID != NULL) {
src/submit-o.c:             ASN1_BIT_STRING_set((*cert)->cert_info->subjectUID, uuid, 16);
src/submit-o.c:             if (signer->cert_info->subjectUID != NULL) {
src/submit-o.c:                     (*cert)->cert_info->issuerUID = ASN1_STRING_dup((const ASN1_STRING *)signer->cert_info->subjectUID);
src/submit-o.c:             (*cert)->cert_info->issuerUID = ASN1_BIT_STRING_new();
src/submit-o.c:             if ((*cert)->cert_info->issuerUID != NULL) {
src/submit-o.c:                     ASN1_BIT_STRING_set((*cert)->cert_info->issuerUID, uuid, 16);

I found a getter X509_get0_uids but I have no idea how to set them.

I see a rebuild apparently against openSSL 1.1 went through Rawhide today:
https://koji.fedoraproject.org/koji/buildinfo?buildID=862860

so is this now resolved?

I hit a failure in a self-test in a Debian jessie container after the F26 build went through, but it may turn out to just be a problem with my setup. I think F26 should be fine.