When there is a lot of users, ipa CLIs are delayed by initial kerberos lookup. Those lookup are searching real entries but need to the entire suffix. A consequence is that schema compat will evaluate the maps even searched entries are real (cn=kerberos or cn=accounts..).
Kerberos could use a new slapi-nis control (https://fedorahosted.org/freeipa/ticket/5597) to accelerate his searches
IPA cli can be delayed by kerberos lookup that occur between the connection establishment and the BIND The delay is also related to the number of users in the DB. In the following example this is a 3 sec delay while the DB has 50K users.
[11/Jan/2016:14:35:24 +0100] conn=86 fd=107 slot=107 connection from <client> to <server> ... [11/Jan/2016:14:35:27 +0100] conn=86 op=0 BIND dn="" method=sasl version=3 mech=GSSAPI
The kerberos lookup are
[11/Jan/2016:14:35:24 +0100] conn=4 op=367 SRCH base="<SUFFIX>" scope=2 filter="(&(|(objectClass=krbprincipalaux)(objectClass=krbprincipal)(objectClass=ipakrbprincipal))(|(ipaKrbPrincipalAlias=krbtgt/<realm>@<realm>)(krbPrincipalName=krbtgt/<realm>@<realm>)))" [11/Jan/2016:14:35:24 +0100] conn=4 op=367 RESULT err=0 tag=101 nentries=1 etime=0.648000 ... [11/Jan/2016:14:35:24 +0100] conn=4 op=369 SRCH base="<SUFFIX>" scope=2 filter="(&(|(objectClass=krbprincipalaux)(objectClass=krbprincipal)(objectClass=ipakrbprincipal))(|(ipaKrbPrincipalAlias=ldap/<host_fqdn>@<realm>)(krbPrincipalName=ldap/<host_fqdn>@<realm>)))" [11/Jan/2016:14:35:25 +0100] conn=4 op=369 RESULT err=0 tag=101 nentries=1 etime=0.646000 ... [11/Jan/2016:14:35:25 +0100] conn=4 op=371 SRCH base="<SUFFIX>" scope=2 filter="(&(|(objectClass=krbprincipalaux)(objectClass=krbprincipal))(krbPrincipalName=HTTP/<host_fqdn>@<realm>))" [11/Jan/2016:14:35:26 +0100] conn=4 op=371 RESULT err=0 tag=101 nentries=1 etime=0.530000 ... [11/Jan/2016:14:35:26 +0100] conn=4 op=373 SRCH base="<SUFFIX>" scope=2 filter="(&(objectClass=ipaKrb5DelegationACL)(memberPrincipal=HTTP/<host_fqdn>@<realm>))" [11/Jan/2016:14:35:26 +0100] conn=4 op=373 RESULT err=0 tag=101 nentries=1 etime=0.424000 ... [11/Jan/2016:14:35:26 +0100] conn=4 op=374 SRCH base="<SUFFIX>" scope=2 filter="(&(|(objectClass=krbprincipalaux)(objectClass=krbprincipal))(krbPrincipalName=admin@<realm>))" [11/Jan/2016:14:35:27 +0100] conn=4 op=374 RESULT err=0 tag=101 nentries=1 etime=0.551000 ... [11/Jan/2016:14:35:34 +0100] conn=4 op=378 SRCH base="<SUFFIX>" scope=2 filter="(&(|(objectClass=krbprincipalaux)(objectClass=krbprincipal)(objectClass=ipakrbprincipal))(|(ipaKrbPrincipalAlias=krbtgt/<realm>@<realm>)(krbPrincipalName=krbtgt/<realm>@<realm>)))" [11/Jan/2016:14:35:34 +0100] conn=4 op=378 RESULT err=0 tag=101 nentries=1 etime=0.675000 [11/Jan/2016:14:35:34 +0100] conn=4 op=379 SRCH base="<SUFFIX>" scope=2 filter="(&(|(objectClass=krbprincipalaux)(objectClass=krbprincipal)(objectClass=ipakrbprincipal))(|(ipaKrbPrincipalAlias=ldap/<host_fqdn>@<realm>)(krbPrincipalName=ldap/<host_fqdn>@<realm>)))" [11/Jan/2016:14:35:35 +0100] conn=4 op=379 RESULT err=0 tag=101 nentries=1 etime=0.636000
Most of the delay is spent in schema compat plugin. Those lookup returned one entry in the following subtrees:
time ldapsearch -LLL -D "cn=directory manager" -w xxx -b <SUFFIX> (&(|(objectClass=krbprincipalaux)(objectClass=krbprincipal)(objectClass=ipakrbprincipal))(|(ipaKrbPrincipalAlias=krbtgt/<realm>@<realm>)(krbPrincipalName=krbtgt/<realm>@<realm>))) dn dn: krbPrincipalName=krbtgt/<realm>@<realm>,cn=<realm>,cn=kerberos,<SUFFIX> real 0m0.707s user 0m0.007s sys 0m0.005s time ldapsearch -LLL -D "cn=directory manager" -w xxx -b <SUFFIX> (&(|(objectClass=krbprincipalaux)(objectClass=krbprincipal)(objectClass=ipakrbprincipal))(|(ipaKrbPrincipalAlias=ldap/<host_fqdn>@<realm>)(krbPrincipalName=ldap/<host_fqdn>@<realm>))) dn dn: krbprincipalname=ldap/<host_fqdn>@<realm>,cn=services,cn=accounts,<SUFFIX> real 0m0.708s user 0m0.010s sys 0m0.003s time ldapsearch -LLL -D "cn=directory manager" -w xxx -b <SUFFIX> (&(|(objectClass=krbprincipalaux)(objectClass=krbprincipal))(krbPrincipalName=HTTP/<host_fqdn>@<realm>)) dn dn: krbprincipalname=HTTP/<host_fqdn>@<realm>,cn=services,cn=accounts,<SUFFIX> real 0m0.583s user 0m0.012s sys 0m0.001s time ldapsearch -LLL -D "cn=directory manager" -w xxx -b <SUFFIX> (&(objectClass=ipaKrb5DelegationACL)(memberPrincipal=HTTP/<host_fqdn>@<realm>)) dn dn: cn=ipa-http-delegation,cn=s4u2proxy,cn=etc,<SUFFIX> real 0m0.474s user 0m0.010s sys 0m0.002s time ldapsearch -LLL -D "cn=directory manager" -w xxx -b <SUFFIX> (&(|(objectClass=krbprincipalaux)(objectClass=krbprincipal))(krbPrincipalName=admin@<realm>)) dn dn: uid=admin,cn=users,cn=accounts,<SUFFIX> real 0m0.562s user 0m0.008s sys 0m0.004s time ldapsearch -LLL -D "cn=directory manager" -w xxx -b <SUFFIX> (&(|(objectClass=krbprincipalaux)(objectClass=krbprincipal)(objectClass=ipakrbprincipal))(|(ipaKrbPrincipalAlias=krbtgt/<realm>@<realm>)(krbPrincipalName=krbtgt/<realm>@<realm>))) dn dn: krbPrincipalName=krbtgt/<realm>@<realm>,cn=<realm>,cn=kerberos,<SUFFIX> real 0m0.697s user 0m0.011s sys 0m0.002s time ldapsearch -LLL -D "cn=directory manager" -w xxx -b <SUFFIX> (&(|(objectClass=krbprincipalaux)(objectClass=krbprincipal)(objectClass=ipakrbprincipal))(|(ipaKrbPrincipalAlias=ldap/<host_fqdn>@<realm>)(krbPrincipalName=ldap/<host_fqdn>@<realm>))) dn dn: krbprincipalname=ldap/<host_fqdn>@<realm>,cn=services,cn=accounts,<SUFFIX> real 0m0.695s user 0m0.009s sys 0m0.003s
The impact of Schema compat plugin would be minimal if we can control Schema compat so that it returns immediately (using slapi-nis control https://fedorahosted.org/freeipa/ticket/5597). In the following example, scoping the search out of schema compat scope shows the cost of the search without schema compat impact.
time ldapsearch -LLL -D "cn=directory manager" -w xxx -b cn=kerberos,<SUFFIX> (&(|(objectClass=krbprincipalaux)(objectClass=krbprincipal)(objectClass=ipakrbprincipal))(|(ipaKrbPrincipalAlias=krbtgt/<realm>@<realm>)(krbPrincipalName=krbtgt/<realm>@<realm>))) dn dn: krbPrincipalName=krbtgt/<realm>@<realm>,cn=<realm>,cn=kerberos,<SUFFIX> real 0m0.021s user 0m0.007s sys 0m0.004s time ldapsearch -LLL -D "cn=directory manager" -w xxx -b cn=accounts,<SUFFIX> (&(|(objectClass=krbprincipalaux)(objectClass=krbprincipal)(objectClass=ipakrbprincipal))(|(ipaKrbPrincipalAlias=ldap/<host_fqdn>@<realm>)(krbPrincipalName=ldap/<host_fqdn>@<realm>))) dn dn: krbprincipalname=ldap/<host_fqdn>@<realm>,cn=services,cn=accounts,<SUFFIX> real 0m0.019s user 0m0.008s sys 0m0.002s time ldapsearch -LLL -D "cn=directory manager" -w xxx -b cn=accounts,<SUFFIX> (&(|(objectClass=krbprincipalaux)(objectClass=krbprincipal))(krbPrincipalName=HTTP/<host_fqdn>@<realm>)) dn dn: krbprincipalname=HTTP/<host_fqdn>@<realm>,cn=services,cn=accounts,<SUFFIX> real 0m0.019s user 0m0.008s sys 0m0.002s time ldapsearch -LLL -D "cn=directory manager" -w xxx -b cn=etc,<SUFFIX> (&(objectClass=ipaKrb5DelegationACL)(memberPrincipal=HTTP/<host_fqdn>@<realm>)) dn dn: cn=ipa-http-delegation,cn=s4u2proxy,cn=etc,<SUFFIX> real 0m0.017s user 0m0.008s sys 0m0.002s time ldapsearch -LLL -D "cn=directory manager" -w xxx -b cn=accounts,<SUFFIX> (&(|(objectClass=krbprincipalaux)(objectClass=krbprincipal))(krbPrincipalName=admin@<realm>)) dn dn: uid=admin,cn=users,cn=accounts,<SUFFIX> real 0m0.019s user 0m0.010s sys 0m0.001s time ldapsearch -LLL -D "cn=directory manager" -w xxx -b cn=kerberos,<SUFFIX> (&(|(objectClass=krbprincipalaux)(objectClass=krbprincipal)(objectClass=ipakrbprincipal))(|(ipaKrbPrincipalAlias=krbtgt/<realm>@<realm>)(krbPrincipalName=krbtgt/<realm>@<realm>))) dn dn: krbPrincipalName=krbtgt/<realm>@<realm>,cn=<realm>,cn=kerberos,<SUFFIX> real 0m0.019s user 0m0.008s sys 0m0.002s time ldapsearch -LLL -D "cn=directory manager" -w xxx -b cn=accounts,<SUFFIX> (&(|(objectClass=krbprincipalaux)(objectClass=krbprincipal)(objectClass=ipakrbprincipal))(|(ipaKrbPrincipalAlias=ldap/<host_fqdn>@<realm>)(krbPrincipalName=ldap/<host_fqdn>@<realm>))) dn dn: krbprincipalname=ldap/<host_fqdn>@<realm>,cn=services,cn=accounts,<SUFFIX> real 0m0.019s user 0m0.009s sys 0m0.001s
comments from triage, #5597 was also releated:
[thierry] The goal is to avoid slapi-nis evaluation cost when it is not needed. The filter does no prevent the evaluation, only changing base search can avoid slapi-nis. [ab] new slapi-nis version implements a simple filter that avoids answering for internal searches and searches over ldapi+DM. None of the existing uses need cn=compat data this way. Speedup is quite noticeable. ab: we have changes for existing slapi-nis code without breaking use cases ab: in 7.3, we will switch to different model and this will no longer be a problem. ab: we can move the ticket to slapi-nis, that slapi-nis should improve performance of a map cache, make it more generic "Be faster in processing map cache" simo: client should request principals outside of slapi-nis tree, in the suffix 01/26 pv: same as 5597 ab: another option was to have KDB driver to support search over several subtrees which explicitly exclude cn=compat,$SUFFIX. ab: nobody did the work yet
4.4 priority
This ticket is blocked by #5597 that was closed as invalid. In fact the kerberos perf issue (during a user-add), was fixed in #5448. With #5448 requests coming through ldapi on behalf of 'Directory manager' (assuming krb) are ignored by slapi-nis.
Metadata Update from @tbordaz: - Issue assigned to mbasti - Issue set to the milestone: FreeIPA 4.3.2
Login to comment on this ticket.