freeipa

Clone
Source Code
GIT

#5072 ipa-replica-manage del hangs if domain level = 1
Closed: Fixed None Opened 8 years ago by pspacek.

Closed: Fixed
Reopen Issue

Version: 6960725

Steps to reproduce:

  • Install master M with domain level = 1 (vm-059 in the log below)
  • Add replica A to the master M (vm-090 in the log below)
  • Add replica B to the master M (vm-134 in the log below)

  • From replica A attempt to delete replica B:

    [root@A]# ipa-replica-manage del B
    Waiting for removal of replication agreements

At this point ipa-replica-manage hangs.

errors log

[18/Jun/2015:13:26:32 +0200] ipa-topology-plugin - ipa_topo_post_mod - segment to be modified does not exist
[18/Jun/2015:13:26:32 +0200] ipa-topology-plugin - segment to be deleted does not exist
[18/Jun/2015:13:26:32 +0200] ipa-topology-plugin - ipa_topo_post_mod - segment to be modified does not exist
[18/Jun/2015:13:26:32 +0200] ipa-topology-plugin - ipa_topo_post_mod - segment to be modified does not exist
[18/Jun/2015:13:26:32 +0200] ipa-topology-plugin - segment to be deleted does not exist
[18/Jun/2015:13:26:32 +0200] ipa-topology-plugin - ipa_topo_post_mod - segment to be modified does not exist
[18/Jun/2015:15:59:20 +0200] ipa-topology-plugin - ipa_topo_util_modify: failed to modify entry (cn=replica,cn=dc\3Dipa\2Cdc\3Dexample,cn=mapping tree,cn=config): error 16

access log

[18/Jun/2015:15:59:20 +0200] conn=59 fd=83 slot=83 SSL connection from 10.34.78.90 to 10.34.78.90
[18/Jun/2015:15:59:20 +0200] conn=59 TLS1.2 128-bit AES
[18/Jun/2015:15:59:20 +0200] conn=59 op=0 BIND dn="" method=sasl version=3 mech=GSSAPI
[18/Jun/2015:15:59:20 +0200] conn=59 op=1 BIND dn="" method=sasl version=3 mech=GSSAPI
[18/Jun/2015:15:59:20 +0200] conn=59 op=2 BIND dn="" method=sasl version=3 mech=GSSAPI
[18/Jun/2015:15:59:20 +0200] conn=59 op=1 RESULT err=14 tag=97 nentries=0 etime=0, SASL bind in progress
[18/Jun/2015:15:59:20 +0200] conn=59 op=3 SRCH base="dc=ipa,dc=example" scope=2 filter="(&(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff)(objectClass=nstombstone))" attrs="nsds50ruv"
[18/Jun/2015:15:59:20 +0200] conn=59 op=2 RESULT err=0 tag=97 nentries=0 etime=0 dn="uid=admin,cn=users,cn=accounts,dc=ipa,dc=example"
[18/Jun/2015:15:59:20 +0200] conn=59 op=3 RESULT err=0 tag=101 nentries=1 etime=0
[18/Jun/2015:15:59:20 +0200] conn=59 op=4 SRCH base="cn=schema" scope=0 filter="(objectClass=*)" attrs="attributeTypes objectClasses"
[18/Jun/2015:15:59:20 +0200] conn=59 op=0 RESULT err=14 tag=97 nentries=0 etime=0, SASL bind in progress
[18/Jun/2015:15:59:20 +0200] conn=59 op=4 RESULT err=0 tag=101 nentries=1 etime=0
[18/Jun/2015:15:59:20 +0200] conn=56 op=16 DEL dn="cn=vm-134.abc.idm.lab.eng.brq.redhat.com,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=example"
[18/Jun/2015:15:59:20 +0200] conn=56 op=16 RESULT err=66 tag=107 nentries=0 etime=0
[18/Jun/2015:15:59:20 +0200] conn=56 op=17 SRCH base="cn=ipaconfig,cn=etc,dc=ipa,dc=example" scope=0 filter="(objectClass=*)" attrs=ALL
[18/Jun/2015:15:59:20 +0200] conn=56 op=17 RESULT err=0 tag=101 nentries=1 etime=0
[18/Jun/2015:15:59:20 +0200] conn=56 op=18 SRCH base="cn=vm-134.abc.idm.lab.eng.brq.redhat.com,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=example" scope=1 filter="(objectClass=*)" attrs=""
[18/Jun/2015:15:59:20 +0200] conn=56 op=18 RESULT err=0 tag=101 nentries=9 etime=0 notes=U
[18/Jun/2015:15:59:20 +0200] conn=56 op=19 SRCH base="cn=ipaconfig,cn=etc,dc=ipa,dc=example" scope=0 filter="(objectClass=*)" attrs=ALL
[18/Jun/2015:15:59:20 +0200] conn=56 op=19 RESULT err=0 tag=101 nentries=1 etime=0
[18/Jun/2015:15:59:20 +0200] conn=56 op=20 SRCH base="cn=KDC,cn=vm-134.abc.idm.lab.eng.brq.redhat.com,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=example" scope=1 filter="(objectClass=*)" attrs=""
[18/Jun/2015:15:59:20 +0200] conn=56 op=20 RESULT err=0 tag=101 nentries=0 etime=0 notes=U
[18/Jun/2015:15:59:20 +0200] conn=56 op=21 DEL dn="cn=KDC,cn=vm-134.abc.idm.lab.eng.brq.redhat.com,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=example"
[18/Jun/2015:15:59:20 +0200] conn=56 op=22 SRCH base="cn=ipaconfig,cn=etc,dc=ipa,dc=example" scope=0 filter="(objectClass=*)" attrs=ALL
[18/Jun/2015:15:59:20 +0200] conn=56 op=22 RESULT err=0 tag=101 nentries=1 etime=0
[18/Jun/2015:15:59:20 +0200] conn=56 op=23 SRCH base="cn=KPASSWD,cn=vm-134.abc.idm.lab.eng.brq.redhat.com,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=example" scope=1 filter="(objectClass=*)" attrs=""
[18/Jun/2015:15:59:20 +0200] conn=56 op=23 RESULT err=0 tag=101 nentries=0 etime=0 notes=U
[18/Jun/2015:15:59:20 +0200] conn=56 op=24 DEL dn="cn=KPASSWD,cn=vm-134.abc.idm.lab.eng.brq.redhat.com,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=example"
[18/Jun/2015:15:59:20 +0200] conn=56 op=21 RESULT err=0 tag=107 nentries=0 etime=0 csn=5582eaac001500050000
[18/Jun/2015:15:59:20 +0200] conn=56 op=25 SRCH base="cn=ipaconfig,cn=etc,dc=ipa,dc=example" scope=0 filter="(objectClass=*)" attrs=ALL
[18/Jun/2015:15:59:20 +0200] conn=56 op=25 RESULT err=0 tag=101 nentries=1 etime=0
[18/Jun/2015:15:59:20 +0200] conn=56 op=26 SRCH base="cn=MEMCACHE,cn=vm-134.abc.idm.lab.eng.brq.redhat.com,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=example" scope=1 filter="(objectClass=*)" attrs=""
[18/Jun/2015:15:59:20 +0200] conn=56 op=26 RESULT err=0 tag=101 nentries=0 etime=0 notes=U
[18/Jun/2015:15:59:20 +0200] conn=56 op=27 DEL dn="cn=MEMCACHE,cn=vm-134.abc.idm.lab.eng.brq.redhat.com,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=example"
[18/Jun/2015:15:59:20 +0200] conn=56 op=24 RESULT err=0 tag=107 nentries=0 etime=0 csn=5582eaac001600050000
[18/Jun/2015:15:59:20 +0200] conn=56 op=28 SRCH base="cn=ipaconfig,cn=etc,dc=ipa,dc=example" scope=0 filter="(objectClass=*)" attrs=ALL
[18/Jun/2015:15:59:20 +0200] conn=56 op=28 RESULT err=0 tag=101 nentries=1 etime=0
[18/Jun/2015:15:59:20 +0200] conn=56 op=29 SRCH base="cn=HTTP,cn=vm-134.abc.idm.lab.eng.brq.redhat.com,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=example" scope=1 filter="(objectClass=*)" attrs=""
[18/Jun/2015:15:59:20 +0200] conn=56 op=29 RESULT err=0 tag=101 nentries=0 etime=0 notes=U
[18/Jun/2015:15:59:20 +0200] conn=56 op=30 DEL dn="cn=HTTP,cn=vm-134.abc.idm.lab.eng.brq.redhat.com,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=example"
[18/Jun/2015:15:59:20 +0200] conn=56 op=27 RESULT err=0 tag=107 nentries=0 etime=0 csn=5582eaac001800050000
[18/Jun/2015:15:59:20 +0200] conn=56 op=31 SRCH base="cn=ipaconfig,cn=etc,dc=ipa,dc=example" scope=0 filter="(objectClass=*)" attrs=ALL
[18/Jun/2015:15:59:20 +0200] conn=56 op=31 RESULT err=0 tag=101 nentries=1 etime=0
[18/Jun/2015:15:59:20 +0200] conn=56 op=32 SRCH base="cn=OTPD,cn=vm-134.abc.idm.lab.eng.brq.redhat.com,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=example" scope=1 filter="(objectClass=*)" attrs=""
[18/Jun/2015:15:59:20 +0200] conn=56 op=32 RESULT err=0 tag=101 nentries=0 etime=0 notes=U
[18/Jun/2015:15:59:20 +0200] conn=56 op=33 DEL dn="cn=OTPD,cn=vm-134.abc.idm.lab.eng.brq.redhat.com,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=example"
[18/Jun/2015:15:59:20 +0200] conn=56 op=30 RESULT err=0 tag=107 nentries=0 etime=0 csn=5582eaac001900050000
[18/Jun/2015:15:59:20 +0200] conn=56 op=34 SRCH base="cn=ipaconfig,cn=etc,dc=ipa,dc=example" scope=0 filter="(objectClass=*)" attrs=ALL
[18/Jun/2015:15:59:20 +0200] conn=56 op=34 RESULT err=0 tag=101 nentries=1 etime=0
[18/Jun/2015:15:59:20 +0200] conn=56 op=35 SRCH base="cn=DNS,cn=vm-134.abc.idm.lab.eng.brq.redhat.com,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=example" scope=1 filter="(objectClass=*)" attrs=""
[18/Jun/2015:15:59:20 +0200] conn=56 op=35 RESULT err=0 tag=101 nentries=0 etime=0 notes=U
[18/Jun/2015:15:59:20 +0200] conn=56 op=36 DEL dn="cn=DNS,cn=vm-134.abc.idm.lab.eng.brq.redhat.com,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=example"
[18/Jun/2015:15:59:20 +0200] conn=56 op=33 RESULT err=0 tag=107 nentries=0 etime=0 csn=5582eaac001a00050000
[18/Jun/2015:15:59:20 +0200] conn=56 op=37 SRCH base="cn=ipaconfig,cn=etc,dc=ipa,dc=example" scope=0 filter="(objectClass=*)" attrs=ALL
[18/Jun/2015:15:59:20 +0200] conn=56 op=37 RESULT err=0 tag=101 nentries=1 etime=0
[18/Jun/2015:15:59:20 +0200] conn=56 op=38 SRCH base="cn=DNSKeySync,cn=vm-134.abc.idm.lab.eng.brq.redhat.com,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=example" scope=1 filter="(objectClass=*)" attrs=""
[18/Jun/2015:15:59:20 +0200] conn=56 op=38 RESULT err=0 tag=101 nentries=0 etime=0 notes=U
[18/Jun/2015:15:59:20 +0200] conn=56 op=39 DEL dn="cn=DNSKeySync,cn=vm-134.abc.idm.lab.eng.brq.redhat.com,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=example"
[18/Jun/2015:15:59:20 +0200] conn=56 op=36 RESULT err=0 tag=107 nentries=0 etime=0 csn=5582eaac001b00050000
[18/Jun/2015:15:59:20 +0200] conn=56 op=40 SRCH base="cn=ipaconfig,cn=etc,dc=ipa,dc=example" scope=0 filter="(objectClass=*)" attrs=ALL
[18/Jun/2015:15:59:20 +0200] conn=56 op=40 RESULT err=0 tag=101 nentries=1 etime=0
[18/Jun/2015:15:59:20 +0200] conn=59 op=5 UNBIND
[18/Jun/2015:15:59:20 +0200] conn=56 op=39 RESULT err=0 tag=107 nentries=0 etime=0 csn=5582eaac001c00050000
[18/Jun/2015:15:59:20 +0200] conn=56 op=41 SRCH base="cn=DNSKeyExporter,cn=vm-134.abc.idm.lab.eng.brq.redhat.com,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=example" scope=1 filter="(objectClass=*)" attrs=""
[18/Jun/2015:15:59:20 +0200] conn=56 op=41 RESULT err=0 tag=101 nentries=0 etime=0 notes=U
[18/Jun/2015:15:59:20 +0200] conn=56 op=42 DEL dn="cn=DNSKeyExporter,cn=vm-134.abc.idm.lab.eng.brq.redhat.com,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=example"
[18/Jun/2015:15:59:20 +0200] conn=56 op=43 SRCH base="cn=ipaconfig,cn=etc,dc=ipa,dc=example" scope=0 filter="(objectClass=*)" attrs=ALL
[18/Jun/2015:15:59:20 +0200] conn=56 op=43 RESULT err=0 tag=101 nentries=1 etime=0
[18/Jun/2015:15:59:20 +0200] conn=56 op=44 SRCH base="cn=DNSSEC,cn=vm-134.abc.idm.lab.eng.brq.redhat.com,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=example" scope=1 filter="(objectClass=*)" attrs=""
[18/Jun/2015:15:59:20 +0200] conn=56 op=44 RESULT err=0 tag=101 nentries=0 etime=0 notes=U
[18/Jun/2015:15:59:20 +0200] conn=56 op=45 DEL dn="cn=DNSSEC,cn=vm-134.abc.idm.lab.eng.brq.redhat.com,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=example"
[18/Jun/2015:15:59:20 +0200] conn=56 op=42 RESULT err=0 tag=107 nentries=0 etime=0 csn=5582eaac001d00050000
[18/Jun/2015:15:59:20 +0200] conn=56 op=46 DEL dn="cn=vm-134.abc.idm.lab.eng.brq.redhat.com,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=example"
[18/Jun/2015:15:59:20 +0200] conn=56 op=45 RESULT err=0 tag=107 nentries=0 etime=0 csn=5582eaac001e00050000
[18/Jun/2015:15:59:20 +0200] conn=57 op=3 SRCH base="dc=ipa,dc=example" scope=2 filter="(krbPrincipalName=*/vm-134.abc.idm.lab.eng.brq.redhat.com@IPA.EXAMPLE)" attrs=ALL
[18/Jun/2015:15:59:20 +0200] conn=56 op=46 RESULT err=0 tag=107 nentries=0 etime=0 csn=5582eaac001f00050000
[18/Jun/2015:15:59:20 +0200] conn=57 op=3 RESULT err=0 tag=101 nentries=5 etime=0
[18/Jun/2015:15:59:20 +0200] conn=57 op=4 SRCH base="cn=schema" scope=0 filter="(objectClass=*)" attrs="attributeTypes objectClasses"
[18/Jun/2015:15:59:21 +0200] conn=57 op=4 RESULT err=0 tag=101 nentries=1 etime=1
[18/Jun/2015:15:59:21 +0200] conn=57 op=5 DEL dn="krbprincipalname=ldap/vm-134.abc.idm.lab.eng.brq.redhat.com@IPA.EXAMPLE,cn=services,cn=accounts,dc=ipa,dc=example"
[18/Jun/2015:15:59:21 +0200] conn=57 op=6 DEL dn="fqdn=vm-134.abc.idm.lab.eng.brq.redhat.com,cn=computers,cn=accounts,dc=ipa,dc=example"
[18/Jun/2015:15:59:21 +0200] conn=57 op=5 RESULT err=0 tag=107 nentries=0 etime=0 csn=5582eaac002600050000
[18/Jun/2015:15:59:21 +0200] conn=57 op=7 DEL dn="krbprincipalname=HTTP/vm-134.abc.idm.lab.eng.brq.redhat.com@IPA.EXAMPLE,cn=services,cn=accounts,dc=ipa,dc=example"
[18/Jun/2015:15:59:21 +0200] conn=57 op=6 RESULT err=0 tag=107 nentries=0 etime=0 csn=5582eaac002700050000
[18/Jun/2015:15:59:21 +0200] conn=57 op=8 DEL dn="krbprincipalname=DNS/vm-134.abc.idm.lab.eng.brq.redhat.com@IPA.EXAMPLE,cn=services,cn=accounts,dc=ipa,dc=example"
[18/Jun/2015:15:59:21 +0200] conn=57 op=7 RESULT err=0 tag=107 nentries=0 etime=0 csn=5582eaac002c00050000
[18/Jun/2015:15:59:21 +0200] conn=57 op=9 DEL dn="krbprincipalname=ipa-dnskeysyncd/vm-134.abc.idm.lab.eng.brq.redhat.com@IPA.EXAMPLE,cn=services,cn=accounts,dc=ipa,dc=example"
[18/Jun/2015:15:59:21 +0200] conn=57 op=8 RESULT err=0 tag=107 nentries=0 etime=0 csn=5582eaac002d00050000
[18/Jun/2015:15:59:21 +0200] conn=57 op=10 MOD dn="cn=ipa-http-delegation,cn=s4u2proxy,cn=etc,dc=ipa,dc=example"
[18/Jun/2015:15:59:21 +0200] conn=57 op=9 RESULT err=0 tag=107 nentries=0 etime=0 csn=5582eaac002f00050000
[18/Jun/2015:15:59:21 +0200] conn=57 op=11 MOD dn="cn=ipa-ldap-delegation-targets,cn=s4u2proxy,cn=etc,dc=ipa,dc=example"
[18/Jun/2015:15:59:21 +0200] conn=57 op=10 RESULT err=0 tag=103 nentries=0 etime=0 csn=5582eaac003100050000
[18/Jun/2015:15:59:21 +0200] conn=57 op=12 MOD dn="cn=ipa-cifs-delegation-targets,cn=s4u2proxy,cn=etc,dc=ipa,dc=example"
[18/Jun/2015:15:59:21 +0200] conn=57 op=13 SRCH base="cn=vm-134.abc.idm.lab.eng.brq.redhat.com,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=example" scope=2 filter="(objectClass=*)" attrs=ALL
[18/Jun/2015:15:59:21 +0200] conn=57 op=12 RESULT err=16 tag=103 nentries=0 etime=0 csn=5582eaac003300050000
[18/Jun/2015:15:59:21 +0200] conn=57 op=13 RESULT err=32 tag=101 nentries=0 etime=0
[18/Jun/2015:15:59:21 +0200] conn=57 op=14 SRCH base="cn=ipa,cn=etc,dc=ipa,dc=example" scope=0 filter="(objectClass=*)" attrs="aci"
[18/Jun/2015:15:59:21 +0200] conn=57 op=14 RESULT err=0 tag=101 nentries=1 etime=0
[18/Jun/2015:15:59:21 +0200] conn=57 op=15 MOD dn="cn=ipa,cn=etc,dc=ipa,dc=example"
[18/Jun/2015:15:59:21 +0200] conn=57 op=11 RESULT err=0 tag=103 nentries=0 etime=0 csn=5582eaac003200050000
[18/Jun/2015:15:59:21 +0200] conn=57 op=16 SRCH base="cn=masters,cn=ipa,cn=etc,dc=ipa,dc=example" scope=0 filter="(objectClass=*)" attrs="aci"
[18/Jun/2015:15:59:21 +0200] conn=57 op=16 RESULT err=0 tag=101 nentries=1 etime=0
[18/Jun/2015:15:59:21 +0200] conn=57 op=17 MOD dn="cn=masters,cn=ipa,cn=etc,dc=ipa,dc=example"
[18/Jun/2015:15:59:21 +0200] conn=57 op=15 RESULT err=0 tag=103 nentries=0 etime=0 csn=5582eaac003400050000
[18/Jun/2015:15:59:21 +0200] conn=57 op=18 SRCH base="cn=certificates,cn=ipa,cn=etc,dc=ipa,dc=example" scope=0 filter="(objectClass=*)" attrs="aci"
[18/Jun/2015:15:59:21 +0200] conn=57 op=17 RESULT err=0 tag=103 nentries=0 etime=0 csn=5582eaac003500050000
[18/Jun/2015:15:59:21 +0200] conn=57 op=18 RESULT err=0 tag=101 nentries=1 etime=0
[18/Jun/2015:15:59:21 +0200] conn=57 op=19 MOD dn="cn=certificates,cn=ipa,cn=etc,dc=ipa,dc=example"
[18/Jun/2015:15:59:21 +0200] conn=57 op=20 SRCH base="cn=etc,dc=ipa,dc=example" scope=2 filter="(dnaHostname=vm-134.abc.idm.lab.eng.brq.redhat.com)" attrs=ALL
[18/Jun/2015:15:59:21 +0200] conn=57 op=19 RESULT err=0 tag=103 nentries=0 etime=0 csn=5582eaac003600050000
[18/Jun/2015:15:59:21 +0200] conn=57 op=20 RESULT err=0 tag=101 nentries=1 etime=0 notes=U
[18/Jun/2015:15:59:21 +0200] conn=57 op=21 DEL dn="dnaHostname=vm-134.abc.idm.lab.eng.brq.redhat.com+dnaPortNum=389,cn=posix-ids,cn=dna,cn=ipa,cn=etc,dc=ipa,dc=example"
[18/Jun/2015:15:59:21 +0200] conn=57 op=22 SRCH base="cn=default,ou=profile,dc=ipa,dc=example" scope=0 filter="(objectClass=*)" attrs=ALL
[18/Jun/2015:15:59:21 +0200] conn=57 op=22 RESULT err=0 tag=101 nentries=1 etime=0
[18/Jun/2015:15:59:21 +0200] conn=56 op=47 SRCH base="cn=ipaconfig,cn=etc,dc=ipa,dc=example" scope=0 filter="(objectClass=*)" attrs=ALL
[18/Jun/2015:15:59:21 +0200] conn=56 op=47 RESULT err=0 tag=101 nentries=1 etime=0

Last 4 lines are repeating indefinitelly:

[18/Jun/2015:15:59:21 +0200] conn=56 op=48 SRCH base="cn=realm,cn=topology,cn=ipa,cn=etc,dc=ipa,dc=example" scope=1 filter="(&(objectClass=iparepltoposegment)(ipaReplTopoSegmentLeftNode=vm-134.abc.idm.lab.eng.brq.redhat.com))" attrs="ipaReplTopoSegmentRightNode ipaReplTopoSegmentLeftNode cn ipaReplTopoSegmentDirection ipaReplTopoSegmentLeftNode"
[18/Jun/2015:15:59:21 +0200] conn=56 op=48 RESULT err=0 tag=101 nentries=0 etime=0
[18/Jun/2015:15:59:21 +0200] conn=56 op=49 SRCH base="cn=ipaconfig,cn=etc,dc=ipa,dc=example" scope=0 filter="(objectClass=*)" attrs=ALL
[18/Jun/2015:15:59:21 +0200] conn=56 op=49 RESULT err=0 tag=101 nentries=1 etime=0

Note that this doesn't happen every time. This is a second occurrence I know of.

ipa-replica-manage del hangs because it is waiting for removal of all segments with deleted master in left or right node.

There is an issue, most likely in topology plugin, that it doesn't delete all such segments.

A workaround which helped at least once:

  • re-add the server entry in cn=masters,cn=ipa,cn=etc,$SUFFIX
  • restart dirsrv (maybe)
  • run ipa-replica-manage del <masterfqdn> again
  • topology plugin will then correctly delete the segments

Also I think that the " Last 4 lines are repeating indefinitelly: " should be 8 lines. One search for
{{{ipaReplTopoSegmentLeftNode=vm-134.abc.idm.lab.eng.brq.redhat.com}}}
and second for
{{{ipaReplTopoSegmentRightNode=vm-134.abc.idm.lab.eng.brq.redhat.com}}}

the second would be more interesting because vm-134 is most likely a right node. And therefore this search would have nentries=1

are the full logs somewhere ?
the four repeated lines show the search for the segment leftNode=vm-0134 and it returns 0 entries, so why is it repeated ?
could you show a result for a search for all segments ?

No logs unfortunately, Petr said that he no longer has the vms.

But as I wrote in comment 2, there are probably missing 4 lines which are almost the same with the exception of Left -> Right.

the relevant ipa-replica-manage code which produces it is:

i = 0
    while True:
        left = api.Command.topologysegment_find(
            u'realm', iparepltoposegmentleftnode=hostname_u, sizelimit=0)['result']
        right = api.Command.topologysegment_find(
            u'realm', iparepltoposegmentrightnode=hostname_u, sizelimit=0)['result']
        if not left and not right:
            print "Agreements deleted"
            break
        time.sleep(1)
        if i == 5: # taking too long, something is wrong, report
            print "Waiting for removal of replication agreements"
        i += 1

The question is why there could be a segment without relevant entry in cn=masters...

too bad we no longer have logs, the removal of the master should trigger the deletion of the segments, these are internal ops on the server where they are created and don't show up in the logs, but should be replicated and visible in the other logs.

in the error log snippet there are some errors about two minutes before the master was deleted, what was attempted there ?

and since the vm's no longer exist we can't also check if the agreement shave been removed

one more question, what was the version of 389-ds ?

DS version was 389-ds-base-1.3.4.a1-20150512143653.git1bf67a4.fc22.x86_64.

I'm sorry about the logs, for some reason I thought that the problem is known and we just do not have a ticket for it. Sorry!

I did run into the hanging replica-manage del, but only when removing a central node.

So far we cannot guarantee that if in a A <--> B <--> C topology, the node B is removed on A, that the removal of the segment B<-->C is visible on A, so the "waiting for removal" will loop forever.

In case we already know that the replica will be disconnected this check should be relaxed.

I will continue to investigate, if this can happen if a node is removed, which will not disconnect the topology.

The check will be loosen in ipa-replica-manage del, but one issue remains:

When also C is removed when the topology is still disconnected, then there is still segment B-C which can't be removed. This segment also prevents to recreate replica B(resp. C) from C(resp. B) if B(resp. C) is create from A because ipa-replica-install will say that there is existing replication agreement which has to be deleted.

I think that topology plugin should allow to delete a segment manually if one of the nodes does not have an entry in cn=masters to allow cleaning the invalid segments.

With that, the dangling segments could be deleted in ipa-replica-manage del if --force option is supplied as ipa-replica-install suggests:

A replication agreement for this host already exists. It needs to be removed.
Run this on the master that generated the info file:
    % ipa-replica-manage del deleted.example.test --force

Replying to [comment:11 pvoborni]:

I think that topology plugin should allow to delete a segment manually if one of the nodes does not have an entry in cn=masters to allow cleaning the invalid segments.

yes, this can be done, but the problem is still that the removal of a master might not be replicated to all servers. There are two different scenarios for breaking the topology:
Assume a topology A <--> B <--> C

  1. remove B on A, while B is up and replication is still working
  2. remove B on A, while B is DOWN and replication is NOT working

I have a fix in progress for 1]. When a removal of B is received on A, delay the deletion of segment A-B, so the removal of B will be replicated to B and C. If on A (and C) after some time the reḿoval of the segments connection B is not received delete all danglin segments to removed masters.

But in 2] this cannot work, there is no way to propagate the operations to C, until the topology is repaired C will have a wrong state, assuming a connected topology with A,B,C and reject any removal of segments.

I was thinking about a situation to force removal of segment when a node which should remove the segment no longer exists.

E.g.:

  1. initial setup: A<->B<->C
  2. remove B, topology is: A C
  3. remove C, topology is: A
    • there is still segment B-C
  4. create: D, E, F...

e.g. topology is in order but there is still a dangling segment which can't be removed.

Actually it can, but the way is too unpleasant: I had to (assuming topology from point 3):
4a. recreate B and C from A
5. connect each node with each other
- I've also created new B-C even though it existed, because it told me that it can't be removed.
*It somehow created wrong segment B-C with direction: left-right(or maybe right-left, I'm not sure).
6. at one time B crashed (not sure if it was after recreation of B-C when the other B-C segment existed)
{{{
ipa-topology-plugin - ipa_topo_util_modify: failed to modify entry (cn=vm-094.abc.idm.lab.eng.brq.redhat.com-to-vm-121.abc.idm.lab.eng.brq.redhat.com,cn=replica,cn=dc\3Dabc\2Cdc\3Didm\2Cdc\3Dlab\2Cdc\3Deng\2Cdc\3Dbrq\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping tree,cn=config): error 20
}}}
7. but after recreation of the segments and restart of B and A (was doing all on A), segments B-C could be removed
8. after cleaning old RUVs, all seems in order

May be I am missing something but shouldn't we delete all the agreements connecting to a replica when we deleting a replica? Then the situation described in the previous comment would never happen.Once we delete B the adjusting agreements will be deleted too.

Replying to [comment:15 dpal]:

May be I am missing something but shouldn't we delete all the agreements connecting to a replica when we deleting a replica? Then the situation described in the previous comment would never happen.Once we delete B the adjusting agreements will be deleted too.

In the normal scenario, yes, all segments, agreements will be deleted. But in scearios, where you remove a central node of a topology, maybe even shutting it down before the removal, then the removal applied on one node cannot be replicated to other, disconnected, nodes.
We can do a best effort, not good enough at the moment, but no guarantee that every can be automatically healed.

master:

  • 6f916b0 - allow deletion of segment if endpoint is not managed

Topology plugin was moved to next short release - 4.3, the bugfixing and verification of this ticket can thus wait.

fix in comment 17 prevents the hang. After reconnecting the topology, it should work OK.

Metadata Update from @pspacek:
- Issue assigned to lkrispen
- Issue set to the milestone: FreeIPA 4.3
7 years ago

Login to comment on this ticket.

Metadata
None
None
None
normal
None
None
None
None
defect
None
None
Replication
None
0
None
None
None
None
todo
None
None
None
Powered by Pagure 5.13.3
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.