#6487 ipa-replica-conncheck fails randomly (race condition)
Closed: Fixed None Opened 7 years ago by pspacek.

ipa-replica-conncheck on replica side is not correctly working with listening sockets and it is opening and closing them in meanwhile.

Naturally master side cannot now when the port is open and when not so it is causing random failures.

This can be seen with netstat utility in shell.

Steps to reproduce

  1. Run ipa-replica-conncheck on replica:

    $ ipa-replica-conncheck -m ipa.demo1.freeipa.org.

  2. Let it work until it reaches following line:

    Please run the following command on remote master:
    /usr/sbin/ipa-replica-conncheck --replica <FQDN of replica>

  3. Run following command on the replica. It should always list all ports used by FreeIPA. Interestingly, it sometimes prints only subset of the ports. This is what is causing random failues in ipa-replica-conncheck from master's side.

    $ for i in $(seq 1 100)
    do
    date
    netstat -ltup | grep python
    sleep 0.33333333333
    done

I'm attaching the log from my experiments. It shows that number of open ports varies over time.

Probable cause

The listener side is using socket timeout options combined with messy error handling and no logging.

It seems that it is broken since its implementation in ticket:1107 but it manifested now because we finally have automated tests which are executed often enough to detect this.


Petr, do you have a suggestion what would be a better approach to check open ports from master side?

Well, I would find a random article about programming server applications and implement that.

Let me Google this:
https://www.google.com/search?q=Python+server+bind+multiple+ports

For example
http://stackoverflow.com/questions/22468160/python-tcp-several-listen-on-several-ports-at-once
seems like reasonable start.

#6456 was closed as duplicate of this bug.

master:

  • af0ba66 ipa-replica-conncheck: do not close listening ports until required

I didn't implement a proper solution and af0ba66 introduced another race condition. Sorry about that. Fix is ready in PR https://github.com/freeipa/freeipa/pull/309

master:

  • a44974c ipa-replica-conncheck: fix race condition

Metadata Update from @pspacek:
- Issue assigned to tkrizek
- Issue set to the milestone: FreeIPA 4.5

7 years ago

Login to comment on this ticket.

Metadata