#47816 internal syncrepl searches are flagged as unindexed
Closed: wontfix None Opened 9 years ago by rmeggins.

For example, running syncrepl with named, gives these messages in the errors log:

Internal search - Unindexed search: source (cn=server,cn=plugins,cn=config) search base="cn=changelog" filter="(&(changenumber>=42)(targetuniqueid=xxxxxxx))" etime=0 nentries=1 notes=U

Not sure which one is unindexed - could be changenumber is not indexed for range/ordering? could be targetuniqueid is not indexed for equality?


I would welcome this fix backported also to F20 Directory Server as FreeIPA DS log was swarmed with these error messages.

the problem is the missing equality index for targetuniqueid. a fix could add it to the default indexes for the changelog, but for existing deployments a reindexing would be required.
Couldn't IPA if it enables the sync repl plugin also add the index and reindexthe changelog ?

Do we also need to add the index to existing backends?

Replying to [comment:6 rmeggins]:

Do we also need to add the index to existing backends?

I see two problems with this. First: the current update scripts will just process the *ldifs in the .../updates directory, which add or modify entries in the dse.ldif whose DN is known. For adding and index to an existing backend this would require to query the dse.ldif to find the backends and then add it. I don't see how this can be done in the updateDS code.
Second: Even if we succeed it would require a reindex, and if we can trigger it it would start to reindex all backends, with the result that the attribute didn't exist.
If we do not trigger a reindex, I think the index will be used after a restart and newly created entries will be indexed and existing ones not.

Replying to [comment:7 lkrispen]:

Replying to [comment:6 rmeggins]:

Do we also need to add the index to existing backends?

I see two problems with this. First: the current update scripts will just process the *ldifs in the .../updates directory, which add or modify entries in the dse.ldif whose DN is known. For adding and index to an existing backend this would require to query the dse.ldif to find the backends and then add it. I don't see how this can be done in the updateDS code.

Right. You can't do it with a .ldif file. You can do it with a .pl file. See 90subtreereindex.pl and others in ldap/admin/src/scripts

Second: Even if we succeed it would require a reindex, and if we can trigger it it would start to reindex all backends, with the result that the attribute didn't exist.

We do similar things during upgrade e.g. see 90subtreereindex.pl. I don't know if this is something that we will want to do automatically since it affects relatively few users.

If we do not trigger a reindex, I think the index will be used after a restart

A yum update/rpm update will restart the server after the updates are applied.

and newly created entries will be indexed and existing ones not.

If that is going to cause problems, then either we need to add the index and do the reindex during upgrade, or we just add the default index and tell people what they need to do to add the index to all of their existing backend instances and reindex.

Adding an index without reindexing makes searches fail. Here is an example:

Without index:
[21/Aug/2014:10:34:24 +0200] conn=20 op=1 SRCH base="dc=example,dc=com" scope=2 filter="(homePhone=999-697-8966)" attrs=ALL
[21/Aug/2014:10:34:24 +0200] conn=20 op=1 RESULT err=0 tag=101 '''nentries=1''' etime=0 '''notes=U'''

Adding an index for homephone, without reindexing:
21/Aug/2014:10:37:18 +0200] conn=1 op=1 SRCH base="dc=example,dc=com" scope=2 filter="(homePhone=999-697-8966)" attrs=ALL
[21/Aug/2014:10:37:18 +0200] conn=1 op=1 RESULT err=0 tag=101 '''nentries=0''' etime=0

Reindexing:
[21/Aug/2014:10:38:51 +0200] conn=1 op=1 SRCH base="dc=example,dc=com" scope=2 filter="(homePhone=999-697-8966)" attrs=ALL
[21/Aug/2014:10:38:51 +0200] conn=1 op=1 RESULT err=0 tag=101 '''nentries=1''' etime=0

So just adding an index will result in unexpected behaviour. As this attribute is used only in the retro changelog it will not be acceptable to reindex all datbases as this can take very long.

Should we add a special script to do this only for the retro changelog backend ?

Replying to [comment:9 lkrispen]:

So just adding an index will result in unexpected behaviour. As this attribute is used only in the retro changelog it will not be acceptable to reindex all datbases as this can take very long.

Should we add a special script to do this only for the retro changelog backend ?

Yes. And we should be able to do db2index -n changelogdb so that we only reindex the retro changelog. This should only affect people who are using the retrochangelog, and it should not take a long time.

ok, makes sense and avoids adding stuff to release notes, where they could be missed. It could even be db2index -n changelogdb -t targetuniqueid.

]$ git merge ticket47816
Updating 50c05b4..0a69c4d
Fast-forward
Makefile.am | 2 ++
Makefile.in | 2 ++
ldap/admin/src/scripts/50targetuniqueid.ldif | 7 +++++++
ldap/admin/src/scripts/82targetuniqueidindex.pl | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++
ldap/ldif/template-dse.ldif.in | 7 +++++++
5 files changed, 70 insertions(+)
create mode 100644 ldap/admin/src/scripts/50targetuniqueid.ldif
create mode 100644 ldap/admin/src/scripts/82targetuniqueidindex.pl

$ git push origin master
Counting objects: 21, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (11/11), done.
Writing objects: 100% (12/12), 1.97 KiB, done.
Total 12 (delta 8), reused 0 (delta 0)
To ssh://git.fedorahosted.org/git/389/ds.git
50c05b4..0a69c4d master -> master

committed to 1.3.2
New commits:
commit abc947a

Metadata Update from @lkrispen:
- Issue assigned to lkrispen
- Issue set to the milestone: 1.3.3 - 8/14 (August)

7 years ago

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/389ds/389-ds-base/issues/1147

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.

Metadata Update from @spichugi:
- Issue close_status updated to: wontfix (was: Fixed)

3 years ago

Login to comment on this ticket.

Metadata