Comparing the following 2 mods in one modify, the first one is much faster. Tested wit 10K values, the first case took 23 sec. vs. the second 80 sec.
mods[0].mod_op: LDAP_MOD_ADD mods[0].mod_type: uniqueMember mods[0].mod_values: <value_0> <value_1> ... <value_N> mods[0].mod_op: LDAP_MOD_ADD mods[0].mod_type: uniqueMember mods[0].mod_values: <value_0> mods[1].mod_op: LDAP_MOD_ADD mods[1].mod_type: uniqueMember mods[1]mod_values: <value_1> ... mods[N].mod_op: LDAP_MOD_ADD mods[N].mod_type: uniqueMember mods[N]mod_values: <value_N>
We may be able to introduce mods optimizer for some cases like the above.
Note: there are some operations which should not be optimized/cancelled. For instance, mods could be used for check-and-increment as follows. In the example, if foo=1 does not exist, the modify should fail and the next add should not happen.
mods[0].mod_op: LDAP_MOD_DELETE mods[0].mod_type: foo mods[0].mod_values: 1 mods[1].mod_op: LDAP_MOD_ADD mods[1].mod_type: foo mods[1]mod_values: 2
mozldap test program modattrs.c
How to run the test program modattrs:
{{{ 1. Compile modattrs (You may need to modify the server port MY_PORT and the directory manager password MGR_PW) $ gcc -g -I /usr/include/mozldap -o modattrs modattrs.c -lldap60 -lpthread 2. Set up 389-ds-base with the Suffix "o=my.com" 3. Add the following group dn: cn=Accounting Managers,ou=Groups,o=my.com nsUniqueId: fddf0105-cab411e1-bc6184ba-ac438f09 objectClass: top objectClass: groupOfUniqueNames objectClass: eduPerson cn: Accounting Managers ou: groups description: People who can manage accounting entries 4. Run modattrs 4-1. one ldap modify request modifies (adds) one attribute value. ./modattrs [-n <number_of_attribute_values_to_add>] 4-2. one ldap modify request modifies (adds) all attribute values stored in one big mod. (the first case of the Description) ./modattrs [-n <number_of_attribute_values_to_add>] one 4-3. one ldap modify request modifies (adds) all attribute values, where each value is store in each mod. (the second case of the Description) ./modattrs [-n <number_of_attribute_values_to_add>] many }}}
Test results: {{{ attribute type: uniqueMember (DN string) number_of_attribute_values_to_add: 1000 no option (4-1): 71 sec one (4-2): <1 sec many (4-3): 1 sec }}}
{{{ attribute type: uniqueMember (DN string) number_of_attribute_values_to_add: 10000 one (4-2): 23 sec many (4-3): 80 sec }}}
{{{ attribute type: eduPersonEntitlement (Directory string) number_of_attribute_values_to_add: 1000 no option (4-1): 78 sec one (4-2): 1 sec many (4-3): 2 sec }}}
{{{ attribute type: eduPersonEntitlement (Directory string) number_of_attribute_values_to_add: 10000 one (4-2): 83 sec many (4-3): 211 sec }}}
Additional info: {{{ attribute type: uniqueMember (DN string) number_of_attribute_values_to_add: 1000 no option (4-1): 71 sec no option (4-1) with plugins off(): 66 sec }}} {{{ attribute type: uniqueMember (DN string) number_of_attribute_values_to_add: 10000 one (4-2): 23 sec one (4-2) with plugins off(): 20 sec many (4-3): 80 sec many (4-3) with plugins off(*): 75 sec }}}
(*) disabled plugins: Auto Membership Plugin, Managed Entries, Linked Attributes Note: MemberOf, referential integrity postoperation, Distributed Numeric Assignment Plugin, attribute uniqueness are off, by default.
So what does modattrs.c do that we can't do with ldapmodify and an LDIF file?
Replying to [comment:4 rmeggins]:
A good question. :) All can be done by scripts, I believe. It was just handy for me. :p
set default ticket origin to Community
Added initial screened field value.
Test results...
Without patch
{{{ ./modattrs -n 10000 many Start 10000 modify (Many): Fri Sep 6 11:11:29 2013 End 10000 modify (Many): Fri Sep 6 11:11:54 2013 (25 sec) }}}
Results with patch:
{{{
./modattrs -n 10000 many Start 10000 modify (Many): Fri Sep 6 11:08:23 2013 End 10000 modify (Many): Fri Sep 6 11:08:23 2013 (0 sec) }}}
Since the results are good, and no valgrind complaints, I'll be preparing the patch for review...
attachment 0001-Ticket-411-RFE-mods-optimizer.patch
One reason why a user may want to add values one at a time is for error checking. If I attempt to add a single value, I can know exactly if there was an error with that particular value - already exists, invalid syntax, etc. Same with deletion - value doesn't exist. Is this substantially different with the new code? For example, what does ldapmodify -c -S errfile report in the errfile? If the behavior is different, we need to document the new behavior.
Replying to [comment:19 rmeggins]:
This fix takes this "single" mod operation:
dn: uid=m,dc=example,dc=com changetype: modify delete: description description: 1 - delete: description description: 2 - delete: description description: 3
and converts it to:
dn: uid=m,dc=example,dc=com changetype: modify delete: description description: 1 description: 2 description: 3
In either case, if one value does not exist, the entire mod operation will fail. I found no difference between these two "mods" when one of the values does not exist. (Error 16)
Replying to [comment:20 mreynolds]:
Replying to [comment:19 rmeggins]: <snip> In either case, if one value does not exist, the entire mod operation will fail. I found no difference between these two "mods" when one of the values does not exist. (Error 16)
Great! Thanks.
git merge ticket411 Updating 3adc242..8358b43 Fast-forward ldap/servers/slapd/modify.c | 76 +++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 76 insertions(+), 0 deletions(-)
git push origin master Counting objects: 11, done. Delta compression using up to 4 threads. Compressing objects: 100% (6/6), done. Writing objects: 100% (6/6), 1.83 KiB, done. Total 6 (delta 4), reused 0 (delta 0) To ssh://git.fedorahosted.org/git/389/ds.git 3adc242..8358b43 master -> master
commit 8358b43 Author: Mark Reynolds mreynolds@redhat.com Date: Fri Sep 6 11:55:21 2013 -0400
Ticket has been cloned to Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1044153
Metadata Update from @nhosoi: - Issue assigned to mreynolds - Issue set to the milestone: 1.3.2 - 09/13 (September)
389-ds-base is moving from Pagure to Github. This means that new issues and pull requests will be accepted only in 389-ds-base's github repository.
This issue has been cloned to Github and is available here: - https://github.com/389ds/389-ds-base/issues/411
If you want to receive further updates on the issue, please navigate to the github issue and click on subscribe button.
subscribe
Thank you for understanding. We apologize for all inconvenience.
Metadata Update from @spichugi: - Issue close_status updated to: wontfix (was: Fixed)
Login to comment on this ticket.