When an entry in AD has a large number of values for a multi-valued attribute, AD will break the values up into chunks. If we want to get all of the values, we need to use a "range retrieval" mechanism. This is described here:
http://msdn.microsoft.com/en-us/library/cc223242.aspx
Right now, we currently have problems if you try to sync a group with a large number of members (>1500 IIRC).
MaxValRange: default value == 1500 The maximum number of values that can be retrieved from a multivalued attribute in a single search request. Windows 2000 DCs do not support this policy and instead always use a setting of 1000 values.
dn: CN=Default Query Policy,CN=Query-Policies,CN=Directory Service,CN=Windows NT,CN=Services,CN=Configuration,<suffix> lDAPAdminLimits: MaxValRange=1500
The value can be replaced by ldapmodify, e.g., dn: CN=Default Query Policy,CN=Query-Policies,CN=Directory Service,CN=Windows NT,CN=Services,CN=Configuration,<suffix> changetype: modify delete: lDAPAdminLimits lDAPAdminLimits: MaxValRange=1500 - add: lDAPAdminLimits lDAPAdminLimits: MaxValRange=3000
Notes: 1) The minimum MaxValRange is 30. If smaller value is set, it's reset to 30.
2) If you use the openldap ldapsearch, you cannot ignore the filter to set an attribute list with the range retrieval request. $ dapsearch -LLLx -h <host> -p <port> -b "CN=test group,cn=users,dc=test,dc=example,dc=com" "(objectclass=)" "member;range=30-"
Bug description: AD returns up to MaxValRange? number of multi- valued attribute values in one search. If more attribute values exist, subtype ";range=0-(MaxValRange?-1)" is added to the type. AD Client (DS in this case) has to repeat the search with ";range=MaxValRange?-" then ";range=(2MaxValRange?)-" and so on until the values with the subtype ";range=low-" are returned. Windows Sync plugin did not support the range search.
Fix description: This patch implements the range search for the multi-valued attribute.
In addition, 1) For the search request, AD returns an attribute with no values if it was deleted on the AD side even if the delete is one of the multi-valued attributes. Windows Sync code was always adding the attribute to the entry's e_deleted_attr list. This patch stops it if there are more attributes having the same type existing in the entry.
2) slapi_entry_attr_get_ APIs call slapi_entry_attr_find and set the return value from the function to the value to return from the APIs. slapi_entry_attr_find returns -1 if it fails to find the attribute, which was returned if the type did not exist in the entry. This patch stops it and let slapi_entry_attr_get_ APIs return the default return value 0 in case slapi_entry_attr_find fails to find the type.
git patch file (master) 0001-Ticket-47314-Winsync-should-support-range-retrieval.patch
{{{ - if (0 == strcasecmp(a,"streetaddress")) - { + if (0 == strncasecmp(a, "streetaddress", sizeof("streetaddress")-1)) { }}}
This changes the behavior slightly - it will now accept an attribute name that begins with streetaddress instead of exactly matching "streetaddress". Is this necessary? If not, I don't think we should add this.
{{{ windows_private_set_range_attrs dp->range_attrs = attrs; }}}
Do you need to free dp->range_attrs before setting it to attrs? Is there any chance this will cause a memory leak?
Replying to [comment:9 rmeggins]:
{{{ - if (0 == strcasecmp(a,"streetaddress")) - { + if (0 == strncasecmp(a, "streetaddress", sizeof("streetaddress")-1)) { }}} This changes the behavior slightly - it will now accept an attribute name that begins with streetaddress instead of exactly matching "streetaddress". Is this necessary? If not, I don't think we should add this.
You are right. I reverted the change.
{{{ windows_private_set_range_attrs dp->range_attrs = attrs; }}} Do you need to free dp->range_attrs before setting it to attrs? Is there any chance this will cause a memory leak? I'm reusing the attrs for the next range retrieval: 875 exattrs = windows_private_get_range_attrs(conn->agmt); 876 charray_merge(&attrs, exattrs, 0 / pass in /); 877 slapi_ch_free((void )&exattrs); / strings are passed in / 881 rc = ldap_search_ext( conn->ld, dn, scope, filter, attrs, PR_FALSE, server_controls, 882 NULL / ClientControls /, 0,0, &msgid); And frees it as "attrs": 909 slapi_ch_array_free(attrs); 910 attrs = NULL;
Do you need to free dp->range_attrs before setting it to attrs? Is there any chance this will cause a memory leak? I'm reusing the attrs for the next range retrieval: 875 exattrs = windows_private_get_range_attrs(conn->agmt); 876 charray_merge(&attrs, exattrs, 0 / pass in /); 877 slapi_ch_free((void )&exattrs); / strings are passed in / 881 rc = ldap_search_ext( conn->ld, dn, scope, filter, attrs, PR_FALSE, server_controls, 882 NULL / ClientControls /, 0,0, &msgid); And frees it as "attrs": 909 slapi_ch_array_free(attrs); 910 attrs = NULL;
ack
Reviewed by Rich (Thank you!!)
Pushed to master: 36f3f34..7f1631e master -> master commit 7f1631e
Steps to reproduce: Add a group to AD and let the group have members more than MaxValRange value (1500, by default You can reset a smaller value for testing). Run incremental update and verify all the (unique)member values in the group are synchronized to the DS. Run total update and verify the values, as well.
Ticket has been cloned to Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1044149
Metadata Update from @nhosoi: - Issue assigned to nhosoi - Issue set to the milestone: 1.3.2 - 08/13 (August)
389-ds-base is moving from Pagure to Github. This means that new issues and pull requests will be accepted only in 389-ds-base's github repository.
This issue has been cloned to Github and is available here: - https://github.com/389ds/389-ds-base/issues/651
If you want to receive further updates on the issue, please navigate to the github issue and click on subscribe button.
subscribe
Thank you for understanding. We apologize for all inconvenience.
Metadata Update from @spichugi: - Issue close_status updated to: wontfix (was: Fixed)
Login to comment on this ticket.