#47314 Winsync should support range retrieval
Closed: wontfix None Opened 10 years ago by nkinder.

When an entry in AD has a large number of values for a multi-valued attribute, AD will break the values up into chunks. If we want to get all of the values, we need to use a "range retrieval" mechanism. This is described here:

http://msdn.microsoft.com/en-us/library/cc223242.aspx

Right now, we currently have problems if you try to sync a group with a large number of members (>1500 IIRC).


MaxValRange: default value == 1500
The maximum number of values that can be retrieved from a multivalued attribute in a single search request. Windows 2000 DCs do not support this policy and instead always use a setting of 1000 values.

dn: CN=Default Query Policy,CN=Query-Policies,CN=Directory Service,CN=Windows NT,CN=Services,CN=Configuration,<suffix>
lDAPAdminLimits: MaxValRange=1500

The value can be replaced by ldapmodify, e.g.,
dn: CN=Default Query Policy,CN=Query-Policies,CN=Directory Service,CN=Windows NT,CN=Services,CN=Configuration,<suffix>
changetype: modify
delete: lDAPAdminLimits
lDAPAdminLimits: MaxValRange=1500
-
add: lDAPAdminLimits
lDAPAdminLimits: MaxValRange=3000

Notes:
1) The minimum MaxValRange is 30. If smaller value is set, it's reset to 30.

2) If you use the openldap ldapsearch, you cannot ignore the filter to set an attribute list with the range retrieval request.
$ dapsearch -LLLx -h <host> -p <port> -b "CN=test group,cn=users,dc=test,dc=example,dc=com" "(objectclass=)" "member;range=30-"

Bug description: AD returns up to MaxValRange? number of multi-
valued attribute values in one search. If more attribute values
exist, subtype ";range=0-(MaxValRange?-1)" is added to the type.
AD Client (DS in this case) has to repeat the search with
";range=MaxValRange?-" then ";range=(2MaxValRange?)-" and so on
until the values with the subtype ";range=low-
" are returned.
Windows Sync plugin did not support the range search.

Fix description: This patch implements the range search for the
multi-valued attribute.

In addition,
1) For the search request, AD returns an attribute with no values
if it was deleted on the AD side even if the delete is one of the
multi-valued attributes. Windows Sync code was always adding the
attribute to the entry's e_deleted_attr list. This patch stops it
if there are more attributes having the same type existing in the
entry.

2) slapi_entry_attr_get_ APIs call slapi_entry_attr_find and set
the return value from the function to the value to return from the
APIs. slapi_entry_attr_find returns -1 if it fails to find the
attribute, which was returned if the type did not exist in the entry.
This patch stops it and let slapi_entry_attr_get_
APIs return the
default return value 0 in case slapi_entry_attr_find fails to find
the type.

{{{
- if (0 == strcasecmp(a,"streetaddress"))
- {
+ if (0 == strncasecmp(a, "streetaddress", sizeof("streetaddress")-1)) {
}}}

This changes the behavior slightly - it will now accept an attribute name that begins with streetaddress instead of exactly matching "streetaddress". Is this necessary? If not, I don't think we should add this.

{{{
windows_private_set_range_attrs
dp->range_attrs = attrs;
}}}

Do you need to free dp->range_attrs before setting it to attrs? Is there any chance this will cause a memory leak?

Replying to [comment:9 rmeggins]:

{{{
- if (0 == strcasecmp(a,"streetaddress"))
- {
+ if (0 == strncasecmp(a, "streetaddress", sizeof("streetaddress")-1)) {
}}}

This changes the behavior slightly - it will now accept an attribute name that begins with streetaddress instead of exactly matching "streetaddress". Is this necessary? If not, I don't think we should add this.

You are right. I reverted the change.

{{{
windows_private_set_range_attrs
dp->range_attrs = attrs;
}}}

Do you need to free dp->range_attrs before setting it to attrs? Is there any chance this will cause a memory leak?
I'm reusing the attrs for the next range retrieval:
875 exattrs = windows_private_get_range_attrs(conn->agmt);
876 charray_merge(&attrs, exattrs, 0 / pass in /);
877 slapi_ch_free((void )&exattrs); / strings are passed in /
881 rc = ldap_search_ext( conn->ld, dn, scope, filter, attrs, PR_FALSE, server_controls,
882 NULL / ClientControls /, 0,0, &msgid);
And frees it as "attrs":
909 slapi_ch_array_free(attrs);
910 attrs = NULL;

Reviewed by Rich (Thank you!!)

Pushed to master:
36f3f34..7f1631e master -> master
commit 7f1631e

Steps to reproduce:
Add a group to AD and let the group have members more than MaxValRange value (1500, by default You can reset a smaller value for testing).
Run incremental update and verify all the (unique)member values in the group are synchronized to the DS.
Run total update and verify the values, as well.

Metadata Update from @nhosoi:
- Issue assigned to nhosoi
- Issue set to the milestone: 1.3.2 - 08/13 (August)

7 years ago

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/389ds/389-ds-base/issues/651

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.

Metadata Update from @spichugi:
- Issue close_status updated to: wontfix (was: Fixed)

3 years ago

Login to comment on this ticket.

Metadata