I will impelment this request, but we should clarify what really is needed and what flexibility is wanted.

Right now ref integrity is all or nothing, if we introduce scope what should it control:
- the scope of the entries referenced eg only if an entry inside scope is deleted, moved should the references be looked up and cleared
- the scope of the references, eg groups: only groups in a given scope will be checked and cleared
- a combination of both
- should it be multivalued, allow multiple scopes to be defined, if so, should it be in the original plugin configuration or multiple plugins (like attribute uniqueness).

I will start with teh case that the scope of entries will be defined and only if an entry in this scope is deleted or moved out of scope the referential integrity plugin will be triggered.

Replying to [comment:5 lkrispen]:

I will impelment this request, but we should clarify what really is needed and what flexibility is wanted.

Right now ref integrity is all or nothing, if we introduce scope what should it control:
- the scope of the entries referenced eg only if an entry inside scope is deleted, moved should the references be looked up and cleared
- the scope of the references, eg groups: only groups in a given scope will be checked and cleared
- a combination of both

I think the current need is a combination of both. We want to be able to define a suffix where referential integrity works. This means the entries that are references and the entries containing the references should both be in the scope.

The use-case that FreeIPA has is that user entries won't be deleted. Instead of deleting a user entry, they will be moved to a "cn=deleted users" tree. This would look similar to this:

{{{
cn=users,dc=example,dc=com - Contains active users and groups
cn=deleted users,dc=example,dc=com - Contains deleted users
}}}

We want referential integrity to work inside of "cn=users". When a user is moved to "cn=deleted users", and references to the user entry in "cn=users" should be removed just as if the user was being deleted with a true DELETE operation.

• should it be multivalued, allow multiple scopes to be defined, if so, should it be in the original plugin configuration or multiple plugins (like attribute uniqueness).

This is a nice to have, but I don't think that it's an immediate requirement since it's not necessary for the FreeIPA use case mentioned above. If multiple scopes are allowed, it seems like we would want multiple plug-in entries so you can have completely separate scopes that don't use cross-references. Multiple scope attributes in one plug-in entry might have value for allowing cross-references between the scopes, but I'd prefer that we wait for a use-case to materialize before implementing either of these.

I will start with teh case that the scope of entries will be defined and only if an entry in this scope is deleted or moved out of scope the referential integrity plugin will be triggered.

attachment
0001-Ticket-47527-Allow-referential-integrity-suffixes-to.patch

Looks good, except for some comment typos on line 1204

Need some error checking in case user did not specify a valid DN here:
{{{
plugin_EntryScope = slapi_sdn_new_dn_byref(plugin_attr_value);
...
plugin_ContainerScope = slapi_sdn_new_dn_byref(plugin_attr_value);
}}}

Otherwise, ack

attachment
0001-Ticket-47527-Allow-referential-integrity-suffixes-to.2.patch

Applied the requested changes from review and committed:

$ git merge ticket47527
Updating fc65a9c..0cb6de1
Fast-forward
ldap/servers/plugins/referint/referint.c | 88 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------
1 file changed, 76 insertions(+), 12 deletions(-)
$ git push origin master
Counting objects: 13, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (7/7), done.
Writing objects: 100% (7/7), 2.01 KiB, done.
Total 7 (delta 4), reused 0 (delta 0)
To ssh://git.fedorahosted.org/git/389/ds.git
fc65a9c..0cb6de1 master -> master

Guys, could you push that to 1.3.2.x in Fedora 20? FreeIPA would need this for performance reasons. Retro changelog trimming is pretty slow with this.

Replying to [comment:6 nkinder]:

Replying to [comment:5 lkrispen]:
...
The use-case that FreeIPA has is that user entries won't be deleted. Instead of deleting a user entry, they will be moved to a "cn=deleted users" tree. This would look similar to this:

{{{
cn=users,dc=example,dc=com - Contains active users and groups
cn=deleted users,dc=example,dc=com - Contains deleted users
}}}

We want referential integrity to work inside of "cn=users". When a user is moved to "cn=deleted users", and references to the user entry in "cn=users" should be removed just as if the user was being deleted with a true DELETE operation.

This is true, but FreeIPA needs referential integrity not only for users, but also for other subtrees, like cn=sudo,dc=example,dc=com where are guarding memberallowcmd, memberdenycmd and other attributes. With current scheme, we would need to create at least 5 different RefInt plugin entries to enable that function, right?

Maybe we should be able to specify a suffix like whole dc=example,dc=com and then blacklisted subtrees where we do not want RefInt to operate, i.e. the mentioned cn=deleted users,dc=example.com. Does that make sense?

By the way, I have tested patch 0001-Ticket-47527-Allow-referential-integrity-suffixes-to.2.patch on top of 1.3.2.7 in Fedora 20 and it solved by performance problems with retro change log trimming.

I have added following configuration:
{{{
nsslapd-plugincontainerscope: dc=ipa,dc=test
nsslapd-pluginentryscope: dc=ipa,dc=test
}}}

I hope that the configuration is valid :-) My intention was to limit all plugin operations to dc=ipa,dc=test subtree so any change in cn=changelog will not trigget the plugin.

To Pspacek: yes that was the inended configuration, good to have positive feedback

To Mkosek: This is not yet handled, but we could extend this to defining a scope for applying referential integrity and within this scope exclude a subtree,
Would this be enough: I'd try to avoid multiple scopes and even combinations of scope and attributes

Replying to [comment:11 mkosek]:

This is true, but FreeIPA needs referential integrity not only for users, but also for other subtrees, like cn=sudo,dc=example,dc=com where are guarding memberallowcmd, memberdenycmd and other attributes. With current scheme, we would need to create at least 5 different RefInt plugin entries to enable that function, right?

Maybe we should be able to specify a suffix like whole dc=example,dc=com and then blacklisted subtrees where we do not want RefInt to operate, i.e. the mentioned cn=deleted users,dc=example.com. Does that make sense?

I see a few ways we can go here. You have multiple suffixes that you want to use for referential integrity. Should these suffixes be treated as independent as far as referential integrity is concerned, or do you expect cross-references between suffixes? Here is what I mean by independent:

{{{
cn=users,dc=example,dc=com - references between entries within this suffix are maintained
cn=sudo,dc=example,dc=com - references between entries within this suffix are maintained
}}}

In the above case, references across the two suffixes would not be maintained. Is this what you want? If so, I think separate plug-in config entries is the right approach.

From a plug-in config perspective, I think having multi-valued scope attributes makes the most sense. A single plug-in config entry with multiple scope attributes would cause references to be maintained across all of those suffixes (not independent). To have a separate suffix that is independent, you would create an additional plug-in config entry. This allows a lot of flexibility for many use-cases. This is what I had in mind when I added comment#6, but I don't want to add this if it's not needed yet.

the basic fix for scoping referential integrity was reviewed and checked in to 1.3.2 and master.
To handle the further requests for multiple scopes and excluding subtrees I have created ticket #47621

Unfortunately, there are references between the scopes. For example sudo rule has a memberUser link to a user DN. So we will need to get them maintained. I assume we can deal with it in ticket #46621, current version should work (by targetting base DN) until FreeIPA user provisioning feature is implemented.

To Mkosek: I just updated #47621 with a design proposal, could you check if that is sufficient and lets continue discussion in that ticket

Ticket has been cloned to Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1044171

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/389ds/389-ds-base/issues/864

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.