Issue #48660: search over rest389 does not work for objects with binary attributes - 389-ds-base

389-ds-base

#48660 search over rest389 does not work for objects with binary attributes

Closed: wontfix None Opened 8 years ago by pspacek.

I'm attaching LDIF for reproducing the problem.

URL /v1/ldap/fqdn=vm-205.abc.idm.lab.eng.brq.redhat.com,cn=computers,cn=accounts,dc=ipa,dc=example/sub/(objectClass=*)/krbExtraData always return a traceback:

UnicodeDecodeError

UnicodeDecodeError: 'utf8' codec can't decode byte 0xe8 in position 4: invalid continuation byte
Traceback (most recent call last)

    File "/usr/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1836, in __call__

    return self.wsgi_app(environ, start_response)

    File "/usr/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1820, in wsgi_app

    response = self.make_response(self.handle_exception(e))

    File "/usr/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1403, in handle_exception

    reraise(exc_type, exc_value, tb)

    File "/usr/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1817, in wsgi_app

    response = self.full_dispatch_request()

    File "/usr/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1477, in full_dispatch_request

    rv = self.handle_user_exception(e)

    File "/usr/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1381, in handle_user_exception

    reraise(exc_type, exc_value, tb)

    File "/usr/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1475, in full_dispatch_request

    rv = self.dispatch_request()

    File "/usr/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1461, in dispatch_request

    return self.view_functions[rule.endpoint](**req.view_args)

    File "/root/rest389/rest389/Authentication.py", line 164, in decorator

    return _connection(f, *args, **kwargs)

    File "/root/rest389/rest389/Authentication.py", line 145, in _connection

    retval = f(dir_srv_conn, *args, **kwargs)

    File "/root/rest389/rest389/LDAPOps.py", line 371, in search_SFA

    return jsonReturnEntries(entries, request.url)

    File "/root/rest389/rest389/LDAPOps.py", line 62, in jsonReturnEntries

    return jsonResponse(results, 200, link)

    File "/root/rest389/rest389/Helpers.py", line 18, in jsonResponse

    resp = jsonify(resp_msg)

    File "/usr/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/json.py", line 238, in jsonify

    indent=indent),

    File "/usr/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/json.py", line 126, in dumps

    rv = _json.dumps(obj, **kwargs)

    File "/usr/lib64/python2.7/site-packages/simplejson/__init__.py", line 382, in dumps

    **kw).encode(obj)

    File "/usr/lib64/python2.7/site-packages/simplejson/encoder.py", line 271, in encode

    chunks = list(chunks)

    File "/usr/lib64/python2.7/site-packages/simplejson/encoder.py", line 632, in _iterencode

    for chunk in _iterencode_dict(o, _current_indent_level):

    File "/usr/lib64/python2.7/site-packages/simplejson/encoder.py", line 591, in _iterencode_dict

    for chunk in chunks:

    File "/usr/lib64/python2.7/site-packages/simplejson/encoder.py", line 481, in _iterencode_list

    for chunk in chunks:

    File "/usr/lib64/python2.7/site-packages/simplejson/encoder.py", line 591, in _iterencode_dict

    for chunk in chunks:

    File "/usr/lib64/python2.7/site-packages/simplejson/encoder.py", line 591, in _iterencode_dict

    for chunk in chunks:

    File "/usr/lib64/python2.7/site-packages/simplejson/encoder.py", line 450, in _iterencode_list

    yield buf + _encoder(value)

    UnicodeDecodeError: 'utf8' codec can't decode byte 0xe8 in position 4: invalid continuation byte

Tested with rest389 2cea971040396e73802aae9854b4dfa9523ee2b0.

pspacek commented 8 years ago

attachment
computers.ldif

firstyear commented 8 years ago

I don't like this. I think that this patch should not be in rest389. Because of where the code is, we'll have to have the decodeEntry part around all over the place.

I would rather have this in lib389/_entry.py so that on any operation the decode is implicit and transparent. It will also be useful to have this in lib389 anyway.

If we commit to this way, we'll end up with messy split up code for managing entries between rest389 and lib389.

Rest389 should do nothing to entries, or the data in them, it should never implement an operation, it should be the most minimal wrapper possible to enable a rest connection into lib389. It shouldn't even be data massaging for formatting IMO.

Anyway, sorry, but it's nack from me :(

mreynolds commented 8 years ago

Replying to [comment:3 firstyear]:

I don't like this. I think that this patch should not be in rest389. Because of where the code is, we'll have to have the decodeEntry part around all over the place.

I would rather have this in lib389/_entry.py so that on any operation the decode is implicit and transparent. It will also be useful to have this in lib389 anyway.

I completely agree - I'll move this function to lib389. Note - it's only JSON dump that has an issue with the binary values

lkrispen commented 8 years ago

the function converts a binary to a base64 string, so I would say it is encoding, not decoding

it looks like only the first value of an attr is encoded, how about multivalued attributes ?

mreynolds commented 8 years ago

Replying to [comment:5 lkrispen]:

the function converts a binary to a base64 string, so I would say it is encoding, not decoding

I'll revise the function name to something more accurate.

it looks like only the first value of an attr is encoded, how about multivalued attributes ?

It currently handles multivalued attributes just fine.

lkrispen commented 8 years ago

Replying to [comment:6 mreynolds]:

Replying to [comment:5 lkrispen]:

it looks like only the first value of an attr is encoded, how about multivalued attributes ?

It currently handles multivalued attributes just fine.
but attr.getValue() just returns one value, shouldn't it be

for v in attr.getValues()

mreynolds commented 8 years ago

Replying to [comment:7 lkrispen]:

Replying to [comment:6 mreynolds]:

Replying to [comment:5 lkrispen]:

it looks like only the first value of an attr is encoded, how about multivalued attributes ?

It currently handles multivalued attributes just fine.
but attr.getValue() just returns one value, shouldn't it be

for v in attr.getValues()

Yes, that seems correct, but the results show all the attribute values being converted with the existing fix:

ipaSshPubKey:: ZWNkc2Etc2hhMi1uaXN0cDI1NiBBQUFBRTJWalpITmhMWE5vWVRJdGJtbHpkSEF5TlRZQUFBQUlibWx6ZEhBeU5UWUFBQUJCQkJKNExZdzFpWnNCN3hJaWFKaERKek1FaGJ4bHEzNWVVVXpaVWdKcHRCaTc2RjVjbUZHUHY1MTM5MHgzMG1OeWJaR0F4UkphdXZUa1ZrclJrdElSS2prPQ==
ipaSshPubKey:: c3NoLWVkMjU1MTkgQUFBQUMzTnphQzFsWkRJMU5URTVBQUFBSUlKY0I0M1BPWC93QnNCME1zWEpabVZiVnVZenJJQVM1R09iYWtBWjE5S0Y=
ipaSshPubKey:: c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCQVFDOURkMGVNZkdXc1Rvb0pKYzZEMGVWWVVqM1NoeEEzRk93Y3VNUys0TkNmdC8ra2FiUWJtRWxpRXk1NTU3enpSZzVxOVN3RmxKdElGcEFuaTJMMnduYTc5L0NwKzVVRnhUTHV6c0toQlNtVWVwdEtvVEc3YkovclYxR09rcyt1V2RscGZqbjNlZFNieWRWbThyTWlyTFpKMUYzWUhjZHRKU0l1NlNFS2R2eGtETTFYR3YrWkM1bjVMd056Tlk5RFV0MG93bWt0WUozbUM5anFsa2VGaEg1OFVROC9UeGtlMzdwNnBmQ2dzMi9ZWFh1RHBsOC9MbkJTaHVwekZ0WDRBdDYyN1h5NStsNEdlSHVaa0o4MGRkYkpmTU9QV0d4aThhSHRNNHdLOVN5SFB1enR6UUdBV0tsOHY1TU0zWExGbXBaa05DQ1lHT0hSVEJmdkNCWVdjUHo=
krbExtraData:: AAJFO+hVcm9vdC9hZG1pbkBJUEEuRVhBTVBMRQA=

To:

"ipasshpubkey": [
"ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBJ4LYw1iZsB7xIiaJhDJzMEhbxlq35eUUzZUgJptBi76F5cmFGPv51390x30mNybZGAxRJauvTkVkrRktIRKjk=",
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIJcB43POX/wBsB0MsXJZmVbVuYzrIAS5GObakAZ19KF",
"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9Dd0eMfGWsTooJJc6D0eVYUj3ShxA3FOwcuMS+4NCft/+kabQbmEliEy5557zzRg5q9SwFlJtIFpAni2L2wna79/Cp+5UFxTLuzsKhBSmUeptKoTG7bJ/rV1GOks+uWdlpfjn3edSbydVm8rMirLZJ1F3YHcdtJSIu6SEKdvxkDM1XGv+ZC5n5LwNzNY9DUt0owmktYJ3mC9jqlkeFhH58UQ8/Txke37p6pfCgs2/YXXuDpl8/LnBShupzFtX4At627Xy5+l4GeHuZkJ80ddbJfMOPWGxi8aHtM4wK9SyHPuztzQGAWKl8v5MM3XLFmpZkNCCYGOHRTBfvCBYWcPz"
],

Perhaps this is already being converted by python-ldap(its already base64). I will do more thorough testing later today.

mreynolds commented 8 years ago

Moving ticket to lib389

mreynolds commented 8 years ago

attachment
0001-Ticket-48660-Add-function-to-convert-binary-values-i.patch

firstyear commented 8 years ago

Thanks for this! Looks good!

mreynolds commented 8 years ago

To ssh://git.fedorahosted.org/git/389/lib389.git
853d950..3f56c8d master -> master
commit 3f56c8d4851ed0d99f2a0b17e51eb249e4597a55
Author: Mark Reynolds mreynolds@redhat.com
Date: Tue Apr 5 16:03:49 2016 -0400

Metadata Update from @mreynolds:
- Issue assigned to mreynolds
- Issue set to the milestone: lib389 1.0.2

7 years ago

spichugi commented 3 years ago

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/389ds/389-ds-base/issues/1789

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.

Metadata Update from @spichugi:
- Issue close_status updated to: wontfix (was: Fixed)

3 years ago

Metadata

Assignee

mreynolds

Tags

None

Blocking

None

Depending on

None

Priority

major

Milestone

lib389 1.0.2

reviewstatus

ack

rhbz

None

origin

IPA

389-ds-base

Source Code

#48660 search over rest389 does not work for objects with binary attributes Closed: wontfix None Opened 8 years ago by pspacek.

Metadata

#48660 search over rest389 does not work for objects with binary attributes

Closed: wontfix None Opened 8 years ago by pspacek.