#4773 Encoding/Marshalling error in ipa -vv / ipa -vvv
Closed: Fixed None Opened 9 years ago by pvoborni.

Some commands produce UnicodeDecodeError or error marshalling data for XML-RPC transport when -vv or -vvv options are used.

Can be observed when there are special types like Datetime, binary (as base64), dns_name and maybe others.

E.g.:

echo 'Foo' | ipa  -vvv  user-add barbar2 --first=Bar --last=Bar --password

produces:

ipa: INFO: trying https://test.example.com/ipa/session/json
ipa: INFO: Forwarding 'user_add' to json server 'https://test.example.com/ipa/session/json'
ipa: INFO: Request: {
    "id": 0, 
    "method": "user_add", 
    "params": [
        [
            "barbar2"
        ], 
        {
            "all": false, 
            "cn": "Bar Bar", 
            "displayname": "Bar Bar", 
            "gecos": "Bar Bar", 
            "givenname": "Bar", 
            "initials": "BB", 
            "krbprincipalname": "barbar2@EXAMPLE.COM", 
            "no_members": false, 
            "noprivate": false, 
            "random": false, 
            "raw": false, 
            "sn": "Bar", 
            "userpassword": "Foo", 
            "version": "2.109"
        }
    ]
}
send: u'POST /ipa/session/json HTTP/1.1\r\nHost: test.example.com\r\nAccept-Encoding: gzip\r\nAccept-Language: en-us\r\nReferer: https://test.example.com/ipa/xml\r\nCookie: ipa_session=c525c26e22712ae92c7de89c0429451b;\r\nUser-Agent: xmlrpclib.py/1.0.1 (by www.pythonware.com)\r\nContent-Type: application/json\r\nContent-Length: 359\r\n\r\n{"params": [["barbar2"], {"all": false, "no_members": false, "displayname": "Bar Bar", "cn": "Bar Bar", "noprivate": false, "userpassword": "Foo", "random": false, "raw": false, "version": "2.109", "gecos": "Bar Bar", "sn": "Bar", "krbprincipalname": "barbar2@EXAMPLE.COM", "givenname": "Bar", "initials": "BB"}], "method": "user_add", "id": 0}'
reply: 'HTTP/1.1 200 Success\r\n'
header: Date: Fri, 28 Nov 2014 11:56:37 GMT
header: Server: Apache/2.4.10 (Fedora) mod_auth_kerb/5.4 mod_nss/2.4.10 NSS/3.16.2 Basic ECC mod_wsgi/3.5 Python/2.7.8
header: Set-Cookie: ipa_session=c525c26e22712ae92c7de89c0429451b; Domain=test.example.com; Path=/ipa; Expires=Fri, 28 Nov 2014 12:16:37 GMT; Secure; HttpOnly
header: Vary: Accept-Encoding
header: Content-Encoding: gzip
header: Content-Length: 826
header: Content-Type: application/json; charset=utf-8
body: '{\n    "error": null, \n    "id": 0, \n    "principal": "admin@EXAMPLE.COM", \n    "result": {\n        "result": {\n            "cn": [\n                "Bar Bar"\n            ], \n            "displayname": [\n                "Bar Bar"\n            ], \n            "dn": "uid=barbar2,cn=users,cn=accounts,dc=example,dc=com", \n            "gecos": [\n                "Bar Bar"\n            ], \n            "gidnumber": [\n                "771200309"\n            ], \n            "givenname": [\n                "Bar"\n            ], \n            "has_keytab": true, \n            "has_password": true, \n            "homedirectory": [\n                "/home/barbar2"\n            ], \n            "initials": [\n                "BB"\n            ], \n            "ipantsecurityidentifier": [\n                "S-1-5-21-2363626577-2684070434-1954711201-1309"\n            ], \n            "ipauniqueid": [\n                "9bfed9a2-76f5-11e4-9609-001a4a22217c"\n            ], \n            "krbextradata": [\n  '
body: '              {\n                    "__base64__": "AAL1YnhUcm9vdC9hZG1pbkBJRE0uTEFCLkVORy5CUlEuUkVESEFULkNPTQA="\n                }\n            ], \n            "krblastpwdchange": [\n                {\n                    "__datetime__": "20141128115637Z"\n                }\n            ], \n            "krbpasswordexpiration": [\n                {\n                    "__datetime__": "20141128115637Z"\n                }\n            ], \n            "krbprincipalname": [\n                "barbar2@EXAMPLE.COM"\n            ], \n            "loginshell": [\n                "/bin/sh"\n            ], \n            "mail": [\n                "barbar2@example.com"\n            ], \n            "memberof_group": [\n                "ipausers"\n            ], \n            "mepmanagedentry": [\n                "cn=barbar2,cn=groups,cn=accounts,dc=example,dc=com"\n            ], \n            "objectclass": [\n                "ipaSshGroupOfPubKeys", \n                "krbticketpolicyaux", '
body: '\n                "ipaobject", \n                "mepOriginEntry", \n                "ipantuserattrs", \n                "top", \n                "ipasshuser", \n                "inetorgperson", \n                "organizationalperson", \n                "person", \n                "krbprincipalaux", \n                "inetuser", \n                "posixaccount"\n            ], \n            "sn": [\n                "Bar"\n            ], \n            "uid": [\n                "barbar2"\n            ], \n            "uidnumber": [\n                "771200309"\n            ]\n        }, \n        "summary": "Added user \\"barbar2\\"", \n        "value": "barbar2"\n    }, \n    "version": "4.1.2.201411271350GITbfa0947"\n}'
ipa: ERROR: non-public: UnicodeDecodeError: 'utf8' codec can't decode byte 0xf5 in position 2: invalid start byte
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipalib/backend.py", line 129, in execute
    result = self.Command[_name](*args, **options)
  File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 439, in __call__
    ret = self.run(*args, **options)
  File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 755, in run
    return self.forward(*args, **options)
  File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 776, in forward
    return self.Backend.rpcclient.forward(self.name, *args, **kw)
  File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 883, in forward
    return self._call_command(command, params)
  File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 860, in _call_command
    return command(*params)
  File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1007, in _call
    return self.__request(name, args)
  File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 989, in __request
    json.dumps(response, sort_keys=True, indent=4))
  File "/usr/lib64/python2.7/json/__init__.py", line 250, in dumps
    sort_keys=sort_keys, **kw).encode(obj)
  File "/usr/lib64/python2.7/json/encoder.py", line 209, in encode
    chunks = list(chunks)
  File "/usr/lib64/python2.7/json/encoder.py", line 434, in _iterencode
    for chunk in _iterencode_dict(o, _current_indent_level):
  File "/usr/lib64/python2.7/json/encoder.py", line 408, in _iterencode_dict
    for chunk in chunks:
  File "/usr/lib64/python2.7/json/encoder.py", line 408, in _iterencode_dict
    for chunk in chunks:
  File "/usr/lib64/python2.7/json/encoder.py", line 408, in _iterencode_dict
    for chunk in chunks:
  File "/usr/lib64/python2.7/json/encoder.py", line 313, in _iterencode_list
    yield buf + _encoder(value)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xf5 in position 2: invalid start byte
ipa: ERROR: an internal error has occurred

Looks like a regression of #4233.

I got another error with marshaling:

# ipa -vv dnszone-add .
ipa: INFO: trying https://ipa.example.com/ipa/json
ipa: INFO: Forwarding 'dnszone_add' to json server 'https://ipa.example.com/ipa/json'
ipa: ERROR: error marshalling data for XML-RPC transport: <DNS name .> is not JSON serializable

Maybe there is also problem with datetime object.

master:

  • a18ef90 rpcclient: use json_encode_binary for verbose output

ipa-4-1:

  • 872ba41 rpcclient: use json_encode_binary for verbose output

Metadata Update from @pvoborni:
- Issue assigned to pvoborni
- Issue set to the milestone: FreeIPA 4.1.3

7 years ago

Login to comment on this ticket.

Metadata