#48450 RFE Systemd password agent support
Closed: wontfix None Opened 8 years ago by firstyear.

When built --with-systemd we should support and utilise a systemd password agent for unlocking of the nss database keys in the absence of pin.txt.

http://www.freedesktop.org/wiki/Software/systemd/PasswordAgents/


Systemd has a number of apis we could use for this: But they are all private and hidden. Given there is no specification for what a client has to do, we need to copy the behaviour of another.

Here is there systemd utility to ask for passwords from the agent.

This has no guarantee of api stability or behaviour. We use at our own risk ....

https://github.com/systemd/systemd/blob/47889a8367aa791efc9e83b619ec1fec53e218b0/src/shared/ask-password-api.c#L455

Okay, so it seems like there are some more gotchas.

The client I linked to does some detection of if we are attached to a TTY or not. It changes the way it works between the socket client, and directly asking depending on.

So we need to choose: If we go "socket only" we can prompt during system start up similar to full disk encryption. When we run "systemctl start dirsrv" it would start the instances, and await us to call "systemd-tty-ask-password-agent" IE:

{{{
% sudo systemctl start dirsrv@localhost
[sudo] password for william:
% sudo systemd-tty-ask-password-agent
Enter password for dirsrv@localhost nss db: ****
%
}}}

Alternately, we need to implement both socket and TTY methods. This would give us the same crypt-setup like behaviour during system boot. However, it would change the behaviour of systemctl to ask before the command return. It would appear as:

{{{
% sudo systemctl start dirsrv@localhost
[sudo] password for william:
Enter password for dirsrv@localhost nss db: ****
%
}}}

The sad part about this is that the systemd password agent api really only gives us the socket use case, and even that only helps us at boot. The tty attached prompt would really be custom / ours. We could defer out to something in nss like svrcore (I think it is) if we didn't want to re-invent the wheel.

Alternately could ask the systemd team to make public the contents of their password-ask client, so that other projects could link to it. Which really, is meant to be the whole point of this api.

Finally, we could use a shell / system call to systemd-ask-password, but this wouldn't be the cleanest or most reliable solution IMO.

tl;dr:

The final solution has the lowest barrier to entry, but may cause some problems.

The socket only solution has some end user over head, and a fair bit of work for us to implement, but it will likely end up as the most reliable.

The socket and tty solution has the most work involved, but gives us the best user experience.

Asking the systemd team to open the api would likely be very difficult and time consuming to achieve.

Thank you, William, for your comprehensive research!

Replying to [comment:2 firstyear]:
[...]

The socket only solution has some end user over head, and a fair bit of work for us to implement, but it will likely end up as the most reliable.

The socket and tty solution has the most work involved, but gives us the best user experience.

Asking the systemd team to open the api would likely be very difficult and time consuming to achieve.

So, the most reliable means the best in the security aspect? If we take another choice, it'd be less secure? Typing the password is local to the host, but could it have a chance to get stolen?

Another question... We can configure to start the DS at the OS boot time. If it prompts there, the system boot stops there?

Thanks!

Most reliable as in "least likely to cause ns-slapd to crash or explode on a start up".

With regards to system boot stopping until we have the password, this is the socket method. We can choose to block start up or not though.

Blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1322167

For now work around is:

{{{
setfacl -m g:dirsrv:rwx /var/run/systemd/ask-password
}}}

Must be run every boot.

Usage:

{{{

systemctl start dirsrv@localhost.service

Broadcast message from root@rei.bne.redhat.com (Wed 2016-03-30 11:10:58 AEST):

Password entry required for 'Enter PIN for Internal (Software) Token:' (PID 1583).
Please enter password with the systemd-tty-ask-password-agent tool!

systemd-tty-ask-password-agent

Enter PIN for Internal (Software) Token: **

systemctl status dirsrv@localhost.service

● dirsrv@localhost.service - 389 Directory Server localhost.
Loaded: loaded (/etc/systemd/system/dirsrv@.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2016-03-30 11:10:58 AEST; 1min 22s ago
...
Mar 30 11:11:35 rei.bne.redhat.com ns-slapd[1579]: [30/Mar/2016:11:11:35.312517487 +1000] Configured SSL version range: min: TLS1.0, max: TLS1.2
}}}

In the error case (IE no perms to /var/run/systemd/ask-password)

{{{

systemctl start dirsrv@localhost.service

systemctl status dirsrv@localhost.service

...
Mar 30 11:03:14 rei.bne.redhat.com ns-slapd[32373]: SVRCORE systemd:getPin() -> creating socket FAILED 7
Mar 30 11:03:14 rei.bne.redhat.com ns-slapd[32373]: [30/Mar/2016:11:03:14.217408693 +1000] SSL alert: Security Initialization: Unable to authenticate (Netscape Portable Runtime...incorrect.)
Mar 30 11:03:14 rei.bne.redhat.com ns-slapd[32373]: [30/Mar/2016:11:03:14.228946802 +1000] ERROR: SSL Initialization Failed. Disabling SSL.
}}}

To fix run: setfacl -m g:dirsrv:rwx /var/run/systemd/ask-password.

{{{
diff --git a/m4/systemd.m4 b/m4/systemd.m4
new file mode 100644
index 0000000..139217c

1   # -*- tab-width: 4; -*- 
2   # Configure paths for systemd functionality 
3   # Public domain - William Brown <wibrown@redhat.com> 2016-03-22 
4    
5   AC_CHECKING(for systemd) 
6    
7   ## This is a nice simple check: As we don't need to link to systemd, only be able 
8   ## to consume some of it's socket api features, we just need to check for user 
9   ## intent with --with-systemd 
10   
11  AC_MSG_CHECKING(for --with-systemd) 
12  AC_ARG_WITH(systemd, AS_HELP_STRING([--with-systemd],[Enable Systemd native integration.]), 
13  [ 
14      if test "$withval" = yes 
15      then 
16          AC_MSG_RESULT([using systemd native features]) 
17          with_systemd=yes 
18      else 
19          AC_MSG_RESULT(no) 
20      fi 
21  ], 
22  AC_MSG_RESULT(no)) 
23   
24  if test "$with_systemd" = yes; then 
25      SYSTEMD_CFLAGS="-DWITH_SYSTEMD" 
26  else 
27      SYSTEMD_CFLAGS="" 
28  fi

}}}

It would be good to avoid using variable SYSTEMD_CFLAGS for such purpose. Variables _CFLAGS and _LIBS age generated by pkg-config and contain flags required for compilation.

The utility pkg-config creates empty SYSTEMD_CFLAGS on fedora but it needn't be true on other distributions. And your patch would override it.

I don't think this is an issue here as we don't actually use a link to systemd: We had to implement their apis because they don't provide a client api to ask password.

However, I will still change these variable names.

Maybe this will affect 389ds too because I have previously added a systemd_cflags var to the auto tools there.

Excellent! I applied your patches and ran the command line: {{{ # setfacl -m g:dirsrv:rwx /var/run/systemd/ask-password }}} Then, my server prompted for a password! {{{ # start-dirsrv test1 Enter PIN for Internal (Software) Token: **************************************** }}} BTW, starting from the raw command line now stopped working... :) {{{ # /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-test1 -i /tmp/pid.txt }}} The command line quits and the ns-slapd process is created. If I attach it to gdb, then it's waiting here: {{{ (gdb) bt #0 0x00007f91ec509d20 in __nanosleep_nocancel () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f91ec509bd4 in __sleep (seconds=0, seconds@entry=2) at ../sysdeps/unix/sysv/linux/sleep.c:138 #2 0x00007f91eca225a1 in getPin (obj=<optimized out>, tokenName=<optimized out>, retry=<optimized out>) at systemd-ask-pass.c:361 #3 0x00007f91eca215ea in getPin (ctx=0x5563804269a0, tokenName=0x55638041f1b1 "Internal (Software) Token", retry=<optimized out>) at cache.c:154 #4 0x00007f91ed6bf949 in pk11_GetPassword (wincx=0x0, retry=0, slot=0x55638041f070) at pk11auth.c:517 #5 PK11_DoPassword (slot=0x55638041f070, session=16777217, loadCerts=<optimized out>, wincx=0x0, alreadyLocked=0, contextSpecific=0) at pk11auth.c:589 #6 0x00007f91ef55dc8e in slapd_pk11_authenticate (slot=0x55638041f070, loadCerts=1, wincx=0x0) at ldap/servers/slapd/security_wrappers.c:222 #7 0x00007f91ef564016 in slapd_ssl_init () at ldap/servers/slapd/ssl.c:1383 #8 0x000055637df7f1a5 in slapd_do_all_nss_ssl_init (slapd_exemode=1, importexport_encrypt=0, s_port=10637, ports_info=0x7ffefe08f5a0) at ldap/servers/slapd/main.c:2983 #9 0x000055637df7573b in detach (slapd_exemode=1, importexport_encrypt=0, s_port=10637, ports_info=0x7ffefe08f5a0) at ldap/servers/slapd/detach.c:79 #10 0x000055637df7a6d7 in main (argc=5, argv=0x7ffefe08f6e8) at ldap/servers/slapd/main.c:960 }}} It's understandable since now I'm not depending upon systemd. I'd think it's not a problem for customers. But we may need to start the process from gdb to debug some issues observed at the startup time. Could it be possible to support it? Or somehow work around it?

Replying to [comment:11 nhosoi]:

Excellent! I applied your patches and ran the command line:
{{{

setfacl -m g:dirsrv:rwx /var/run/systemd/ask-password

}}}

Hopefully we shouldn't need to always do this. As /var/run/systemd/ask-password is on a tmpfs we would need to do this every boot :(

Then, my server prompted for a password!
{{{

start-dirsrv test1

Enter PIN for Internal (Software) Token: ******
}}}

I'm glad it worked for you!

BTW, starting from the raw command line now stopped working... :)
{{{

/usr/sbin/ns-slapd -D /etc/dirsrv/slapd-test1 -i /tmp/pid.txt

}}}
The command line quits and the ns-slapd process is created. If I attach it to gdb, then it's waiting here:
{{{
(gdb) bt

0 0x00007f91ec509d20 in __nanosleep_nocancel () at ../sysdeps/unix/syscall-template.S:81

1 0x00007f91ec509bd4 in __sleep (seconds=0, seconds@entry=2) at ../sysdeps/unix/sysv/linux/sleep.c:138

2 0x00007f91eca225a1 in getPin (obj=<optimized out>, tokenName=<optimized out>, retry=<optimized out>)

at systemd-ask-pass.c:361

3 0x00007f91eca215ea in getPin (ctx=0x5563804269a0, tokenName=0x55638041f1b1 "Internal (Software) Token",

retry=<optimized out>) at cache.c:154

4 0x00007f91ed6bf949 in pk11_GetPassword (wincx=0x0, retry=0, slot=0x55638041f070) at pk11auth.c:517

5 PK11_DoPassword (slot=0x55638041f070, session=16777217, loadCerts=<optimized out>, wincx=0x0, alreadyLocked=0,

contextSpecific=0) at pk11auth.c:589

6 0x00007f91ef55dc8e in slapd_pk11_authenticate (slot=0x55638041f070, loadCerts=1, wincx=0x0)

at ldap/servers/slapd/security_wrappers.c:222

7 0x00007f91ef564016 in slapd_ssl_init () at ldap/servers/slapd/ssl.c:1383

8 0x000055637df7f1a5 in slapd_do_all_nss_ssl_init (slapd_exemode=1, importexport_encrypt=0, s_port=10637,

ports_info=0x7ffefe08f5a0) at ldap/servers/slapd/main.c:2983

9 0x000055637df7573b in detach (slapd_exemode=1, importexport_encrypt=0, s_port=10637, ports_info=0x7ffefe08f5a0)

at ldap/servers/slapd/detach.c:79

10 0x000055637df7a6d7 in main (argc=5, argv=0x7ffefe08f6e8) at ldap/servers/slapd/main.c:960

}}}
It's understandable since now I'm not depending upon systemd. I'd think it's not a problem for customers. But we may need to start the process from gdb to debug some issues observed at the startup time. Could it be possible to support it? Or somehow work around it?

Interesting. It should be either:

  • Detecting you are still on a TTY and using the original svrcore prompt. You need to still be attached to the process here for this to work, so you need '-d 0' at the minimum.
  • Asking systemd, which should post a message via wall to all users to get the password.

In this case, I am surprised you didn't get the wall message.

Next time you try this, can you show me the contents of /var/run/systemd/ask-password/ask.< ns-slapd pid>, and the ls -l of /var/run/systemd/ask-password/

I want to see if the ask pass files were setup correctly.

Replying to [comment:12 firstyear]: [...] > > Interesting. It should be either: > > * Detecting you are still on a TTY and using the original svrcore prompt. You need to still be attached to the process here for this to work, so you need '-d 0' at the minimum. > * Asking systemd, which should post a message via wall to all users to get the password. > > In this case, I am surprised you didn't get the wall message. > > Next time you try this, can you show me the contents of /var/run/systemd/ask-password/ask.< ns-slapd pid>, and the ls -l of /var/run/systemd/ask-password/ > > I want to see if the ask pass files were setup correctly. Actually, what I did was ... 1) started DS with this command line. # /usr/sbin/ns-slapd -D /path/to/confdir -i /path/to/pidfile -d 0 Then, it hanged. 2) attached the process to gdb. Then, got this stacktrace. {{{ (gdb) bt #0 0x00007f3b19f65d20 in __nanosleep_nocancel () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f3b19f65bd4 in __sleep (seconds=0, seconds@entry=2) at ../sysdeps/unix/sysv/linux/sleep.c:138 #2 0x00007f3b1a47e5a1 in getPin (obj=<optimized out>, tokenName=<optimized out>, retry=<optimized out>) at systemd-ask-pass.c:361 #3 0x00007f3b1a47d5ea in getPin (ctx=0x56109dac69a0, tokenName=0x56109dabf1b1 "Internal (Software) Token", retry=<optimized out>) at cache.c:154 #4 0x00007f3b1b11b949 in pk11_GetPassword (wincx=0x0, retry=0, slot=0x56109dabf070) at pk11auth.c:517 #5 PK11_DoPassword (slot=0x56109dabf070, session=16777217, loadCerts=<optimized out>, wincx=0x0, alreadyLocked=0, contextSpecific=0) at pk11auth.c:589 #6 0x00007f3b1cfb9c8e in slapd_pk11_authenticate (slot=0x56109dabf070, loadCerts=1, wincx=0x0) at ldap/servers/slapd/security_wrappers.c:222 #7 0x00007f3b1cfc0016 in slapd_ssl_init () at ldap/servers/slapd/ssl.c:1383 #8 0x000056109cf041a5 in slapd_do_all_nss_ssl_init (slapd_exemode=1, importexport_encrypt=0, s_port=10637, ports_info=0x7ffe35fdce30) at ldap/servers/slapd/main.c:2983 #9 0x000056109cefa970 in detach (slapd_exemode=1, importexport_encrypt=0, s_port=10637, ports_info=0x7ffe35fdce30) at ldap/servers/slapd/detach.c:127 #10 0x000056109ceff6d7 in main (argc=7, argv=0x7ffe35fdcf78) at ldap/servers/slapd/main.c:960 }}} 3) Here's the ask file. {{{ # cat /var/run/systemd/ask-password/ask.17524 [Ask] PID=17524 Socket=/run/systemd/ask-password/sck.17524 AcceptCached=0 Echo=0 NotAfter=357210079142 Message=Enter PIN for Internal (Software) Token: }}} Please note that if I start the process from gdb, it still hangs... (gdb) run -D /etc/dirsrv/slapd-test1 -i /tmp/pid.txt -d 0 Starting program: /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-test1 -i /tmp/pid.txt -d 0 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Another question... I attached the hung process (pid=17524) to gdb, then I terminated it with "kill 17524". Then, the ask file and the socket still exist even though the process is gone. # cat /var/run/systemd/ask-password/ask.17524 [Ask] PID=17524 Socket=/run/systemd/ask-password/sck.17524 AcceptCached=0 Echo=0 NotAfter=357210079142 Message=Enter PIN for Internal (Software) Token: # file /run/systemd/ask-password/sck.17524 /run/systemd/ask-password/sck.17524: socket # ps -ef | egrep 17524 root 17591 14435 0 09:36 pts/7 00:00:00 grep -E --color=auto 17524 I guess it's ok to delete them by hand? (It'd be nice if it is automatically taken care... But unless we start it via systemd, it's difficult?) One more question... Does this prompt get timed out? If I leave it, it quits like this... Is this a common behavior? # start-dirsrv test1 Enter PIN for Internal (Software) Token: Job for dirsrv@jiji1.service failed. See "systemctl status dirsrv@jiji1.service" and "journalctl -xe" for details. Indeed systemd seems playing some magic...

Ok. If I launch ns-slapd from xterm (which was opened from the terminal), it gave me the message.
Broadcast message from root@<host>.<domain> (Mon 2016-04-04 16:23:04 PDT):

Password entry required for 'Enter PIN for Internal (Software) Token:' (PID 26762).
Please enter password with the systemd-tty-ask-password-agent tool!

And I could pass the password via systemd-tty-ask-password-agent.

systemd-tty-ask-password-agent

Enter PIN for Internal (Software) Token: ******

For some reason, I don't see the wall message on the terminals on F22 and 23.
nhosoi 3613 1 0 Mar19 tty2 00:12:02 /usr/libexec/gnome-terminal-server
nhosoi 3620 3613 0 Mar19 tty2 00:00:00 gnome-pty-helper
nhosoi 3621 3613 0 Mar19 pts/0 00:00:04 -bash

The wall command works on FC19 even though I did the same thing on the terminal... Odd

Hi William,

Your svrcore patch looks good and is ready to contribute to the upstream svrcore in Mozilla.

Here's the URL which lists the mozilla products. But svrcore is not treated as an independent component.
https://bugzilla.mozilla.org/enter_bug.cgi#h=otherProducts|

So, let's file it in NSS and ask the NSS people to review & check in, or move to the more appropriate component.
https://bugzilla.mozilla.org/enter_bug.cgi#h=dupes|NSS

Could you please file a mozilla bug with your patch and include me in the CC list? Thanks!

I have another request...

Could it be possible to file this message in the error log when it was not launched from systemd?
Please enter password with the systemd-tty-ask-password-agent tool!

Just in case I'm not the only one who does not receive the wall message...

Interestingly, "mesg" is not the issue. It just changes the group permission of the tty (and anyway, it's 'y' by default :). But in my case on "terminal", I cannot receive the message even on the same terminal the wall is executed... puzzled But the priority of the issue is low. And we don't want to spend much time on this... Thanks!

Replying to [comment:16 nhosoi]:

The wall command works on FC19 even though I did the same thing on the terminal... Odd

This may be a regression or feature of newer versions of gnome terminal.

Could you please file a mozilla bug with your patch and include me in the CC list? Thanks!

Done and cced as requested.

https://bugzilla.mozilla.org/show_bug.cgi?id=1262334

I have another request...

Could it be possible to file this message in the error log when it was not launched from systemd?
Please enter password with the systemd-tty-ask-password-agent tool!

Well, the prompt asking for the password is generated by svrcore itself. By the time we are there, it's well out of slapi code.

So either we need to log to slapi_log_error before we go into svrcore and say "you might need to call systemd-tty-ask-password-agent". Or we need a way to pass a communication callback into svrcore for it to call when needed. But that would be a bit of re-work, and I'm not sure of the gain.

Replying to [comment:17 firstyear]:

Could you please file a mozilla bug with your patch and include me in the CC list? Thanks!

Done and cced as requested.

https://bugzilla.mozilla.org/show_bug.cgi?id=1262334

Thank you! And I've just read the interesting conversation with ryan.sleevi@gmail.com. We are discussing in the team. If the project is abandoned and there is no problem to re-license it, we could have it just in fedora...

I have another request...

Could it be possible to file this message in the error log when it was not launched from systemd?
Please enter password with the systemd-tty-ask-password-agent tool!

Well, the prompt asking for the password is generated by svrcore itself. By the time we are there, it's well out of slapi code.

So either we need to log to slapi_log_error before we go into svrcore and say "you might need to call systemd-tty-ask-password-agent". Or we need a way to pass a communication callback into svrcore for it to call when needed. But that would be a bit of re-work, and I'm not sure of the gain.

Agh... A good point. No need to do re-work. This is for helping a poor user like me who could be just < 1%. I think logging "you might need to call systemd-tty-ask-password-agent" should be good enough. Thanks for the research!

Thank you! And I've just read the interesting conversation with ryan.sleevi@gmail.com. We are discussing in the team. If the project is abandoned and there is no problem to re-license it, we could have it just in fedora...

That's underway!

Well, the prompt asking for the password is generated by svrcore itself. By the time we are there, it's well out of slapi code.

So either we need to log to slapi_log_error before we go into svrcore and say "you might need to call systemd-tty-ask-password-agent". Or we need a way to pass a communication callback into svrcore for it to call when needed. But that would be a bit of re-work, and I'm not sure of the gain.

Agh... A good point. No need to do re-work. This is for helping a poor user like me who could be just < 1%. I think logging "you might need to call systemd-tty-ask-password-agent" should be good enough. Thanks for the research!

Okay. I think that the log of "you may need to call systemd-tty-ask-password-agent" is a simple, but not perfect solution. I'll add this to the next patch.

Add notification to log for request to forward to systemd-ask-pass.
0001-Ticket-48450-Systemd-password-agent-support.patch

As requested, we now send a message to the log, detailing that you may need to call the systemd tool to provide the password.

{{{
[08/Apr/2016:09:44:24.576939699 +1000] 389-Directory/1.3.5.1 B2016.098.2327 starting up
[08/Apr/2016:09:44:24.908893371 +1000] convert_pbe_des_to_aes: Checking for DES passwords to convert to AES...
[08/Apr/2016:09:44:24.921228761 +1000] convert_pbe_des_to_aes - No DES passwords found to convert.
[08/Apr/2016:09:44:24.934509478 +1000] slapd started. Listening on All Interfaces port 389 for LDAP requests
[08/Apr/2016:09:50:56.401999354 +1000] SSL alert: Sending pin request to SVRCore. You may need to run systemd-tty-ask-password-agent to provide the password.
}}}

commit ff2b8470fd99d8bb6db05e485d40ed5178b4fd40
Total 6 (delta 4), reused 0 (delta 0)
To ssh://git.fedorahosted.org/git/389/ds.git
472a96b..6186290 master -> master

The patches:
0001-Ticket-5-Integrate-asan-support-for-code-quality-che.patch​ through
0007-Ticket-12-update-spec-to-match-fedora-4.1.0.patch​ added
look good to me.
Ack.

Due to the lack of movement from the systemd team to fix the issues in permission of ask-password. I am re-opening for a wrapper that will fix this at server start.

commit e6b4892
Writing objects: 100% (7/7), 1.54 KiB | 0 bytes/s, done.
Total 7 (delta 5), reused 0 (delta 0)
To ssh://git.fedorahosted.org/git/389/ds.git
802224f..e6b4892 master -> master

Hi William,

Could you investigate this issue? It is a test blocker...
https://bugzilla.redhat.com/show_bug.cgi?id=1316580#c11

Pushed to master on behalf of William.
commit 632e5b6

Thanks for the patch, William.

Metadata Update from @firstyear:
- Issue assigned to firstyear
- Issue set to the milestone: 1.3.5.0

7 years ago

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/389ds/389-ds-base/issues/1772

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.

Metadata Update from @spichugi:
- Issue close_status updated to: wontfix (was: Fixed)

3 years ago

Login to comment on this ticket.

Metadata