Ticket #1104 (closed task: fixed)

Opened 13 months ago

Last modified 12 months ago

Expanding the list of "Hardened Packages"

Reported by: halfie Owned by:
Priority: major Keywords: security,hardening,packaging,guidelines
Cc: Blocked By:
Blocking:

Description

phenomenon

http://fedoraproject.org/wiki/Hardened_Packages page mentions that "FESCo requires some packages to use PIE and relro hardening by default."

It would be great if this list could be expanded to include packages which are at comparatively more risk of being exploited (locally or remotely).

Such packages will typically include various system daemons, network daemons and network enabled applications.

(Implementing this will require changes to "Packaging Guidelines")

background analysis

Lot of network daemons are already using PIE and RELRO (e.g. httpd, MariaDB). So a natural question is why aren't packages in same "network daemons" class like PostgreSQL, Dovecot and MongoDB aren't being hardened?

implementation recommendation

I believe that hardening flags should be turned on (by default) for all packages which are at the risk of being exploited.

"Packaging Guidelines" say that "Other packages may enable the flags at the maintainer's discretion."

Thinking from a security perspective, I find "Hardening flags can be disabled for other packages at the maintainer's discretion provided enough justification is given to FESCo" to be more appropriate.

For a start, packages from the following RPM groups can be targeted,

"Applications/CGI", "Network/Daemons", "Applications/Communications", 
"Applications/Internet", "System Environment/Daemons", "Applications/Databases"

Change History

comment:1 Changed 13 months ago by mitr

http://fedoraproject.org/wiki/Packaging:Guidelines#PIE also says:

If your package meets any of the following criteria you MUST enable the PIE compiler flags:

    Your package is long running. This means it's likely to be started and keep running until the machine is rebooted, not start on demand and quit on idle. 

    Your package has suid binaries, or binaries with capabilities. 

    Your package runs as root.

which is a superset of the criteria suggested in "phenomenon" above.

I read the guideline as "if a package matches the above criteria OR is on http://fedoraproject.org/wiki/Hardened_Packages , it MUST use PIE - but it could definitely be clarified. Anyway, this is probably up to FPC.

comment:2 Changed 13 months ago by notting

Note that RPM groups are almost entirely obsolete - they don't make a good guide here.

comment:3 Changed 13 months ago by kevin

I'm afraid "all packages which are at the risk of being exploited" is pretty broad and could include most everything.

I'd rather suggest adding specific packages to the must list, or figuring out a better critera that maintainers could use to decide.

comment:4 follow-up: ↓ 5 Changed 13 months ago by halfie

I'm afraid "all packages which are at the risk of being exploited" is pretty broad and could include most everything.


You are right. I think packages which meet the criteria described in "comment:1" by mitr are suitable candidates for hardening.

However lot of packages (like PostgreSQL, Dovecot and MongoDB) which meet the criteria are still not being hardened. I am not sure if I should file bugs against individual packages *or* if FESCo can expand its list to include mentioned packages as well as similar packages (the latter option being more preferable).

I can work towards identifying more packages which could use hardening.


Note that RPM groups are almost entirely obsolete - they don't make a good guide here.


Yes, people in #fedora-devel told me the same. I am still looking for better ways to identify suitable packages. Suggestions are welcome!

Last edited 13 months ago by halfie (previous) (diff)

comment:5 in reply to: ↑ 4 Changed 13 months ago by tmraz

Replying to halfie:

However lot of packages (like PostgreSQL, Dovecot and MongoDB) which meet the criteria are still not being hardened. I am not sure if I should file bugs against individual packages *or* if FESCo can expand its list to include mentioned packages as well as similar packages (the latter option being more preferable).

If you think that some package meets the current criteria and it is not hardened yet, I think that filling a bug on the package cannot do any harm. So feel free to identify such packages and fill the bugs.

comment:6 Changed 13 months ago by halfie

https://wiki.ubuntu.com/Security/Features says "PIE on x86_64 does not have the same penalties, and will eventually be made the default, but more testing is required."

What about Fedora taking the lead on this one?

comment:7 Changed 13 months ago by mmaslano

I guess PIE has some impact on performance. Therefore I'd rather use PIE on limited list of packages. Databases might be a good addition into the current group.

If you have a list what you'd like to see there, then please add it here.

comment:8 follow-up: ↓ 9 Changed 13 months ago by tmraz

PIE disables use of prelink - so this is another performance impact on startup. On the other hand we should evaluate the impact of non-prelinked vs. prelinked startup time on modern computers, maybe it is no longer much relevant.

comment:9 in reply to: ↑ 8 Changed 13 months ago by halfie

Replying to tmraz:

PIE disables use of prelink - so this is another performance impact on startup. On the other hand we should evaluate the impact of non-prelinked vs. prelinked startup time on modern computers, maybe it is no longer much relevant.


Please see http://people.redhat.com/~gmurphy/files/pie.odt for such an evaluation.

In short. "... the average delay of a PIE application over a non-PIE application was significantly below perceivable threshold."

comment:10 follow-up: ↓ 13 Changed 13 months ago by tmraz

I'm not sure that in the study above the non-PIE binaries were prelinked or not. Also it would be more interesting to see the result for bigger desktop applications like LibreOffice?, Firefox, Evolution.

comment:11 Changed 13 months ago by halfie

Replying to mmaslano:

I guess PIE has some impact on performance. Therefore I'd rather use PIE on limited list of packages. Databases might be a good addition into the current group.

ftp://ftp.inf.ethz.ch/doc/tech-reports/7xx/766.pdf mentions an average overhead of 3.6% on AMD64 (x64) which is not too bad.

However, the overhead is too high on x86 (32-bit) in my opinion. Is it possible to use different compilation flags on different platforms?

If you have a list what you'd like to see there, then please add it here.

I am working on it and it should be somewhat ready by early next week.

comment:12 follow-up: ↓ 14 Changed 13 months ago by mitr

Would it make sense to move the dicussion to fedora-devel to get a wider audience and more comments, and then return to the FESCo trac (and is it FESCo or FPC?) to discuss a specific policy proposal?

comment:13 in reply to: ↑ 10 Changed 13 months ago by halfie

Replying to tmraz:

I'm not sure that in the study above the non-PIE binaries were prelinked or not. Also it would be more interesting to see the result for bigger desktop applications like LibreOffice?, Firefox, Evolution.

http://en.wikipedia.org/wiki/Prelink mentions "Jakub Jelínek points out that position independent executables ignore prelinking on Red Hat Enterprise Linux and Fedora Core, and recommends that network and SUID programs be built PIE to facilitate a more secure environment."

I agree with your suggestion about measuring the impact of PIE on bigger Desktop applications. I will work on it (after the list is done).

comment:14 in reply to: ↑ 12 Changed 13 months ago by halfie

Replying to mitr:

Would it make sense to move the dicussion to fedora-devel to get a wider audience and more comments, and then return to the FESCo trac (and is it FESCo or FPC?) to discuss a specific policy proposal?

Agreed. I have posted the proposal to fedora-devel, http://lists.fedoraproject.org/pipermail/devel/2013-March/180827.html

comment:15 Changed 13 months ago by halfie

Here is a list of packages (daemons only) which are (possibly) violating packaging guidelines with respect to hardening.

http://dl.dropbox.com/u/1522424/probable-violations-F19.xls

http://dl.dropbox.com/u/1522424/probable-violations-F19.csv

(Please note that this list was generated by a program and might be buggy!)

Last edited 13 months ago by halfie (previous) (diff)

comment:16 follow-up: ↓ 19 Changed 13 months ago by halfie

To summarize,

  1. Many packages which meet the hardening criteria (described at http://fedoraproject.org/wiki/Packaging:Guidelines#PIE) are not being hardened.

For example, postgresql which is "long running" is not being built with PIE flags. This is a clear violation of the packaging guidelines.

  1. After many discussions, I believe that the existing "hardening criteria" is good enough but I would like to see the hardening policy actually being enforced.

I have posted tools and data to help in doing this (in an almost automated fashion).

  1. For postgresql, I have opened a bug https://bugzilla.redhat.com/show_bug.cgi?id=947022 requesting hardening to be enabled but so far I haven't got any response.

In my opinion, any bugs reporting violation of packaging guidelines should be looked at in a prompt fashion.

  1. Opening bugs against tons of packages (possibly) manually is a bit tedious. Ideas on how to solve this bit of the problem are welcome!
  1. "hard data that demonstrate the impact, if any, in a situation relevant to Fedora (in particular, taking into account prelink as it is deployed by default) would be very welcome but is not a strict requirement".

Please see comment:10. In short, "... the average delay of a PIE application over a non-PIE application was significantly below perceivable threshold". Please note that "position independent executables ignore prelinking on Red Hat Enterprise Linux and Fedora Core", so the numbers presented in the report linked to in comment:10 should be good as they are).

Additionally, I have posted benchmarking numbers for Firefox ("big Desktop application") at https://lists.fedoraproject.org/pipermail/devel/2013-April/181170.html. In short, post-hardening numbers are same as before!

  1. Currently, I am working on getting more benchmarking numbers out (in order to address Jakub's concerns, see https://lists.fedoraproject.org/pipermail/devel/2013-April/181171.html). This is being done to study the possibility of bringing even more packages under hardening criteria in future.

comment:17 follow-ups: ↓ 18 ↓ 20 Changed 13 months ago by pwouters

Remember when Sun said "the network is the computer"? All packages handle untrusted user input and network input.

I'm strongly in favour of dropping prelink, and enabling hardening per default. Especially if the only price we pay is "a few seconds of waiting on initial loading of large applications". libreoffice is so slow to start, adding three, five or ten seconds isn't going to make a difference.

Short of that, I'd say any every daemon or service started through systemd/(x)inetd/ should enable hardening.

comment:18 in reply to: ↑ 17 Changed 13 months ago by halfie

Replying to pwouters:

I'm strongly in favour of dropping prelink, and enabling hardening per default. Especially if the only price we pay is "a few seconds of waiting on initial loading of large applications". libreoffice is so slow to start, adding three, five or ten seconds isn't going to make a difference.

Well, the difference in loading time (for "large Desktop applications") does not even amount to a second (according to my tests which may be flawed).

Please see https://gist.github.com/kholia/807f463235ab6eacac96 for Gimp benchmarks.

comment:19 in reply to: ↑ 16 ; follow-up: ↓ 21 Changed 13 months ago by mitr

Thanks for the summary.

Replying to halfie:

  1. Many packages which meet the hardening criteria (described at http://fedoraproject.org/wiki/Packaging:Guidelines#PIE) are not being hardened.

I think that's a subset of point 4 and possibly point 2.

  1. After many discussions, I believe that the existing "hardening criteria" is good enough but I would like to see the hardening policy actually being enforced.

I have posted tools and data to help in doing this (in an almost automated fashion).

That would be great; could this be integrated into fedora-review (and maybe rpmlint)?

  1. For postgresql, I have opened a bug https://bugzilla.redhat.com/show_bug.cgi?id=947022 requesting hardening to be enabled but so far I haven't got any response.

In my opinion, any bugs reporting violation of packaging guidelines should be looked at in a prompt fashion.

I'm sorry to say I'm much more worried about unhandled crash reports (which may indicate possibly exploitable bugs) than missing hardening; that's not to say the bug shouldn't be handled - just that I don't think some kind of forceful prioritization of these bugs makes sense.

  1. Opening bugs against tons of packages (possibly) manually is a bit tedious. Ideas on how to solve this bit of the problem are welcome!

There certainly are automated tools; is python-bugzilla the best one nowadays?

  1. "hard data that demonstrate the impact, if any, in a situation relevant to Fedora (in particular, taking into account prelink as it is deployed by default) would be very welcome but is not a strict requirement".

Please see comment:10. In short, "... the average delay of a PIE application over a non-PIE application was significantly below perceivable threshold".

That measures startup, not long-term performance impact. (PIE affects both, prelink optimizes primarily startup.)

Please note that "position independent executables ignore prelinking on Red Hat Enterprise Linux and Fedora Core", so the numbers presented in the report linked to in comment:10 should be good as they are).

So the PIE numbers are relevant, but the non-PIE ones are not?

Additionally, I have posted benchmarking numbers for Firefox ("big Desktop application") at https://lists.fedoraproject.org/pipermail/devel/2013-April/181170.html. In short, post-hardening numbers are same as before!

It might be a 5% performance difference, which is quite big, but it in the unexpected direction... Surprising.

  1. Currently, I am working on getting more benchmarking numbers out (in order to address Jakub's concerns, see https://lists.fedoraproject.org/pipermail/devel/2013-April/181171.html). This is being done to study the possibility of bringing even more packages under hardening criteria in future.

(FWIW, I'm looking for hard data as much preferable to opinions of the kind "I always disable $foo because it is slow". I don't currently have a specific threshold in mind that would sway me pro/con any decision.)

So in summary, your proposal is to keep the existing guidelines and enforce them better? I'm in any case fine with that.

comment:20 in reply to: ↑ 17 Changed 13 months ago by mitr

Replying to pwouters:

Remember when Sun said "the network is the computer"? All packages handle untrusted user input and network input.

I'm strongly in favour of dropping prelink, and enabling hardening per default. Especially if the only price we pay is "a few seconds of waiting on initial loading of large applications".

That's the price for prelink (and btw. preferable overall latency is 0.1s), not for PIE, which is an ongoing cost (... for code located in the executable and not in libraries).

libreoffice is so slow to start, adding three, five or ten seconds isn't going to make a difference.

Actually libreoffice seems not to be getting slower, unlike the rest of the system... On my desktop even a cold start of libreoffice takes less than a second, which is curiously faster than gnome-documents as far as I can see (using a clock and my eyes, not at all scientific).

For the substance of the proposal, see my recent mail on fedora-devel about mutable/static ASLR - after thinking about this more I can't see prelink as a significant problem (... which is not making the decision easier - as argued above PIE is probably the decision that has larger cost).

comment:21 in reply to: ↑ 19 ; follow-up: ↓ 23 Changed 13 months ago by halfie

Replying to mitr:

  1. For postgresql, I have opened a bug https://bugzilla.redhat.com/show_bug.cgi?id=947022 requesting hardening to be enabled but so far I haven't got any response.

In my opinion, any bugs reporting violation of packaging guidelines should be looked at in a prompt fashion.

I'm sorry to say I'm much more worried about unhandled crash reports (which may indicate possibly exploitable bugs) than missing hardening; that's not to say the bug shouldn't be handled - just that I don't think some kind of forceful prioritization of these bugs makes sense.

I should not have used the word "prompt". I meant to say that such bugs should be handled in a reasonable amount of time instead of just being ignored and moved from one release to another.

  1. "hard data that demonstrate the impact, if any, in a situation relevant to Fedora (in particular, taking into account prelink as it is deployed by default) would be very welcome but is not a strict requirement".

Please see comment:10. In short, "... the average delay of a PIE application over a non-PIE application was significantly below perceivable threshold".

That measures startup, not long-term performance impact. (PIE affects both, prelink optimizes primarily startup.)

You are right. For performance impact please see the ETH paper I have already linked to in comment:11


So in summary, your proposal is to keep the existing guidelines and enforce them better? I'm in any case fine with that.

Yes.

Last edited 13 months ago by halfie (previous) (diff)

comment:22 Changed 12 months ago by halfie

Is this ticket ready now to be discussed in the upcoming FESCo meeting?

comment:23 in reply to: ↑ 21 Changed 12 months ago by mitr

Replying to halfie:

Replying to mitr:

So in summary, your proposal is to keep the existing guidelines and enforce them better? I'm in any case fine with that.

Yes.

Thanks, closing.

For the other discussed aspects (PIE everywhere, dropping prelink), if you are interested please open a separate ticket with a proposal.

comment:24 Changed 12 months ago by mitr

  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.