Ticket #940 (closed task: wontfix)

Opened 20 months ago

Last modified 17 months ago

The tmp-on-tmpfs feature should be disabled by default

Reported by: rjones Owned by:
Priority: major Keywords: meeting
Cc: gholms, jstanley, jdulaney, fche, mattdm Blocked By:
Blocking:

Description

The tmp-on-tmpfs feature (https://fedoraproject.org/wiki/Features/tmp-on-tmpfs) was approved for Fedora 18. However it is problematic and should not be enabled by default for the reasons outlined below.

  • There are no performance benefits since even if backed by a disk, the kernel will cache /tmp in memory as required. The originator of the feature did no testing to demonstrate any performance benefits. Benefits are claimed for SSDs, power etc but there are no facts to back this up.
  • On small memory systems (ie. VMs) the default is half of available memory which can be too small for ordinary workloads.

Change History

comment:1 follow-up: ↓ 16 Changed 20 months ago by lennart

Well, this request is without any merit.

Performance benefits have been shown aplenty:

http://slacy.com/blog/2008/08/tmpfs-vs-ext3-performance-on-large-file-sets/

http://forums.bukkit.org/threads/harddrive-vs-ramdisk-vs-tmpfs-benchmark.2710/

https://lists.fedoraproject.org/pipermail/devel/2012-June/167910.html

and so on.

There is exactly one bug regarding tmpfs tracked in bugzilla right now:

https://bugzilla.redhat.com/show_bug.cgi?id=826015

And that one should be easy to fix. So it appears everything is perfectly in order.

If the default size of /tmp is too small for some systems then we can certainly increase it. However, so far we have not seen any bug report about this.

My suggestion for rjones is to first file bugs, and see what we can do about them. Right now almost *everything* points in the direction that tmp-on-tmps is very stable, and with much less problems than many naysayers assumed it would be. I find it very uncordial to escalate this immediately to FESCO rather then filing bugs first like we do it for most other issues.

comment:2 Changed 20 months ago by djdelorie

As an owner of the "target system" (SSD boot, 24GB ram), I do *not* want /tmp in RAM. I have seen no performance issues with /tmp on SSD (or on disk, before) as the default disk caching works quite well. Also, I bought lots of [expensive] RAM because I need the RAM for programs, not storage, and my /tmp needs still often exceed available RAM. I will not "just add swap" because that much swap kills performance worse than regular disk I/O. I also prefer that /tmp not be cleaned just because my machine unexpectedly rebooted. Just because a file is "temporary" does not mean it is unimportant, or that re-procuring it would be easy.

The arguments about setting TMPDIR or using /var/tmp are a farce. The issues are the same regardless of where your temporary files are stored.

Aside: Reading through the original Features and Talk pages, there is absolutely no record of any actual real-world performance testing or user experience surveys done on this feature. For that reason alone, FESCO should have deferred a decision. Sweeping changes which have the potential to cripple a system should require more than "we think this is a good idea" before deciding.

comment:3 Changed 20 months ago by lennart

djdelorie, please understand that Fedora is supposed to provide good defaults to majority of people, but that this doesn't necessarily mean that everything is right out of the box for everybody. If you really don't want tmpfs on /tmp, then just disable it: "systemctl mask tmp.mount", but there's no need to make that a Fedora default. Other than that, tmpfs is faster, more appropriate for the purpose and from all we know (i.e. Bugzilla reports) doesn't create much problems, hence I see no reason at all to turn this default off.

comment:4 follow-up: ↓ 5 Changed 20 months ago by djdelorie

IMHO Fedora should provide *safe* defaults for the majority. The massive thread about this "feature" indicates that it's riskier to enable tmpfs than to not, and the large number of vocal and angry feedback indicates that it pleases far less than the majority. This, to me, indicates a failure in the decision process itself, not just the feature.

As for "tmpfs is faster", I disagree. Perhaps a synthetic benchmark could show some benefit, but in real life, I see no difference. And F18 hasn't released yet, so how would there be bug reports yet? Were there bug reports before, which prompted this change?

And I really hate features that are controlled by some obscure black magic command line, especially systemd ones.

comment:5 in reply to: ↑ 4 Changed 20 months ago by lennart

Replying to djdelorie:

IMHO Fedora should provide *safe* defaults for the majority. The massive thread about this "feature" indicates that it's riskier to enable tmpfs than to not, and the large number of vocal and angry feedback indicates that it pleases far less than the majority. This, to me, indicates a failure in the decision process itself, not just the feature.

Well, we shouldn't be taken hostage by noise on mailing lists. I kinda prefer to look at raw data, such as bug reports and so on. And honestly, a *single* (!) bug report that has been filed until now (that is even easy to fix) is really nothing that makes me believe there is an actual problem here...

As for "tmpfs is faster", I disagree. Perhaps a synthetic benchmark could show some benefit, but in real life, I see no difference. And F18 hasn't released yet, so how would there be bug reports yet? Were there bug reports before, which prompted this change?

Cool, so hard numbers don't convince you. Technical reasons don't convince you. You suggest making these decisions based on djdelorie's feelings would be the better option. Nah, thanks.

Well, if you think that the testing scheme/QA of Fedora doesn't work, and needs revisiting, then please bring that up on the mailing lists. If you believe that the testing F18 is getting before the release is not sufficient, then this is something that has to be dealt with in general, not just specificly to this one feature.

Also, do you really honestly believe we should add new features only if there are bug reports asking for them? Yikes. Then we'd never innovate...

And I really hate features that are controlled by some obscure black magic command line, especially systemd ones.

Ah, so this is about hating systemd? is that something like the new Godwin's law? Can you please throw in some PulseAudio hatred as well? That definitely helps your case...

comment:6 Changed 20 months ago by gholms

  • Cc gholms added

comment:7 follow-up: ↓ 22 Changed 20 months ago by rjones

Do you know that bonnie and iozone are designed to defeat the RAM cache and measure the underlying disk performance? They are entirely the wrong way to measure this.

I also received an interesting private email from a Debian maintainer. Some highlights:

comment:8 Changed 20 months ago by brummbq

For me, /tmp ond tmpfs sounds like a bad idea, too. Here is an exhaustive summary from the debian mailinglist (he presents also alternatives): http://lists.debian.org/debian-devel/2012/06/msg00311.html

I'm still missing some real world benchmarks showing the benefits of this feature. The fedora feature page is rather superficial here.

comment:9 Changed 20 months ago by sochotni

I"ve been running tmp on tmpfs for a few years. There were a few hiccups which normal users would *not* be able to figure out though. So I'd join the "revert" club.

Lack of bugreports is not an good enough reason to keep it. F18 has not been released yet so only developers run F18 and even then only small parts of the system are tested most likely.

lennar when you say:

... Fedora is supposed to provide good defaults to majority of people

I believe what you meant to say was this:

... Fedora is supposed to provide *safe/sane* defaults to majority of people

tmp on tmpfs will *maybe* bring measurable performance improvements on some systems, but will cause weird, hard to debug problems for common desktop use cases.

I'd love to have this as easy opt-in feature, but it should *not* be opt-out. With opt-in, we can provide text describing the feature and advantages and possible pitfalls in release notes.

comment:10 Changed 20 months ago by bdwheele

There are big problems with this feature:

  • /tmp is now significantly smaller than it was before. Fedora has traditionally placed everything into a single partition so the possible temp space could be in the hundreds of GB. Even on my system with a 32G root and the remainder being data, I've got 21G available for /tmp. Under tmpfs I have 4G.
  • /tmp size is directly proportional to memory size. Two default installations will give very large differences in the size of /tmp: my workstation would have 4G and my laptop would have 1.5G. On an extreme case, if RHEL7 picks this up, my server /tmp space would range from 4G to 60G depending on which machine is being used. Prepare to see crap "howto" articles about how fedora requires more memory if you want a larger /tmp (or to write dvds, or whatever)
  • The "solution" to the issues above is to keep everything in /var/tmp. That's a red herring. Patching a ton of packages fixes it for fedora, but doesn't fix it for real. External packages will continue to write to /tmp as will most users because that's the way it has worked for 40 years - and they will fail in unreproducable ways because of the limited space. Under this solution, the rule for storing temporary files now depends on what size you expect the temporary data to be ... and since it may not be known at the start, you're going to stuff everything into /var/tmp which defeats the purpose of this feature entirely.
  • Related to the solution is that every program should be using TMPDIR instead of hardcoding /tmp. Yeah, well, good luck with that. The faculty I deal with aren't going to change their ways and desktop users aren't going to know how to do that -- and even with a howto its going to put everything in /var/tmp because it can't know what size the data is going to be in advance.

In short, this is a feature with no real benefits and a lot of changes and pitfalls. If there is a need for some super-fast-secure-must-delete-at-power-off storage, then put it somewhere other than /tmp.

As a long time RHL/Fedora user and professional RHEL admin, I say +1 to reverting this feature. Make it opt-in and track the bugs (and # of users!) and re-evaluate it at a later date.

comment:11 Changed 20 months ago by sandeen

I have no real horse in the race for this as a feature, but seeing filesystem benchmarks piqued my interest. As one of the filesystem guys for Fedora, I'll just agree that bonnie++ and iozone benchmarks are largely irrelevant to this question, unless most users use /tmp for bonnie++ and iozone testing purposes.

If one of the stated goals was to speed up the system by changing how /tmp is stored (edit: feature says it "makes things a bit faster"), then real-world _system_ performance with and without /tmp on tmpfs needs to be measured, not some synthetic benchmark which has nothing to do with typical /tmp IO patterns. I'd find some heavy users of /tmp and measure them with and without the change.

The other proposed benefits can be proved or disproved with measurement as well; for power, there are tools such as The Linux Battery Life Toolkit which might give a quantifiable answer. For IO, iostat numbers for /tmp on tmpfs can be collected over time, to see how much typical Fedora systems actually write to /tmp and quantify that goal as well. Hard numbers showing real benefits might add weight to the argument in favor of the feature.

Last edited 20 months ago by sandeen (previous) (diff)

comment:12 Changed 20 months ago by tmraz

  • Keywords meeting added

comment:13 follow-up: ↓ 23 Changed 20 months ago by rjones

Since I'm actually on holiday when this meeting comes up (on a non-standard day), here is what I would say if I was at the meeting. I will try to be there, but if not, please someone copy this into IRC:

The last meeting (http://meetbot.fedoraproject.org/fedora-meeting/2012-04-02/fesco.2012-04-02-17.00.log.html) was very close on this topic: 5 for, 3 against.

Firstly, a lot of weight was placed on Debian's move to tmpfs, but since that meeting this has changed: Debian is NOT going to enable this by default (nor is Ubuntu).

Secondly, I don't think enough consideration was given of the virtualization case at the original meeting. VMs are often packed in with small RAM (512 MB) and small disks (10 GB). This means that tmpfs would be very limited: 256 MB (default) to 2.25 GB (max) before complete system failure. The current default is around 6 GB on that size of system.

Thirdly, there is no proof of performance or power gains. See Eric Sandeen's full response in the ticket. Given that, what is this change for exactly?

Fourthly, there is a lot of software outside Fedora (eg. Oracle installer) which cannot be fixed. Plus a lot of users who are used to using /tmp as scratch space for arbitrary sized files.

comment:14 Changed 20 months ago by mattdm

+1 for giving more consideration to the virtualization case. Even when systems have nominally adequate RAM, the host may be oversubscribed. RAM is _the_ precious resource in the virtualization world.

comment:15 Changed 20 months ago by notting

From 2012-09-05 FESCo meeting:

AGREED: Discussion of this deferred until some point after alpha where this has had a chance to be more widely tested (+:7, -1:1, 0:1)

comment:16 in reply to: ↑ 1 Changed 20 months ago by mganisin

Replying to lennart:

If the default size of /tmp is too small for some systems then we can certainly increase it. However, so far we have not seen any bug report about this.

So far we have not seen any bug report for issue with /tmp outside of tmpfs.

My suggestion for rjones is to first file bugs, and see what we can do about them. Right now almost *everything* points in the direction that tmp-on-tmps is very stable, and with much less problems than many naysayers assumed it would be. I find it very uncordial to escalate this immediately to FESCO rather then filing bugs first like we do it for most other issues.

My suggestion for feature owner is to first file bugs against /tmp on disk and see if those are real issues. Right now almost *everything* points to rock solid and very stable /tmp on disk *as is* with *none* problem pointed out. I find it very insincere to implement features ignoring broad audience rather then doing a homework and provide *real* benefits like we do it for most other issues.

At the moment there is 1 bug for tmp-on-tmpfs vs 0 bugs for tmp-on-disk. It seems to be quite easy to identify winner.

comment:17 follow-ups: ↓ 18 ↓ 25 Changed 20 months ago by notting

There are also no bugs on libc5 in Fedora, so it's obviously the better choice for a libc, as well. An argument that consists entirely of "we must have no change ever" is no argument at all.

We do have to support both local filesystem /tmp and /tmp-on-tmpfs; we have no choice. It therefore behooves us to do the proper level of testing, bug filing, and fixing, and then use that information to make an informed choice on defaults, rather than polemics.

comment:18 in reply to: ↑ 17 ; follow-up: ↓ 19 Changed 20 months ago by mganisin

Replying to notting:

There are also no bugs on libc5 in Fedora, so it's obviously the better choice for a libc, as well.

Next time seriously please. :(

An argument that consists entirely of "we must have no change ever" is no argument at all.

Right. And an argument that consists entirely of "we must have a change everytime" is no argument at all as well.

We do have to support both local filesystem /tmp and /tmp-on-tmpfs;

No complain here, this ticket is about _default_.

we have no choice.

Please stop polemics.

It therefore behooves us to do the proper level of testing, bug filing, and fixing, and then use that information to make an informed choice on defaults,

Both sides have to comply same rules, if you want some meaningful output from opponents you have to want same from advocate. This is what I have asked for.

There are already many complains raised, you can find them in relevant mail thread, you can find it in already mentioned summary made for debian. Are they ignored?

rather than polemics.

Right. Howgh.

comment:19 in reply to: ↑ 18 ; follow-up: ↓ 21 Changed 19 months ago by notting

If you honestly think that me calling for public testing and pushing issues into bugzilla (rather than anecdotes on a mailing list) before making a decision on a default is not being serious, and is polemics... not sure we can have a meaningful conversation.

comment:20 Changed 19 months ago by hpa

The premise that "tmpfs is memory, filesystems is disk" is just plain false.

If the system is installed correctly (i.e. with a sanely sized swap area), tmpfs is *swap*. Similarly, filesystem data is kept in memory when there is memory to spare.

The big difference is that tmpfs doesn't have to worry about post-boot consistency, and so it can provide substantially better performance -- sometimes dramatically so.

If there are use cases where tmpfs (with a sanely sized swap area) provides bad performance, that is a bug and should be treated as one.

comment:21 in reply to: ↑ 19 Changed 19 months ago by mganisin

Replying to notting:

If you honestly think that me calling for public testing and pushing issues into bugzilla (rather than anecdotes on a mailing list) before making a decision on a default is not being serious, and is polemics... not sure we can have a meaningful conversation.

We are on same boat here. Where is a bug list of issues and/or improvement which has to be covered by this feature?

comment:22 in reply to: ↑ 7 ; follow-up: ↓ 24 Changed 19 months ago by lennart

Replying to rjones:

Do you know that bonnie and iozone are designed to defeat the RAM cache and measure the underlying disk performance? They are entirely the wrong way to measure this.

Well, every single benchmark you find on the interwebs will tell you that tmpfs is faster by a wide margin. You can certainly dismiss the tools used, but they'll all tell you the same story.

  • it breaks many applications. He mentions: sort [sic], k3b, midnight commander, debdelta, video streaming in an unspecified browser, fsarchiver.

Well, things can be fixed. And so far we have only one reported issue in rhbz. Maybe we should focus on our own bugs, instead of staying stuck in fear of bugs in Debian. And honestly even 6 more bugs doesn't sound like much to me.

comment:23 in reply to: ↑ 13 ; follow-up: ↓ 26 Changed 19 months ago by lennart

Replying to rjones:

Since I'm actually on holiday when this meeting comes up (on a non-standard day), here is what I would say if I was at the meeting. I will try to be there, but if not, please someone copy this into IRC:

The last meeting (http://meetbot.fedoraproject.org/fedora-meeting/2012-04-02/fesco.2012-04-02-17.00.log.html) was very close on this topic: 5 for, 3 against.

Firstly, a lot of weight was placed on Debian's move to tmpfs, but since that meeting this has changed: Debian is NOT going to enable this by default (nor is Ubuntu).

Debian is a very conservative distro. I would not expect them to be the first ones to adopt this. And Ubuntu just does whatever Debian does, they can't maintain their own changes.

Secondly, I don't think enough consideration was given of the virtualization case at the original meeting. VMs are often packed in with small RAM (512 MB) and small disks (10 GB). This means that tmpfs would be very limited: 256 MB (default) to 2.25 GB (max) before complete system failure. The current default is around 6 GB on that size of system.

Honestly, I actually believe tmpfs on /tmp is a great thing for virtualization/containers, as this allows us to run a lot of container instances from the same image or even root dir. And you actually want small /tmp for VMs, to limit their impact, and so that they can share the common swap space as backend. Running a lot of VMs with a single swap as backing store allows you to much nicer overcommit things than it would be to run them with big explicit file systems each.

Thirdly, there is no proof of performance or power gains. See Eric Sandeen's full response in the ticket. Given that, what is this change for exactly?

Ah, but you do acknowledge the speed gains now?

Fourthly, there is a lot of software outside Fedora (eg. Oracle installer) which cannot be fixed. Plus a lot of users who are used to using /tmp as scratch space for arbitrary sized files.

Well, we shouldn't let us being taken hostage by 3rd party sw (closed source even), and not do what is technically right.

comment:24 in reply to: ↑ 22 Changed 19 months ago by mitr

Replying to lennart:

Well, every single benchmark you find on the interwebs will tell you that tmpfs is faster by a wide margin. You can certainly dismiss the tools used, but they'll all tell you the same story.

(Careful with the quantifiers: http://lists.debian.org/debian-devel/2012/06/msg00107.html )

  • it breaks many applications. He mentions: sort [sic], k3b, midnight commander, debdelta, video streaming in an unspecified browser, fsarchiver.

Well, things can be fixed. And so far we have only one reported issue in rhbz. Maybe we should focus on our own bugs, instead of staying stuck in fear of bugs in Debian. And honestly even 6 more bugs doesn't sound like much to me.

Well, so now you (== feature owners) know about about 6 more bugs, and have no specific reason to assume that the same bugs wouldn't happen on Fedora, and in the April 2 meeting you promised to "fix whatever needs fixing". Could you, please, proactively verify whether these cases apply to Fedora, and fix them in Fedora if they do apply?

You own the quality of this feature. You can of course point at others because they haven't done the research for you, but users won't care whose fault it is "it works on Ubuntu/Windows").

Replying to lennart:

Well, we shouldn't let us being taken hostage by 3rd party sw (closed source even), and not do what is technically right.

The recent discussions about attractiveness of the Linux/Linux desktop platform to 3rd-party ISVs seem to point in the opposite direction: making the platform stable is more important than having it technically excellent. And even granting the quoted preference, "technically right" would be to follow the kernel rule - whoever changes an interface is responsible for fixing all its users.

comment:25 in reply to: ↑ 17 ; follow-up: ↓ 32 Changed 19 months ago by jstanley

Replying to notting:

We do have to support both local filesystem /tmp and /tmp-on-tmpfs; we have no choice. It therefore behooves us to do the proper level of testing, bug filing, and fixing, and then use that information to make an informed choice on defaults, rather than polemics.

100% agreed. My concern comes from having it as the default - in an idealistic world, this would be fine. However, we have users that we've trained over the years to "store stuff that you don't care about in /tmp". And people do. Gigs and gigs and gigs worth.

I haven't tested this yet, so forgive the hypothetical nature of this question, but what happens in the event of memory pressure in this world? In other words, I have 4GB of RAM, and 2GB of stuff in /tmp-on-tmpfs. I then require 3GB of anonymous memory - I clearly don't have enough, even though I've got plenty. What gets swapped? My $IMPORTANT_PROCESS process because it came later? I certainly hope not!

Also, as you allude to, the defaults are what gets the most test coverage. Therefore, in exchange for finding bugs that are related to /tmp-on-tmpfs, we *fail* to find bugs that are related to having it on a filesystem.

comment:26 in reply to: ↑ 23 Changed 19 months ago by jstanley

Replying to lennart:

Well, we shouldn't let us being taken hostage by 3rd party sw (closed source even), and not do what is technically right.

This I cannot agree with. There is lots of software that is non-free that runs on Linux. I hate to say it, but this is the world that we live in. Also, it's not only ISV software that you have to be concerned with - it's line-of-business apps that are written internally, may have developed over 20+ years, and consist of hundreds of thousands LOC. It is cost-prohibitive to change these apps, and when it's done, it's because the business user wants to click on the new red widget that makes lots of money, not some infrastructure change that provides no value.

comment:27 Changed 19 months ago by jstanley

  • Cc jstanley added

comment:28 Changed 19 months ago by rjones

In order to get some sort of idea of the extent of problems that might be caused by defaulting to /tmp on tmpfs, I checked out 100 random Fedora packages and examined them by hand for issues.

There are currently 12153 source packages in Fedora, so my subset represents 0.8% of the total.

I found 18 bugs in a cursory examination.

_If_ it is valid to scale this small sample to the whole of Fedora (and it may not be), then we would expect > 2000 bugs.

Methodology:

(1) Use repoquery to choose 100 source packages at random.

repoquery --enablerepo=fedora-source -q --archlist=src -a --qf='%{name}' |
     sort -R | head -100

(2) These are the packages I chose. Obviously a different run of the command above would choose a completely different set of packages.

latex2emf libcue gdb perl-Text-Smart-Plugin VMDKstream libtpms
rubygem-mkrf scsi-target-utils perl-Net-Lite-FTP eurephia cinepaint
lunatic-python taginfo yokadi prelude-correlator mod_nss libident
python-pep8 perl-Net-SSH dpm-xrootd system-config-nfs
perl-Carp-Assert-More eet osgal bootchart perl-Olson-Abbreviations
authd tmda manaworld-music gtk2hs-buildtools eclipse-wtp-jeetools
perl-Module-Refresh ghc-vault libnih monotone-viz rpmreaper libreport
perl-Config-Record jabref astronomy-menus fedora-gnat-project-common
smem gtkpod perl-MasonX-Interp-WithCallbacks perl-Net-Ping-External
synce-trayicon python-repoze-what perl-Perl-OSType akonadi hunspell-mk
udisks miredo php-magickwand rubygem-main perl-B-Compiling
sonic-visualiser txt2rss perl-DateTime-Format-DateManip
perl-DBD-SQLite2 fishpoll perl-Test-WWW-Mechanize rdiff-backup
python-django-tagging libconfuse libytnef gtk3 libmusicbrainz4
perl-CatalystX-Component-Traits drascula libcgroup stage paco
felix-osgi-core pastebinit metagoofil shortrpm plotmm libgarmin kmplot
libnetfilter_log hunspell-nso perl-PerlIO-via-symlink tunctl
gtksourceview-sharp perl-Crypt-OpenSSL-X509 hct kmess postr
rubygem-aruba libnl3 rubygem-systemu sisu kakasi monodevelop editarea
perl-Test-Assertions sane-frontends php-pear-HTTP-Request sugar-xoirc
jspeex

(3) I cloned each package and did 'fedpkg prep' in each master branch, in order that I had a copy of the source of each of these packages.

Four packages failed to prep, so are discarded from this analysis: php-magickwand, rubygem-mkrf, sisu, synce-trayicon

(4) I searched the sources for

    \b(/tmp|TMPDIR|mktemp|mkstemp|tempnam|tmpfile|tmpnam|mkdtemp)\b

and then manually identified issues. I was looking for only the following problem:

  • Package creates a file under /tmp which could grow to an arbitrary size. ie. Not a socket, or file containing just a number, and there is no prior check on maximum file size, and no attempt to place "large" files in /var/tmp instead of /tmp.

Note that although that's the only problem I looked for, it's not the only problem that tmp-on-tmpfs could cause. Others include: storing any kind of precious data in /tmp, expecting /tmp to reside on disk for performance-versus-size trade-offs, expecting features (ioctl, chattr, O_-flags etc.) of a real filesystem that are not provided by tmpfs.

(5) After manually discarding non-source files (eg. *.po, config.guess, etc) there were 125 matching regexps that I examined by hand. As a result of this examination, I found the following bugs [ (E) = possibly exploitable ]:

bootchart: Creates arbitrary-sized log files in /tmp/bootchart.X dir.

cinepaint: ICC Examin plugin will wget arbitrary-sized web pages to /tmp

cinepaint: ICC Examin has a hard-coded /tmp directory (E)

cinepaint: PDF loader creates arbitrary-sized PDFs in /tmp (E)

cinepaint: RAW decoder creates arbitrary-sized file in /tmp (size depends on user input)

gdb: Code is too complex for me to make a definite call in a short time, but it appears to contain user functions that would create arbitrary-sized files in /tmp, based on user input.

gtk2hs-buildtools: c2hs: There's some freaky stuff going on with temporary files here, but it does appear that it will create arbitrary-sized files in /tmp based on user input.

gtk3: Creates bitmaps in /tmp. Since it actually appears to want shared memory, this code should probably be using /dev/shm.

gtk3: tests: Two places will create arbitrary files in /tmp. Give this a pass since tests probably don't involve arbitrary user data.

gtkpod: Build system creates arbitrary-sized output files in /tmp.

jabref: Will copy arbitrary user files or external (from http: sources) into /tmp

kmplot: Creates arbitrary-sized user documents in /tmp when requested to upload them to a remote location.

kmplot: Ditto, when downloading from arbitrary remote URL.

3 x libreport: Writes crash reports to /tmp (hard-coded, cannot be overridden with TMPDIR). Actually there are several dubious things going on in this library. I'm counting this as 3 bugs, and that's probably being generous.

perl-Olson-Abbreviations: Converts user-supplied data to a file in /tmp

yokadi: User edits text file in /tmp (arbitrary sized, and precious)

(6) I did find exactly one package that uses /var/tmp for large files. Step forward perl-DBD-SQLite2.

comment:29 Changed 19 months ago by rjones

I filed some actual bugs (although I did not file bugs for the issues outlined in the previous comment).

https://bugzilla.redhat.com/show_bug.cgi?id=858264 https://bugzilla.redhat.com/show_bug.cgi?id=858265 https://bugzilla.redhat.com/show_bug.cgi?id=858266

comment:30 Changed 19 months ago by rjones

I was asked on IRC how long the analysis in comment 28 took. I would say about 1 hour to check out and prep the code, and about 1-2 hours to do the analysis (but I wasn't very thorough and relied mainly on regexps followed up by reading the source that these regexps highlighted).

comment:31 Changed 19 months ago by jdulaney

  • Cc jdulaney added

comment:32 in reply to: ↑ 25 ; follow-up: ↓ 36 Changed 19 months ago by lennart

Replying to jstanley:

Replying to notting:

We do have to support both local filesystem /tmp and /tmp-on-tmpfs; we have no choice. It therefore behooves us to do the proper level of testing, bug filing, and fixing, and then use that information to make an informed choice on defaults, rather than polemics.

100% agreed. My concern comes from having it as the default - in an idealistic world, this would be fine. However, we have users that we've trained over the years to "store stuff that you don't care about in /tmp". And people do. Gigs and gigs and gigs worth.

Due to the common namespace problems it is a really bad idea to teach people this.

I haven't tested this yet, so forgive the hypothetical nature of this question, but what happens in the event of memory pressure in this world? In other words, I have 4GB of RAM, and 2GB of stuff in /tmp-on-tmpfs. I then require 3GB of anonymous memory - I clearly don't have enough, even though I've got plenty. What gets swapped? My $IMPORTANT_PROCESS process because it came later? I certainly hope not!

The memory used for /tmp is subject to more or less the same memory eviction logic than any other memory, including all memory that is backed anyway by disk. In other words: there's next to zero difference in this regard compared to normal files on disk.

comment:33 follow-ups: ↓ 34 ↓ 37 Changed 19 months ago by lennart

rjones: the thing you are misunderstanding here is that there is no problem with enforcing limits on the size of /tmp, in fact it is a necessity and a security requirement, and there has always been a limit on /tmp anyway, since your HDD file system had a size too. It is a basic rule of software design to impose limits on all resources, in order to ensure that DoS vulnerabilities by boundless allocations caused by clients are confined to one specific area.

So, without question /tmp needs a maximum size. It had one before, it has one now. And that's good that way. What is subject to discussion however is where we should put that limit. Right now it defaults to half the physical RAM size. Maybe that's not the perfect choice. To tune that is something we should spend time on during the Alpha. At this point, there's still only one bug filed against the tmpfs-on-tmp tracker bug, so the original choice we made doesn't appear to be too problematic.

Anyway, as said it is not a problem to impose a size limit on "arbitrary size" users of /tmp. It's actually a good thing. Now, what matters is tracking down those program where large amounts of data are allocated in a temporary directory, where that data is known to be inherently and systematically large, and where that temporary directory chosen for that is currently /tmp. Acknowledging that, if I look at your list I immediately see a couple of candidates which are non-problems:

  • if cinepaint downloads profiles with wget to /tmp, then there needs to be a limit on that. We simply cannot allow cinepaint/wget to download unbounded files from the web. Same for jabref, kmplot.
  • bootchart is a non-issue anyway, since the measurement data from initrd/early-boot must be stored in memory anyway, since writable /tmp is only available late at boot.
  • gtk2hs-buildtools: similar to the http issue: we cannot allow unbounded allocations in /tmp for user input. It is common knowledge that all user input needs to be validated anyway before processing it, and that includes emposing size limits on it.

And that's just be having a quick peek on your list.

So, to summarize: arbitrarily sized files in /tmp is a non-issue. Known-to-be-validly-large, those are the problematic cases.

And you know, as notting said, we need to support both /tmp-on-disk, and /tmp-on-tmpfs anyway. We need it for thin clients with no local storage, for stateless systems, for containerized systems, for pre-image systems, and for everybody else who wants to use them. Hence, we need to fix these bugs which might arise anyway. So the question is not whether we should fix them or not, and the question is not whether we should have a size limit on /tmp, the question is simply where to put that limit.

comment:34 in reply to: ↑ 33 ; follow-up: ↓ 38 Changed 19 months ago by mganisin

Replying to lennart:

And you know, as notting said, we need to support both /tmp-on-disk, and /tmp-on-tmpfs anyway. We need it for thin clients with no local storage, for stateless systems, for containerized systems, for pre-image systems, and for everybody else who wants to use them.

Nothing from the list is typical Fedora installation target. Even more none of these statements seem to be true, for example we really do not need /tmp-in-ramfs for thin client as there has to be some rw /home mounted anyway. It is hard (or better to say impossible) to advocate default setup by these marginal Fedora use cases.

Hence, we need to fix these bugs which might arise anyway.

What is pre-image system? initrd? Do we really need to "fix" FF because of this?

So the question is not whether we should fix them or not, and the question is not whether we should have a size limit on /tmp, the question is simply where to put that limit.

Actually question is about reasonable default. Read the topic of this ticket and don't change it.

Limit in size isn't security enhancement (actually it makes DoS even easier and system stability worse), limit in quota is. So far quota is another -1 for this feature from security point of view.

/tmp provides easily accessible storage for temporary files. It is really hard to find meaningful reason why to use expensive storage for this purpose by default.

Last but not least, if advocates will force /tmp-on-tmpfs as default, many typical use cases will be moved to /var/tmp and situation will be exactly same just in another location, no gain will come from this feature.

Btw FF doesn't seem to be fixed yet, it looks like that sort will stay with /tmp as default. If you want to reduce default /tmp size anyway...

comment:35 Changed 19 months ago by tmraz

lennart: And the thing you're quite misunderstanding here is that the limits are very different for the current RAM sizes and harddrive sizes. Also if there was a possibility to make swap files grow and reduce automatically you might be right that the tmpfs with swapping out to a swap can be equivalent to the /tmp on harddrive, but that is not the current situation. And for many workloads it is much better to not have any swap at all or only a small one and then the inefficiency of keeping all of the /tmp contents in RAM is quite apparent.

comment:36 in reply to: ↑ 32 Changed 19 months ago by rjones

Replying to lennart:

The memory used for /tmp is subject to more or less the same memory eviction logic than any other memory, including all memory that is backed anyway by disk. In other words: there's next to zero difference in this regard compared to normal files on disk.

No it isn't! Swap lives in a separate partition. You can run out of swap and still have plenty of free space on your filesystem. All Fedora systems up to this point have put /tmp on the root partition and in the majority of cases there's just one partition, so you wouldn't run out of /tmp until your filesystem is full.

If there was a case where swap somehow used space on the real filesystem things would be different, but the kernel doesn't work like that.

comment:37 in reply to: ↑ 33 ; follow-up: ↓ 39 Changed 19 months ago by rjones

Replying to lennart:

At this point, there's still only one bug filed against the tmpfs-on-tmp tracker bug, so the original choice we made doesn't appear to be too problematic.

Can you PLEASE stop changing the goalposts here.

Firstly FESCO can't be bothered to even read this ticket before discussing it. Then they insist that bugs must be filed. When I file bugs (there are 3 in Bugzilla) there's now some mysterious "tracker" that no one mentioned up til now.

Then I've clearly shown that there could be thousands of bugs in Fedora caused by this change. It's up to YOU to fix these bugs as the feature owner, not up to me.

comment:38 in reply to: ↑ 34 ; follow-up: ↓ 42 Changed 19 months ago by lennart

Replying to mganisin:

Replying to lennart:

And you know, as notting said, we need to support both /tmp-on-disk, and /tmp-on-tmpfs anyway. We need it for thin clients with no local storage, for stateless systems, for containerized systems, for pre-image systems, and for everybody else who wants to use them.

Nothing from the list is typical Fedora installation target. Even more none of these statements seem to be true, for example we really do not need /tmp-in-ramfs for thin client as there has to be some rw /home mounted anyway. It is hard (or better to say impossible) to advocate default setup by these marginal Fedora use cases.

Well, what I am pointing out here is that arguing with "there are too many (potential) bugs" won't hold, since we need to fix them anyway. I mention this simply to point out that rjones' argument doesn't hold.

The reasons why tmpfs-on-/tmp is the right thing to do is independent of that and those have been listed in the original feature page.

Hence, we need to fix these bugs which might arise anyway.

What is pre-image system? initrd? Do we really need to "fix" FF because of this?

One where you distributed pre-generated identical images among a number of systems.

Limit in size isn't security enhancement (actually it makes DoS even easier and system stability worse), limit in quota is. So far quota is another -1 for this feature from security point of view.

Hmm? Sure, we need user quota on /dev/shm, /tmp, /run. We currently don't have that and it#s a security hole. But whether /tmp has is tmpfs or not doesn't really matter in this regard, since having two fs which need quota or three is not much of a difference. Also note that we do not set up quota by default for hard-disk /tmp, so arguing with this is on really shaky ground anyway.

And limit on size does matter. If /var and /tmp are on the same fs, then apps flooding /tmp can easily bring the whole system to a standstill since that's what happens if /var is full. This is much nicer now with a split off /run and and tmpfs on /tmp, since these are now three different pools. Where the one that is the most likely to be flooded (/tmp) can no longer affect the other ones.

So, yeah, applying a limit to /tmp by default is a security enhancement, and a major one.

comment:39 in reply to: ↑ 37 Changed 19 months ago by lennart

Replying to rjones:

Replying to lennart:

At this point, there's still only one bug filed against the tmpfs-on-tmp tracker bug, so the original choice we made doesn't appear to be too problematic.

Can you PLEASE stop changing the goalposts here.

Firstly FESCO can't be bothered to even read this ticket before discussing it. Then they insist that bugs must be filed. When I file bugs (there are 3 in Bugzilla) there's now some mysterious "tracker" that no one mentioned up til now.

This mysterious "tracker" bug is not so mysterious at all. This bug has been created as part of the implementation of the feature. It has been listed on the feature page since about time began. Did you actually ever read the feature page? Honestly, reading the feature page is actually the least I expected from you before you escalate this all the way to FESCO...

This is the tracker bug:

https://bugzilla.redhat.com/show_bug.cgi?id=826015

Then I've clearly shown that there could be thousands of bugs in Fedora caused by this change. It's up to YOU to fix these bugs as the feature owner, not up to me.

Yeah, "thousands of bugs" and "could be"... I mean, I appreciate your work in this regard, but if you look for the wrong things you find the wrong things... And "there could be bugs" doesn't really count since we have to fix them anyway even if we don#t make this the default.

comment:40 Changed 19 months ago by mmaslano

rjones was the only one who tried to do some testing at this point. So it doesn't look fair to attack him.

I guess it's valid point that the owner of the feature should provide patches or find help for his feature.

comment:41 Changed 19 months ago by notting

It is not a personal attack against Richard to mention that filing bugs against the tracker that's been on the feature page since May is in no way moving the goalposts; it's just pointing out an oversight that's certainly unintentional.

Saying that some apps can create arbitrarily sized data in /tmp is not necessarily a meaningful data point here - obviously there are sizes of arbitrary for many common installations with /tmp on real disks that are too big as well; a program that masters dual-layer DVDs with files in /tmp is likely to run out of space on many disk-backed implementations. It's a matter of choosing which arbitrary size is arbitrarily too large.

. Even more none of these statements seem to be true, for example we really do not need /tmp-in-ramfs for thin client as there has to be some rw /home mounted anyway.

Again, if you'd like to discuss that we should tell people who have been implementing stateless systems, such as many virt host providers, that their way of operating their system is invalid, we can do so. But I don't think that's as useful as finding the users, fixing the cases where they're obviously broken, and making an informed decision.

comment:42 in reply to: ↑ 38 Changed 19 months ago by mganisin

Replying to lennart:

The reasons why tmpfs-on-/tmp is the right thing to do is independent of that and those have been listed in the original feature page.

Regarding to SSD lifetime and power consumption - Just thoughts, presumptions, any relevant measurement is missing (to avoid misunderstanding relevant == measure SSD lifetime and power consumption with typical /tmp usage)

Regarding to /tmp cleanup - Right, however what's beneficial here? I would rather mark this as deliverable.

Regarding to commercial Unixes and other Linux distributions. - Honestly /tmp doesn't do the difference of any commercial *nix. Debian won't switch, feature page seems to be little bit outdated.

Regarding to stateless read-only systems. - Untruth, by default /var, /srv, /etc stay on /, all of them are stateful, most of them are rw. No gain from this feature for this goal (the feature _is_not_ /tmp-on-tmpfs, _it_is_ /tmp-on-tmpfs by _default_).

And limit on size does matter. If /var and /tmp are on the same fs, then apps flooding /tmp can easily bring the whole system to a standstill since that's what happens if /var is full. This is much nicer now with a split off /run and and tmpfs on /tmp, since these are now three different pools. Where the one that is the most likely to be flooded (/tmp) can no longer affect the other ones.

Did you actually ever read the feature page? Honestly, reading the feature page is actually the least I expected from you. It orders to put large files to /var/tmp, /var/tmp is by default on /var (actually both on /), the feature doesn't avoid scenario you described at all. You would never notice this story if you were reading feature page.

comment:43 Changed 19 months ago by mmaslano

  • Keywords meeting removed

I do not believe we have different data than we had last time. Let's postpone this ticket for some future meeting, when Fedora will be better tested.

If someone has a new data, then please mention them and I'll add this ticket into FESCo agenda.

comment:44 Changed 17 months ago by fche

  • Cc fche added

comment:45 Changed 17 months ago by mattdm

  • Cc mattdm added

comment:46 Changed 17 months ago by mmaslano

  • Keywords meetign added

We have still see the problem of virtualization: https://bugzilla.redhat.com/show_bug.cgi?id=858265#c4

If we can switch it back to different file system, now is the time.

comment:47 Changed 17 months ago by rbergero

  • Keywords meeting added; meetign removed

Fixing spelling of meeting keyword. :)

comment:48 Changed 17 months ago by rbergero

https://bugzilla.redhat.com/show_bug.cgi?id=873831 seems rather related as well (I could be entirely wrong though) - and is on the proposed blocker list.

comment:49 Changed 17 months ago by notting

  • Status changed from new to closed
  • Resolution set to wontfix

From the 2012-12-05 FESCo meeting:

  • AGREED: #940 (The tmp-on-tmpfs feature should be disabled by default) does not pass. (+:4, -:4, 0:1)

Closing.

Note: See TracTickets for help on using tickets.