Signs of Triviality

Opinions, mostly my own, on the importance of being and other things.
[homepage] [index] [] [@jschauma] [RSS]

Safely Creating And Using Temporary Files

June 5th, 2017

Knock, knock. Race condition. Who's there? Safely creating, using, and removing temporary files requires more attention to detail than most Devs, Ops, and DevOpses assume. This post summarizes the various dangers of using temporary files in an unsafe manner, leading to a number of security vulnerabilities including arbitrary file overwrites, privilege escalation, or information disclosures.

Fun fact: The best way to avoid problems with temporary files is to not use temporary files at all.

Yes, that seems obvious, but it bears repeating: you should avoid using temporary files whenever possible. Temporary files are terribly convenient, but they also provide ample opportunity to to shoot yourself in the foot. If there's any way you can reasonably work without temporary files -- for example by way of pipes or subprocesses -- do so.

For those not patient enough to read this whole post, here's the terse summary:

  • don't use temporary files
  • if you can't avoid it, then:
    • set a restrictive umask
    • use mktemp(3)
    • unlink your temporary files via an exit handler

Temporary Pitfalls

How do temporary files offer opportunity for calamity? Let me count the ways...

Consider the following example. It's given in shell, but I've seen it implemented in any number of programming languages:

you@host$ do-something >/tmp/myfile
you@host$ do-that-other-thing -f /tmp/myfile

Information Disclosure I

If the data do-something generates is sensitive, then I may be able to read it. Most Unix systems have a default umask of 022, meaning that new files (i.e. /tmp/myfile) will likely get read permissions for everybody. All I need to do is wait for the file to exist and access it.

evil@host$ while [ 1 ]; do cat /tmp/myfile ; done

[... meanwhile ...]
you@host$ do-something >/tmp/myfile
you@host$ do-that-other-thing -f /tmp/myfile

Now I've seen developers add a quick chmod 0400 statement:

you@host$ do-something >/tmp/myfile
you@host$ chmod 0400 /tmp/myfile
you@host$ do-that-other-thing -f /tmp/myfile

...but that doesn't help at all, since that just creates a race condition.

To help mitigate: set your umask to 077 before running those commands, preferably in ~/.profile / ~/.bashrc etc. (Even better: set the default umask system wide!)

This doesn't fix the information disclosure completely (see below), but it's a start.

Information Disclosure II

Ok, so you've set your umask, and /tmp/myfile will be created readable only by the owner (i.e. you). My previous ploy was foiled - blast! But wait, I know (or can predict) the name of the file you're going to create, and I have write access to the directory in which you're creating the file...

Let me create a file that I can read and you can write to, then symlink the known temporary file name so that you will end up writing to my file instead:

evil@host$ touch /tmp/.myfile
evil@host$ chmod a+rw /tmp/.myfile
evil@host$ ln -s /tmp/.myfile /tmp/myfile
evil@host$ while [ 1 ]; do cat /tmp/.myfile ; done

[... meanwhile ...]
you@host$ umask 077
you@host$ do-something > /tmp/myfile
you@host$ do-that-other-thing -f /tmp/myfile

To avoid: check for the existence of the file you're writing to before doing so. If the file already exists, remove it or use a different file name. You may also wish to create the file in a private directory (which itself may need to be temporary). The more complete solution here is to use mktemp(1). More on that below.

Arbitrary File Clobbering

Not only can I read the data you're generating (which may or may not be a concern to you), but by staging the symlink attack above, I can also cause you to overwrite any file you have write access to. If you are creating the temporary file as root (which many tools that exhibit this problem do), then I can clobber any file on the entire system.

evil@host$ ln -s /home/you/.ssh/authorized_keys /tmp/myfile

[... meanwhile ...]
you@host$ do-something > /tmp/myfile
you@host$ do-that-other-thing -f /tmp/myfile
you@host$ exit
you@laptop$ ssh host
[fails, your authorized_keys was corrupted]

Note that even if you manually check for the existence of the file first, I can still cause you to clobber any file, since most hastily written code to check for the existence of a file prior to using it includes a TOCTOU vulnerability:

you@host$ if [ ! -f /tmp/myfile ]; then do-something > /tmp/myfile; fi

Sorry, two operations; not atomic. I need to win the race between your time of check and your time of use, but that remains a realistic threat.

To avoid: as above, atomically check for the existence of the file you're writing to before doing so. Better yet: use mktemp(1).

Please use my data instead!

Causing you to clobber a file is not the only threat. I may also want to trick you into using my data instead. That is, let's assume I want you to think that your do-somethinging was successful and then feed you some data that I control to do-that-other-thing with. Let's illustrate this using some crude shell code, once more:

evil@host$ touch /tmp/myfile
evil@host$ chmod a+rw /tmp/myfile
evil@host$ generate-evil-data >/tmp/.myfile
evil@host$ chmod a+r /tmp/.myfile
evil@host$ while [ 1 ]; do
if ps aux | grep -q do-something; then
ln -sf /tmp/.myfile /tmp/myfile

[... meanwhile ...]
you@host$ do-something > /tmp/myfile
you@host$ do-that-other-thing -f /tmp/myfile

To avoid: The only reasonable way of handling any of these issues is via mktemp(1)/mktemp(3). I think you can see a pattern here.

mktemp(3) is your friend

Ok, so creation of temporary files is a pain. You need to account for existence of the file, use a unique name, protect the file, and later (safely) remove the file again. We already discussed setting a safe umask above -- this remains a critical and required part! In addition, you can make use of the mktemp(3) interface provided by most programming languages (e.g. via mktemp(1) for your shell).

Safe removal of the temporary files is something that should be done both explicitly by your script / commands as well as by any exit handlers you use (see this post). Some programming languages also provide convenience library function wrapping mktemp(3) that automatically remove the temporary file after e.g. closing of the file handle -- use those!

With all this in mind, the canonical (shell pseudo-) example should look something like this:

umask 077
[ -z "${file}" ] && do-something >"${file}"
do-that-other-thing -f "${file}"
rm "${file}"

Yep, that's still a pain, and you still have plenty of room for mistakes. Seriously, try to avoid temporary files. But if you can't, the above "umask; mktemp; rm" combination is the only reasonable approach.

Temporary Directories

Sometimes you don't need a temporary file, but a temporary directory. Or a temporary fifo (see below). Or you need a temporary filename, but the file must not exist. You might:

$ umask 077
$ fname=$(mktemp)
$ rm "${fname}"
$ mkfifo "${fname}"

But this of course creates another race condition: right after you removed ${fname}, somebody else might have re-created it. Instead, you want to create a temporary directory (with suitable protections!) immediately, then create the required resource therein:

$ umask 077
$ dname=$(mktemp -d)
$ mkfifo "${dname}/fifo"

The filename component can then be predictable; the temporary directory is protected via your umask and mktemp(1) guaranteed the safe creation of the directory. Yay, mktemp(1)!

Additional Issues

If you're still not convinced that temporary files are more complicated than you thought, here are some more issues. These do not necessarily pose security concerns, but are still so frequently seen especially amongst inexperienced developers and students, that I think it's worth noting them.

Temporary Directory

One common mistake is to not use the correct directory to create temporary files under. For example, I regularly see students write code that assumes that a given program has write access to e.g. the current working directory. Ok, rookie mistake, easy to fix - just use /tmp, right?

But doing that is not always the right solution, either. By convention and necessity, /tmp is of course world-writable with the sticky bit set, meaning that despite being world-writable, only the file owner (or root) may remove a file in it. But the directory remains world-readable and world-executable, meaning you will at the very least reveal that you are creating temporary files as well as what their naming schema may be.

(/tmp may also have less space or be out of inodes or in any other way be an undesirable location for you to create your temporary files under.)

Per-user private temporary directories are kinda nice here. macOS provides those out of the box (go ahead, try echo $TMPDIR -- it's not pointing to /tmp, is it?), but you may also wish to create your own tmpdir, e.g. under your home directory. Proper Unix tools (such as mktemp(1), of course) will gladly accept those via the TMPDIR environment variable. When you create temporary files, you should do the same.

Cleaning up of temporary files

Most scripts and programs create temporary files, but then forget to (reliably) remove them again. That is, they may either completely forget to remove the file or only remove it upon successful completion of the jobs, rather than via an exit handler.

This can lead to temporary directories with thousands or hundreds of thousands of files in them, making basic operations on e.g. /tmp difficult and/or slow.

Disk Space

The output of do-something is stored in /tmp/myfile. We are assuming that /tmp has enough space for however much data do-something generates. This is not so much a vulnerability, but just something to be aware of.

Local vs. Remote

Often times the scripts or tools we write need to perform one step on one system, then perform another step on another system. Care should be taken to safely utilize temporary files on both.

That, however, requires the use of e.g. mktemp(1) on both systems. An often encountered anti-pattern is:

umask 077
file=$(mktemp)  # yay, we have a safe local file!
do-something >"${file}"
scp "${file}" some-host:/tmp
ssh some-host "do-that-other-thing -f /tmp/${file}"

The problem here is that even though the temporary file was safely generated on one host, all the race conditions we mentioned above still apply on the other host! That is, you'd have to repeat the same steps (umask; mktemp; rm) on the remote host, too:

umask 077
do-something >"${file}"
rfile=$(ssh some-host "umask 077; mktemp")
scp "${file}" some-host:"${rfile}"
ssh some-host "do-that-other-thing -f ${rfile}"

This doesn't seem like a big deal, but honestly, how many of your developers are going to get this right, error checking and all?

Look, Ma, no temporary files!

If at all possible, pipe the output directly into the next tool. (Hooray, Unix Philosophy!)

$ do-something | do-that-other-thing 

Or, if you're dealing with multiple hosts, remember that you can feed data into commands straight from ssh:

$ do-something | ssh somehost do-that-other-thing 

If do-that-other-thing doesn't read from stdin, maybe it accepts - as an argument to -f:

$ do-something | do-that-other-thing -f -

Alternatively, /dev/stdin may be available:

$ do-something | do-that-other-thing -f /dev/stdin

If do-that-other-thing is so poorly written that it accepts neither stdin nor - as an argument to an input file flag, but it does have such a flag, you may be able to use process substitution in e.g. bash(1):

$ do-that-other-thing -f <(do-something)

Process substitution generally works by way of a short-lived fifo, so you could try to implement that approach yourself, but then you'd be facing the original dilemma of keeping your fifo's pathname unique and protected, so probably not worth the hassle:

$ umask 077
$ d=$(mktemp -d)
$ mkfifo "${d}/fifo"
$ do-something >"${d}/fifo" &
$ do-that-other-thing -f "${d}/fifo"
$ rm -fr "${d}"

Finally, if the data in question consists of single arguments, you can of course pass them directly:

$ do-that-other-thing $(do-something)

But note that this leaks the output into the process table and shell history, which may not be desirable (such as e.g. when generating or retrieving passwords).

Once more for the cool kids in the back who haven't been paying attention

The simple summary of how to deal with temporary files: don't use temporary files.

If you absolutely can't avoid using temporary files, then:

  • set a restrictive umask
  • use mktemp(3)
  • unlink your temporary files via an exit handler

It sounds so simple...

June 5th, 2017


[Why Companies Should Pay For Their Employees To Attend Conferences] [Index]