Comments on: Tar-filled pipes

By: lacos

lacos — Mon, 01 Mar 2010 20:37:04 +0000

Re: davidben, March 1, 2010 at 11:04 am: yes, the NUL blocks do explain the broken pipe. IIRC I sent the inquiry to help-tar first, then went to reddit second, and saw your blog entry posted there. Thanks.

By: davidben

davidben — Mon, 01 Mar 2010 16:14:05 +0000

I should probably clarify for folks — I didn’t find or investigate the Python SIGPIPE thing. Just looked into a detail of a friend’s discovery I found curious.

By: davidben

davidben — Mon, 01 Mar 2010 16:04:38 +0000

lacos: Oh, the thing I wanted to poke bug-tar for was something else; all_names_found can’t really reliably check if you’ve exhausted a directory. The current code assumes all directories are contiguous (not sure if the spec mandates that they be contiguous… it’s certainly very easy to create one where they aren’t), but does the check in a somewhat strange manner such that, for non-contiguous directories, you get very different results depending on surrounding file names.

As far as I’m aware, the NUL blocks should explain the SIGPIPE sending. It’s a fairly tight race condition, which fits with it only occurring some of the time; there’s a small enough tail that it depends on how much buffering both your filters and tar do, as well as who happens to get ahead of the other.

By: lacos (lbzip2)

lacos (lbzip2) — Mon, 01 Mar 2010 15:13:52 +0000

Thank you very much for investigating the phenomenon in the tar source.

I posted an inquiry to the help-tar mailing list: http://lists.gnu.org/archive/html/help-tar/2010-03/msg00000.html

Perhaps the discussion should be migrated there (or bug-tar as you say).

Thanks,
lacos / lbzip2

By: Matt Sandy

Matt Sandy — Mon, 01 Mar 2010 09:05:47 +0000

This is awesome. I actually just got a VPS (no sites on it yet) and posts like this make me happy. Sometimes simple things can be daunting if you aren’t familiar, and I am a stranger to python but hope to learn.

By: Bron Gondwana

Bron Gondwana — Mon, 01 Mar 2010 07:40:18 +0000

That is interesting, and makes perfect sense. I have written a backup system for Cyrus that makes very sneaky use of the tar format – I wound up writing a custom Perl module that can read and write tar streams without having to touch the disk – in particular they can be used to strip stale files out of a tar file by streaming it through a decider function.

I use an external gzip process, but obviously not an external tar, so I close it explicitly if I don’t need to finish it. It’s possible to embed the gzip directly inside the process too, but I kind of like the “use multiple cores” side effect of letting a separate process do the compression.

By: John

John — Mon, 01 Mar 2010 04:34:23 +0000

Jeebus, thank you so much for posting this. I ran into this issue a couple weeks ago and it was pretty difficult to debug given that it only happened on a specific build machine for some reason. I’ll sleep better tonight knowing that there’s nothing more insidious going on.