2023-06-10

GNU Parallel for Serial Execution

It’s sometimes useful to use GNU Parallel for running things serially. Why? Because it remembers what it’s done, it can pick up where it left off, store output per job, and a ton of other stuff.

Example: re-encoding videos

Input file:

some-video.mp4
some-other-video.mp4
...

Command:

parallel --arg-file input.txt ffmpeg {} -o {.}.reencoded.mp4

This will run ffmpeg on every line of input with -o set to the input line (without the extension) plus .reencoded.mp4.

If --joblog is given, Parallel keeps a log of every completed job and some metadata e.g.

Seq    Host    Starttime    JobRuntime    Send    Receive    Exitval    Signal    Command
1      :    1686425772.721         57.002    0    0    0    0    ffmpeg some-video.mp4 -o some-video.reencoded.mp4
2      :    1686425829.720         85.124    0    0    0    0    ffmpeg some-other-video.mp4 -o some-other-video.reencoded.mp4
...

Parallel can use the joblog to pick up where it left off, just add --resume.

parallel --arg-file input.txt \
         --joblog reencode_job.joblog \
         --resume \
         ffmpeg {} -o {.}.reencoded.mp4

This runs Parallel with one invocation of ffmpeg at a time, keeps track of every command it runs, and can pick up where it left off.