GNU Parallel for Serial Execution


It’s sometimes useful to use GNU Parallel for running things serially. Why? Because it remembers what it’s done, it can pick up where it left off, store output per job, and a ton of other stuff.

Example: re-encoding videos

Input file:



parallel --arg-file input.txt ffmpeg {} -o {.}.reencoded.mp4

This will run ffmpeg on every line of input with -o set to the input line (without the extension) plus .reencoded.mp4.

If --joblog is given, Parallel keeps a log of every completed job and some metadata e.g.

Seq    Host    Starttime    JobRuntime    Send    Receive    Exitval    Signal    Command
1      :    1686425772.721         57.002    0    0    0    0    ffmpeg some-video.mp4 -o some-video.reencoded.mp4
2      :    1686425829.720         85.124    0    0    0    0    ffmpeg some-other-video.mp4 -o some-other-video.reencoded.mp4

Parallel can use the joblog to pick up where it left off, just add --resume.

parallel --arg-file input.txt \
         --joblog reencode_job.joblog \
         --resume \
         ffmpeg {} -o {.}.reencoded.mp4

This runs Parallel with one invocation of ffmpeg at a time, keeps track of every command it runs, and can pick up where it left off.

If you have questions or comments (or corrections 😅) on one of my posts, feel free to email me :)