FFmpeg: Advanced Video & Audio Manipulations

As a Linux sysadmin or anyone delving into the world of video processing, mastering FFmpeg is essential. FFmpeg is an open-source and cross-platform solution for recording, converting, and streaming audio and video. It offers a vast array of options that ensure it can handle just about any video conversion task one can throw at it. Here, we’ll explore some practical examples that serve as a great starting point for beginners.

1. Converting Video Formats

Convert an MP4 file to AVI:

ffmpeg -i input.mp4 output.avi

Explanation: Here, -i specifies the input file. FFmpeg automatically detects the input format and converts it to the output format specified by the extension of the output file.

2. Extracting Audio from Video

Extract audio (in MP3 format) from a video file:

ffmpeg -i input.mp4 -q:a 0 -map a output.mp3

Explanation: -q:a 0 uses the highest audio quality. -map a tells FFmpeg to map all audio streams from the input to the output.

3. Combining Audio and Video

Merge an audio file with a video file (replacing the existing audio):

ffmpeg -i video.mp4 -i audio.mp3 -c:v copy -c:a aac -strict experimental output.mp4

Explanation: Here, -c:v copy copies the video codec from the original, and -c:a aac re-encodes the audio to AAC. -strict experimental is often necessary to use AAC.

4. Resizing Videos

Change the resolution of a video to 720p:

ffmpeg -i input.mp4 -s 1280x720 -c:a copy output.mp4

Explanation: -s sets the frame size. -c:a copy copies the original audio without re-encoding.

5. Adjusting Video Playback Speed

Speed up a video (2x faster):

ffmpeg -i input.mp4 -filter:v "setpts=0.5*PTS" output.mp4

Explanation: The setpts filter adjusts the presentation timestamps of the video frames. Here, 0.5*PTS doubles the speed.

6. Creating Video from Images

Make a video from a series of images (img001.jpg, img002.jpg, …):

ffmpeg -f image2 -i img%03d.jpg -c:v libx264 -vf "fps=25,format=yuv420p" output.mp4

Explanation: -f image2 tells FFmpeg to interpret the input as a series of images. %03d indicates a three-digit numbering format for the images.

7. Adding Simple Text Subtitles to Video

Burn subtitles into a video:

ffmpeg -i input.mp4 -vf subtitles=subtitle.srt output.mp4

Explanation: The subtitles filter overlays the specified subtitle file onto the video.

8. Extracting Image Frames from Video

Extract frames from a video:

ffmpeg -i video.mp4 -vf "fps=1" output%d.png

Explanation: fps=1 extracts one frame per second. %d is a placeholder for the frame index.

9. Converting Video to GIF

Generate a GIF from video:

ffmpeg -i input.mp4 -vf "fps=10,scale=320:-1:flags=lanczos" -gifflags +transdiff -y output.gif

Explanation: fps=10 sets the frame rate, scale=320:-1 resizes the width to 320 pixels and keeps the aspect ratio, and flags=lanczos provides high-quality scaling.

10. Capturing Webcam Video

Record video from a webcam:

ffmpeg -f v4l2 -framerate 25 -video_size 640x480 -i /dev/video0 output.mkv

Explanation: -f v4l2 sets the format to Video4Linux2, a common webcam format. -video_size 640x480 sets the resolution.

Building on the foundation with FFmpeg, let’s delve into some advanced techniques that can significantly enhance our video and audio processing tasks. These examples will focus on complex manipulations, optimizations, and creative tricks used in professional environments.

1. Complex Filter Graphs

Creating a side-by-side video layout from two video sources:

ffmpeg -i left.mp4 -i right.mp4 -filter_complex "[0:v][1:v]hstack=inputs=2" output.mp4

Explanation: -filter_complex is used for applying filters that require multiple input streams. [0:v][1:v] refers to the video streams of the first and second input files, respectively. The hstack filter horizontally stacks them.

2. Audio Noise Reduction

Reduce noise in an audio file using the afftdn filter:

ffmpeg -i noisy_audio.wav -af "afftdn=nr=500:nf=-15" clean_audio.wav

Explanation: The audio filter (-af) afftdn attempts to denoise the audio. nr is the noise reduction amount, and nf is the noise floor.

3. Color Correction

Adjusting brightness and contrast of a video:

ffmpeg -i input.mp4 -vf "eq=brightness=0.06:contrast=2.0" output.mp4

Explanation: The eq filter is used for adjusting brightness and contrast among other image settings. Values are adjusted to achieve desired visual effects.

4. Video Stabilization

Stabilize shaky video:

ffmpeg -i shaky.mp4 -vf "vidstabdetect=shakiness=10:accuracy=15,vidstabtransform=smoothing=30" stable.mp4

Explanation: vidstabdetect analyzes the video for camera movements and vidstabtransform stabilizes the video. Parameters like shakiness, accuracy, and smoothing control the sensitivity and the quality of stabilization.

5. Custom Text Watermark

Adding a text watermark with custom font style:

ffmpeg -i input.mp4 -vf "drawtext=text='Confidential':fontcolor=white:fontsize=24:fontfile=/path/to/font.ttf:x=(w-text_w-10):y=(h-text_h-10)" -codec:a copy output.mp4

Explanation: drawtext filter is used to overlay text. fontfile provides the path to a true type font. x and y set the position of the text on the video frame.

6. Encoding for Streaming

Optimize video for HTTP live streaming:

ffmpeg -i input.mp4 -profile:v baseline -level 3.0 -s 640x360 -start_number 0 -hls_time 10 -hls_list_size 0 -f hls index.m3u8

Explanation: -profile:v baseline ensures compatibility across devices, and -level 3.0 sets limits for decoders. -hls_time 10 specifies each segment’s duration in seconds.

7. Frame Blending for Slo-Mo Effects

Slow-motion effect by blending frames:

ffmpeg -i input.mp4 -filter:v "minterpolate='mi_mode=blend:fps=120'" slowmo.mp4

Explanation: minterpolate interpolates frames to increase the frame rate and create a smooth slow-motion effect. mi_mode=blend blends frames together.

8. Selective Crop and Scale

Crop a video dynamically and scale:

ffmpeg -i input.mp4 -vf "crop=w='if(gte(iw,ih),iw/2,iw)':h='if(gte(iw,ih),ih,ih/2)',scale=720:480" output.mp4

Explanation: This command dynamically crops the video based on its width (iw) and height (ih) and then scales it to 720×480.

9. Complex Audio Mixing

Mix two audio files with volume adjustments:

ffmpeg -i audio1.mp3 -i audio2.wav -filter_complex "[0:a]volume=0.5[a1]; [1:a]volume=0.8[a2]; [a1][a2]amix=inputs=2:duration=longest" mixed_audio.mp3

Explanation: Each audio stream’s volume is adjusted before mixing. The longest option sets the output duration to that of the longest input.

10. 4K Video Encoding with H.265

Encode video to 4K resolution with the H.265 codec:

ffmpeg -i input.mp4 -c:v libx265 -preset slow -x265-params pass=1 -s 3840x2160 -b:v 5000k output.mkv

Explanation: -c:v libx265 selects the H.265 codec which is efficient for 4K. -preset slow improves compression at the cost of speed. pass=1 applies single-pass encoding at a bit rate of 5000kbps.

Tailoring FFmpeg for complex scripting involves combining multiple commands, advanced filtering, and scripting capabilities to automate and refine your video and audio processing. Here are 10 advanced FFmpeg scripting examples that facilitate sophisticated workflows and processing solutions.

1. Automated Batch Processing

Batch convert all .mkv files in a directory to .mp4:

for file in *.mkv; do
  ffmpeg -i "$file" -codec copy "${file%.mkv}.mp4"
done

Explanation: This bash script loops through all .mkv files in the current directory, converting them to .mp4 without re-encoding the video or audio streams.

2. Video Splitting with Timecodes

Split a video file into chunks based on specific start times and durations:

ffmpeg -i input.mp4 -vf "select='between(t,10,20)+between(t,40,50)',setpts=N/FRAME_RATE/TB" -af "aselect='between(t,10,20)+between(t,40,50)',asetpts=N/SR/TB" segment.mp4

Explanation: The video filter select and audio filter aselect choose frames or samples between given time ranges (10-20s and 40-50s). setpts and asetpts adjust timestamps accordingly.

3. Dynamic Overlay Placement

Overlay an image onto a video with changing positions:

ffmpeg -i video.mp4 -i image.png -filter_complex "[1:v]scale=50:-1[ovrl], [0:v][ovrl]overlay=x='if(gte(t,2), (W-w)/2, (W-w)/t)':y='H-h-10'" output.mp4

Explanation: The image is scaled and overlayed onto the video. The x position changes dynamically with time t, moving to the center after 2 seconds.

4. Time-Lapse from Video

Create a time-lapse video by speeding up and stabilizing:

ffmpeg -i input.mp4 -filter_complex "[0:v]setpts=0.1*PTS,vidstabtransform=smoothing=30" timelapse.mp4

Explanation: setpts speeds up the video by a factor of 10. vidstabtransform is used to apply video stabilization.

5. Custom Video Intro Concatenation

Concatenate a custom intro with a main video seamlessly:

ffmpeg -i intro.mp4 -i main.mp4 -filter_complex "[0:v][0:a][1:v][1:a]concat=n=2:v=1:a=1" output.mp4

Explanation: The concat filter in complex mode is used to join the intro and main videos, maintaining both video and audio streams.

6. Motion Detection with Frame Extraction

Extract frames when motion is detected:

ffmpeg -i video.mp4 -vf "select=gt(scene\,0.4),scale=320:240" -vsync vfr frames_%03d.png

Explanation: gt(scene\,0.4) compares each frame’s scene change score to 0.4, extracting frames when the change exceeds this threshold.

7. Automated Video Montage Creation

Generate a montage from video clips stored in a directory:

files=$(printf "file '%s'\n" ./clips/*.mp4)
echo "$files" | ffmpeg -f concat -i - -vf "scale=iw/2:ih/2,tile=3x3" montage.mp4

Explanation: This script first lists all the MP4 files as input for ffmpeg, then scales down each clip and arranges them into a 3×3 tile montage.

8. Logo Watermarking at Specific Intervals

Add a logo watermark for the first minute only:

ffmpeg -i input.mp4 -i logo.png -filter_complex "[1:v] fade=out:st=58:d=2:alpha=1 [watermark]; [0:v][watermark] overlay=10:10:enable='between(t,0,60)'" output.mp4

Explanation: The fade filter applies an alpha fade-out to the logo at 58 seconds over 2 seconds. overlay is then applied with conditional timing.

9. Adjusting Audio Streams in Multiple-container Situations

Normalize audio and convert stereo to mono:

ffmpeg -i video.mp4 -af "pan=mono|c0=c0+c1,volume=volume=2.0" -vcodec copy output.mp4

Explanation: pan is used to combine two stereo channels into one, and volume is adjusted to ensure the audio level is appropriate.

10. High Efficiency Streaming Preparation

Prepare a video for high efficiency streaming with multiple bitrates:

ffmpeg -i input.mp4 -map 0 -b:v:0 500k -b:v:1 1000k -s:v:1 640x360 -c:v libx264 -g 48 -sc_threshold 0 -keyint_min 48 -hls_time 4 -hls_playlist_type vod -var_stream_map "v:0,a:0 v:1,a:0" stream.m3u8

Explanation: This command prepares two video streams with different bitrates and resolutions, suited for adaptive bitrate streaming over HLS.

These advanced FFmpeg scripting examples can be incorporated into larger video processing workflows, offering a toolkit for automated, high-quality video and audio production environments.