H265/HEVC is a viable alternative to H264 encoding in some circumstances.
Terms:
H265/HEVC- Formal Specification
H264- Formal Specification
Notable H265 HEVC Implementations: x265 (used by FFmpeg), Intel MSS HEVC Software/GAcc
Notable H264 Implementations: x264 (used by FFmpeg and VideoLAN)
Intro:
Next generation codecs promise identical quality to H264 at half the file size. In practice, this promise has yet to achieve a full realization. To understand the use cases where H265 currently excels, consider that codecs are designed to balance quality, encoding/decoding time and file-size (bitrate). H264 is an incredibly well-balanced codec, excelling in all areas. H265 is an imbalanced codec that shines compared to H254 when given computational ability (time) and when allowed to slightly reduce quality.
Quality:
Encoding quality, while mostly subjective, can actually be measured objectively via SIMM (structural similarity). Using both this metric and subjective tests, HEVC is superior at any given SIMM value/bitrate.

1) http://x265.ru/en/bolshoe-sravnenie-kodekov-h265-ot-mgu/
2) https://blogs.gnome.org/rbultje/2015/09/28/vp9-encodingdecoding-performance-vs-hevch-264/
Please see Moscow State University's research “HEVC/H.265 Video Codecs Comparison” especially 5.3.3 Figure 17 on page 24 (1). Also see rbultje's blog “VP9 encoding/decoding performance vs. HEVC/H.264” especially the section “Encoding quality” (2).
Encoding Time:
In practical terms however, this increase in quality comes at a significant cost in encoding time with very little improvement in lower bitrate. In the best case scenario, as per figure 17, to match the quality of x264, an x265 encode takes 3.25 times longer to encode. In addition the average bitrate for a fixed quality of x265 is 83% relative to x264 (5.3.5 Table 5).
This means that if a video encodes using x264 to 300 MB in 1 hr, the x265 equivalent would take 3hrs 15min and be 249 MB.
Due to H265 codec design principles (more on this below), certain panning videos that containing lots of small details will take 2.8x longer to encode and will take slightly more bitrate resulting in larger file sizes (5.3.3 figure 18). In situations where time is more of a factor, the average bitrate for a fixed quality of x265 is 94% relative to x264 and the encode will still take 40% longer (5.2.5 Table 4, 5.2.3 Figure 10). In a third scenario, if time and quality are both key factors, all the benefits of HEVC disappear completely with x265 taking 117% the bitrate that x264 does for a given quality (5.1.5 Table 3).
File Sizes:
Some people don't care about file sizes, some (like me) do. The bitrate of a stream directly determines its filesize. To compare quality at a given bitrate is to measure the resulting filesize. Thus to desire low file sizes means to want a low bitrate for a stream. Larger resolution videos take more bits to represent. A 1080p video at a given quality/codec will have a larger size than the 720p equivalent.
The appropriate file size, resolution and quality depends upon the intended use case:
- to play on a 240p iJunk screen
- to play on a 4k TV
- to stream at 480p to a netbook
- end-user archival
- editing archival
In addition, the appropriate balance between file size and resolution/quality is user-specific and I will not attempt to discuss that here objectively. Instead, I'd like to outline HEVC design considerations and lay out certain heuristics to make it clear when HEVC should and should not be used.
Codec design considerations:
- The idea in designing HEVC was for 4k video to be the same size as a 1080p H264 one.
- Due to H264's macro block size limit, H264 tends to hit a computational “wall” at which point throwing more computational ability at the encode will no longer reduce the bitrate.
- HEVC does not have that same “computational wall.” The difference between a preset “slow” and preset “very slow” H265 encode is about 15-25% (this is anecdotal), where as it was 1-5% with H264.
Given enough time:
- On average, an identical quality x265 encode will take 82% that of an x264 one (6.1.3 Figure 24). In practice, this varies dramatically depending upon the video source.
- An x265 encode of certain sources will encode to a similar size as the audio stream (KILL la KILL EDs) and other sources show no improvement over x264 -at all- while still taking longer to encode (5.3.3 Figure 18).
- Video, in general, encodes well (to small sizes):
- when it does not have a lot of change between frames
- when change is minimal
- and when large blocks of a video are the same color.
- This describes certain anime series' very well, much more so than live-action footage.
- H265 can take more advantage of these fundamental characteristics than can H264.
- A slightly lower quality 720p x265 encode will take about 20-25% or 1/5th to 1/4th the space as a typical quality 1080p x264 encode (anecdotal). This implies the same amount of space could be used 4-5 series at 720p x265 over one series at 1080p x264.
Decoding Time and Compatibility:
Decoding time is mostly a measure of how well a given decoder implementation can use its current hardware (available instruction sets) and partly a measure the software's efficiency. If hardware assistance is not available, then a software-only decoder must be used instead. An incompatible video encode can refer to either hardware devices that do not have the option of using software-only decoders (TVs/iJunk) or to decoders that do not have sufficient computational capacity to decode the bitstream in real time using only software (tablets/9” netbooks playing 4k HEVC).
While decoding time and device compatibility are king for most video encodes scenarios, this has taken a back-seat in the anime community (note the near-instant switch from xvid->h264 and the popularity of mkv, advanced substation alpha subtitles, and 10-bit). Personally, I couldn't care less about stream compatibility. Anime does not exist to play on your device; your device exists to play anime.
H264 and H265 Recommendations (tldr):
Use x264 when:
- Time is a key factor
- Compatibility is important
- Desiring more consistent file sizes (why?)
- Retaining a minimum of 99% the quality of the original is a must
Use x265 when:
- Time is not a factor
- Low file sizes are important
- 95% the quality of the original is “good enough”
Note: There is very little difference between in quality/bitrate between HEVC and H264 when HEVC implementations are not given sufficient computational capacity (in the form of time) and using HEVC will make the encode take longer. In that situation, use an H264 implementation instead.
Random:
Most Anime: x264, 720p-1080p, crf=16, aac, 10-bit, yuv420p
Personal fav atm: x265, 720p, crf=17, opus, veryslow, 10-bit, yuv444p
but might switch to 1080p @ crf=20
Edits:
* Grammer and spelingz
* Updated references
* I do not really understand how up-scaling works in relation to quality/bitrate, so I have removed most references to it pending more testing. The post is not really about that anyway.