Lossless Vs Lossy (ABX Comparator)

Audio Hardware & Software Sound Cards & DACs & Other Hardware
1

Hey Guys,

Obviously Logans video of audio myths created a stir with audio enthusiasts. Most people agreed with what was said but the hardest reaction from the comments was do with mp3 vs flac.

Now there's a software program/plugin you can use to do a blind test between files its called a ABX Comparator. What I was hoping to do is to get a group survey of how well everyone is able to tell between lossy and lossless formats.

 

Just download foobar (http://www.foobar2000.org)

Then Downlaod & install the ABX plugin (http://www.foobar2000.org/components/view/foo_abx)

Add 2 different music formats of the song to your playlist, right click > utilities > ABX two tracks

It'll have the 2 songs mapped to 4 buttons, the aim is to match the same files blindly.

 

If you plan to give it a go, please post your results. Give as much detail about your audio setup as possible (DAC,  amp, headphones or speakers), the song, quality of the files and where you sourced the lossless files from (cd, hdtracks.com, ect). There are just too many variables in audio.

 

My Attempt:

I briefly compared Step by Vampire Weekend on my computer. Flac was sourced from the Album CD and the MP3 was 320kbps (CBR), I used my M50 headphones connected to a sound blaster ZxR sound card (all sound enhancements and EQ were turned off), I couldn't tell the difference at all, I failed every test. I tried multiple songs from the album and still failed. I haven't had a chance to try it with my proper audio setup yet though, which consists of a NAD amp/dac and my acoustics concept 20 speakers. But it's 1am where I am and I'm tired so I'll do it tomorrow and may be try some different music, tiredness also means there's probably stuff wrong in this post, like gramma, formating and other stuff so sorry)

 

 

TL;DR

Download foobar

Download ABX plugin http://www.foobar2000.org/components/view/foo_abx

Compare 2 of your music files

Post results with your audio setup, song and info about the 2 files you used for comparasin

 

Edit (07/01 6:10pm AEST):

Removed my comments about comparing bit/sample rates because someone claimed that the foobar ABX Comparator up scales the lower bit/sample rate. I haven't been able to find anything that confirms or denies this but for now we can't be sure. Even though I personally believe that should not make a difference it's not fair to act that my opinion is the only one that is right in a thread about hard facts and proper blind testing. 

I'm a bit short on time but if someone is able to find a way to properly compare bit/sample rates played at thier intended and original rates I will update this post with the new method. Otherwise I will do my own research when I'm free next.

Update: (07/01 10:40pm AEST)

So far the best information I have been able to find is the foobar ABX comparator will convert both files to 32 bit PCM for comparison. Since both files are being unscaled to 32 bit, both audio files will be changed. This will not degrade quality but it will still change the audio. This also means that unless you are using the WASAPI plugin windows is going to be downscaling it again. Whether this has an effect your audio is up to you. But since both of the files you are comparing are going to be up scaled they are not going to loose or gain any details. The audio theoretically will be different from the original but since both the files will be different I doubt this matters at all in a comparison. 

I am not an expert on this so feel free to debate, but for now it should be fine to do an accurate comparison between files of different bit/sample rates. Would also help if someone could find a better source.

(http://www.whatsbestforum.com/showthread.php?5266-Foobar-ABX-testing)

 

Update: (08/01 02:20am AEST)

The biggest argument so far is that windows is interfering with the audio, to remedy this I'm going to use a use an ABX comparator on a Linux OS. Im going to be using my AKG K702 connected to a NAD D 3020 amp/DAC.

I'll post what linux os and abx comparator I'm going to use once I have everything up and running. Tomorrow of course, its a bit late over here in aus -_-

Update: (08/01 10:34pm AEST)

Well after messing around with linux I was able to play my music at bit perfect audio, first impressions was that even without a abx comparator I was unable to notice any improvement from windows. It was definitely not worth loading up a whole new OS just to listen to my music. I'm pretty much a noob when it comes to linux because I personally haven't had a need for it but I've played around with it in the past. I had trouble getting the abx comparator to play nice with JACK, it was eventually taking too much of my time for something I was not going to continue even if the results were better. Booting into a linux OS just to listen to my music is not something I plan to do anyway.

This does not mean everyone wouldn't want too, if you willing give it a shot.


Update: (08/01 11:50pm AEST)

So here's a neat trick if you want to hear what audio mp3 removes from lossless. In Audacity load both your flac and mp3 files then invert the mp3. The audio in the mp3 that matches the flac will be cancelled out only leaving what the flac has left. Make sure to align the audio as best you can, if their not exactly synced the audio will not cancel out properly.

Example: 

When I took my flac version of Unbelievers by Vampire Weekend (https://www.youtube.com/watch?v=2_qKmTLbEPc)

Then applied the method to issolate the sounds mp3 removes: http://tinyurl.com/ny9aq4b

2

Great posts. Also anyone who wants to argue about bit rate should read this paper.

http://www.aes.org/e-lib/browse.cfm?elib=14195

Quote  - 

16 bits is enough to span the real hearing range with room to spare. It does not span the entire possible signal range of audio equipment. The primary reason to use 24 bits when recording is to prevent mistakes; rather than being careful to center 16 bit recording-- risking clipping if you guess too high and adding noise if you guess too low-- 24 bits allows an operator to set an approximate level and not worry too much about it. Missing the optimal gain setting by a few bits has no consequences, and effects that dynamically compress the recorded range have a deep floor to work with.

An engineer also requires more than 16 bits during mixing and mastering. Modern work flows may involve literally thousands of effects and operations. The quantization noise and noise floor of a 16 bit sample may be undetectable during playback, but multiplying that noise by a few thousand times eventually becomes noticeable. 24 bits keeps the accumulated noise at a very low level. Once the music is ready to distribute, there's no reason to keep more than 16 bits.

Another quote - 

In 554 trials, listeners chose correctly 49.8% of the time. In other words, they were guessing. Not one listener throughout the entire test was able to identify which was 16/44.1 and which was high rate [15], and the 16-bit signal wasn't even dithered!"



3

Was that like the clinical trials that confirmed that genetically modified food and all kinds of crop spraying chemicals are completely safe and that everyone should eat Monsanto quality food?

Come on guys, don't make this absurd, stick to science and respect the fact that nobody will be able to tell which one is which, mp3 or flac or wav, 16/24/32-bit, etc... but they're not equal either, they just don't sound the same. So you can test everything towards one hypothesis or the other, if you test besides the point. On Windows, everything is forced into 32-bit floating but requantified in the most broken and stupid way imaginable to mankind, and look at how shitty it sounds in comparison to even 16/44.1 on a Mac or a 50 USD portable CD player...

Those that have experienced audio playback fidelity at a quality level that it invokes the emotions of the live experience, will not want to turn back, and they won't be listening to any of the arguments because they fucking know better because they've experienced it. You can't argue with a person's experience. It was great to scare people away from buying useless consumer "gaming" soundcards, great, but now it's going too far, there is a serious pile of bullshit in the process of being piled up, so high that everybody is losing sight of the essence.

4

I did the test using The Akg Q701s for headphones, and the Fiio E10 for an amp/dac.

The first song I tested was "Setlon" by Eluvietie. Flac vs 320kbps - Can't tell the difference, Flac vs 256kbps - Still can't tell the difference. With 128kbps mp3, audio quality sounded a bit watered down. There was some noticeable loss of treble in parts of the song and the bass didn't seem to sound as full as it did in the flac version.

The next song I tested was the song "Teardrop" by Massive Attack. I compared a flac format of the song with a 320kpbs mp3 format with no noticeable difference. Then with 256kpbs still no noticeable difference. With 128kbps mp3, overall slight, but noticeable decrease in quality just like the first test.

 

5

"On Windows, everything is forced into 32-bit floating but requantified in the most broken and stupid way imaginable to mankind"

evidence?

6

Also here is a another great comparison to try at home: http://youtu.be/BYTlN6wjcvQ?t=45m49s

Since youtube compresses audio a lot he supplied the wav files for download: http://ethanwiner.com/aes/

7

Makes sense.

8

Are you using windows XP ? If you are using anything later than that (windows) than the mixer is probably mangling the sound so your test is pretty close to irrelevant - That is unless your are bit streaming the audio to an external audio receiver using Foobar's WASAPI plugin. If you are running the sound through windows 7 mixer, it is re-sampling it, no matter what Foobar's out put sample rate is set at.I am pretty sure it is the same with Vista but not sure about with windows 8.



There are other limitations to using ABX Comparator, especially if you want to compare files of differing sample rates. Foobar will re-sample at least one of the files when it creates the comparison files.The re-sampling alone is changing how it sounds.

 

 
 
 
9

I'm not fully experienced with all of this so don't take my word as law, but isn't that why you can change the bit & sample rate in the playback settings of your speaker properties. That sets the sample rate for the whole system, lower sample rates with be upsclaed (no difference in audio) and other sampled music will remain the same. Only time the audio will be degrade is if it is downscaled because the win playback rate is set lower than the file your trying to play. (http://www.hydrogenaudio.org/forums/index.php?showtopic=89281)

Also I mainly created this post to compare lossy and lossless files and how lossy file formats actually removes sound it thinks you cant hear to reduce the file size. So lossy is actually removing poarts of the audio, so in thoery lossless should sound better but when compared like this there is no noticable difference. (in my setup at least). I just added bit and sample rates in case anyone else wanted to compare them.

10

For quite some time, I've been a firm believer that there was an audible difference between FLAC and high quality MP3's.  Due to all of this hubub lately, I decided I would do a fairly thorough ABX between a FLAC source and MP3's sourced at varying bitrates.  The results of the test was enlightening, to say the least.

 

Testing Equipment

As it stands right now, I don't have anything all too fancy.  I used my pair of Sony MDR-MA300 open headphones through my computer's onboard audio chipset (Realtek ALC892).  This is the primary way that I listen to music, so I'm thoroughly familiar with this setup's particular sound.

Testing Methodology

Using DBPowerAmp, I encoded several MP3's of various sample rates from a FLAC version of the song that I ripped from a CD using EAC.  I created 5 MP3's to test using this method.  Since (according to my understanding of the encoding methods) variable bit rate encoding has its advantage over constant bit rate encoding, I focused primarily on VBR samples.  I encoded MP3's in V9, V6, V3, and V0, with a lower number corresponding to a higher bitrate, as well as one sample in CBR 320.  Starting with V9, I found a portion of the track that was fairly revealing of the flaws in the encoding.  This section was from approximately the 13 second mark to the 30 second mark.  From here, it was standard ABX comparison:  Listen to A.  Listen to B.  Discern any differences.  Match A with X or Y.

http://www.youtube.com/watch?v=sdvacK6Cmxo

The Results

With V9 and V6, I was able to easily discern the difference between FLAC and the MP3.  In 11 tests, I answered correctly 11 times.  I was not guessing, I was matching.  However, when it came to V3, things were much less clear.  There was certainly a faint difference in the sound, but it was too hard to correctly match up.  I scored 2 correct out of 5 before giving up.  V0 completely dominated me.  I could tell no difference and gave up after only correctly matching 1 out of 4.  CBR 320 held identical results.

My Experience

In the lower quality MP3's (V9-V3) there were some definite auditory differences that slowly faded as quality increased.  In V9, the bassline sounded "choppy" for lack of a better term.  In all of them, the drums sounded muffled (with V3 being within margin of error).  I couldn't tell any difference in guitars or vocals in V3 and higher (and didn't need to listen long enough to get to them in V9 and V6).

The Conclusion

First, it should be noted that I went into this test quite biased.  I 100% expected that I would be able to tell the difference between FLAC and MP3 with greater than 75% accuracy.  However, there is definitely a point where Lossy becomes indistinguishable from Lossless.  This point is roughly at V3.  V0 and 320 CBR were virtually indistinguishable from their FLAC source. 

I'm highly interested to see how this scenario plays out in different genres of music.  Will it be easier to tell the difference in classical or opera?  Will it be harder to tell the difference in electronica?  These are questions that I feel warrant further investigation to satisfy my curiosity, and I will be testing this further in the next few days. 

In these next tests, I will also conduct my tests with more scientific rigor, using the same number of samples (11) for all bitrates, no matter how awfully I'm doing.  This will yield more accurate results that can be taken far more seriously from an objective viewpoint.  I will publish my findings in a blog post here on Tek Syndicate (keep an eye out for it) and may also record and upload my test findings to youtube, so any cheating can be ruled out.

 

I hope that this has been informative, and I urge you all to do this test yourself.  We all perceive things differently, and I would be interested to see if anyone can accurately tell the difference between the two sources.

11

Honestly i will be impressed if you can differentiate between mp3 and flac past the 320Kbs mark consistently too.

12

I was the same as well. "FLAC all the way" then when I watched logans video I decided to properly test it and was surprised at the results. Part of the reason I posted this thread is too see if it was just my audio setup or the genre of music I listen to, I'm eager to see more opinions and results. 

Thanks for your lengthy and detailed input, look forward to your findings in your blog post.

13

Thanks for taking the time to read it.  Deciding which tracks to use for test samples is going to be interesting.  I'm definitely going to use the same track that I did in this one (especially since I know exactly what I'm looking for in that one) and I think I'll do 9 other tracks of varying genres and complexity.

14

Most of the people arguing against us with the mp3 vs FLAC are saying that our music isn't "correct". As a friend on my other forum stated - It could be a song of a walrus farting, but the point is if you can hear a difference.

They are suggesting songs with "great acoustics" whatever that means. Opera and classic rock is what I think they're getting after. Something from Beethoven would work.

Basically anything around 320 kbps (AAC as well) is nearly impossible to differentiate from FLAC for all of my music. I have everything from Trace, to Rock from the 90's and early 00's, to deathmetal and some trap.

Call me what you want but I'm just an average guy with average hearing, but to be honest, I wish I was like all of these people who can claim to hear a difference. If you can hear a difference, great! You have an advantage over the rest of us. 

15

I'm definitely expecting similar results with every other track that I test.  I'm mostly going to do the testing so that people can't complain about not using a wide variety of music.

16

My thoughts exactly.

17

Just finished doing all the ABX testing for the first track.  Videos are uploading to youtube as I type.  I'm going to keep them private until I get all the videos uploaded and I have my analysis finished.  Once everything is done, I'm going to publish it all at once.

18

I know it's a little early but how did you do?

19

With proper hardware and proper source, you can. Really !!

First, DO NOT use on-board DAC of your mainboard, not even a dedicated PCI/PCIe soundcard with analog output (but something like RME's MADI FX is OK, as long as you use another DAC over MADI connection to provide the analog output). You'd better off with outboard DAC, be it a USB/Firewire/HDMI/Optical/Coaxial/MADI/AES-EBU connected DAC.

My setup consist of a Macbook Pro Retina 15" connected to NAD M51 DAC through either USB or HDMI. Then from my NAD, the balanced output goes to ADAM Audio Sub24 Subwoofer with high pass enabled for the "satellite" speakers @85Hz. Then from ADAM Audio Sub24, the connection goes to the "satellite" speakers, a pair of ADAM Audio S3X-H. All three speakers are calibrated to be flat in my room up to within 5 dB in all frequency spectrum with decay time flat in ~200ms from 100Hz up. 100Hz down, the decay time is still around 500ms peaking at 40Hz. I use Neutrik convertCON (NC3FM-C-B) for all cable termination. The interconnect cable is Mogami W2549 High Quality Balanced Microphone cable (as written by them on their site).

All MP3s are at least 320Kbps @44.1KHz sample rate. AAC export has no settings, and all files are produced by Apple Logic Pro X.

With this system, ALL MP3 SOUNDS BAD (compared to the uncompressed or the losslessly compressed material). BAD in terms of the highs are ALL "muffled", there are no "air" - as in like when you hit a snare drum or a cymbal in a room, there should be a very short decay time after the "main hit". With recordings that I know has such decay time, there was NONE in the MP3s, AAC files fares a slightly better, but still no where near lossless or the original AIFF/WAV files. The highs of the compressed formats are ALL "gritty". High frequency of the compressed formats are like "cracking" sound, it's like very hard-hitting, almost like it's close to clipping.

Here's a spectrogram of an AIFF file I use to test, clocking @ 49.7MB uncompressed:

 

Here's a spectrogram of an MP3 file I use to test, clocking @ 11.3MB:

Here's a spectrogram of an AAC file I use to test, clocking @ 4.6MB:

 
 
And just for the sake of comparison, the original 24bit/96KHz file, clocking @ 96.4MB, compressed ALAC:
 

The X-axis is TIME, Y-axis is FREQUENCY, the colors are how loud the sound at any corresponding TIME/FREQUENCY. Note: the lossless (or uncompressed) data here is the source file (which is a transcode from a 24bit 96KHz AIFF file). This is the easiest way for me to tell the story. see the purple haze? That's audio information that's low in volume, the sound decay, the reverb, the tailing echo of a piano concerto in a great sounding hall, the after-glow (if you will) of sound. In the MP3, see that the purple haze is reduced somewhat, this reduces the so called "airy-ness" of the music, the harmonics, the in-between sounds, the "air" is getting the boot simply because the compressor THINKS that we won't be able to hear it. if the "main" content of this particular music is around -75 dB, the purple haze is only some 55 dB to 60 dB less that the "main" audio band, that's where the echo/reverb/air is at. Lose that, and you will lose A LOT of musical information. The halls seem unnatural, the drums sounds muffled, the trumpet is "dead" in a sense that it's like recorded in a bad/dead room. Moving on into the AAC, here's where all of the high frequency is CUT down to a hard limiter at around 17.5KHz, and the purple haze above that frequency is GONE, producing a very small file. Now looking at the original 24 bit 96KHz file, the purple haze is reaching stratospheric HIGH, well beyond 20KHz - our so called theoretical hearing limit - BUT the information IS THERE, it's the 7th octave, 8th octave (and probably 9th octave) of harmonics. Those are the hall's reverb, the tail of "aahhh-s" and "ooohh-s" in a church choir. That's HUGE information discarded by the lossy compressor simply because it thinks that we can't hear it.

There's are HUGE differences in between formats. I personally do not like headphones, simply because headphones can't be calibrated - to be as flat frequency response as possible. There always going to be comb-filtering in the upper-mid-to-high frequency (on the driver it self). That's not even considering HRTF factors... For those who interested on finding out the difference between the lossy and lossless compression - there's an easy way to find out, do a NULL test. It will show you the difference in the most eye opening way.

For me, there's no getting away with ADAM Audio studio monitors, the compression artifact is all there to seen/heard. If your music is badly mixed/mastered, it will show on these speakers. Abbey Road studios (studio 2 IIRC) use this exact speakers (but not the Sub24 probably). For me, a studio monitor IS A MUST, especially the better ones. I my self prefer ADAM Audio simply because their speakers reveals MP3s like nothing else. My previous library was built on 320Kbps MP3s, now I'm (very slowly) rebuilding my library on ALAC/FLAC/AIFF files. There is no going back once you know what to find...

For those who has enough money, get the cheaper ADAM Audio A7X (it's around $1500 a pair IIRC) to see/hear what I mean.

zzz2496.

20

Whenever there is re-sampling, there is always a change in audio. The question is 'Can you hear it ?' Experiment with some of Foobar's re-sampling plugins, there are at least 3 that I know of. Notice all the different re-sampling frequency settings and the quality setting for each individual sampling rate. Each quality is different, consequently each sounds different.