This is the page from the manga that breaks down what happens at each level. Level six seems to indicate that it goes beyond brain damage to physical alteration.
So far though, all I've heard mentioned is that they are the symptoms of trying to climb back out, which seems to suggest that if you stay down there you'll be fine. That explains the Capital of the Unreturned at Layer Six - anyone there who attempts to climb back up will die or become something else.
The fact that it mentions decompression sickness makes sense since that's what happens when deep-sea divers climb back up too quickly - the dissolved gasses in your lungs and bloodstream expand rapidly like opening a can of coke.
I'm guessing it does at least partway have something to do with atmospheric pressure since it increases very quickly the further down you go.
Sea Level - 100kPa (making the gross assumption that this is what this world's sea level pressure is)
-1000m - 114kPa
-2000m - 127kPa
-3000m - 143kPa
-4000m - 160kPa
So you're not even halfway through Layer 3 and the atmospheric pressure just increased by 60%. by the time you reach the Cup of Giants at -7,000m, it's more than doubled at 218kPa. At the Capital of the Unreturned, it's at almost 400kPa. at -20,000m, it's over 700kPa.
Bear in mind that the deepest hole that mankind has ever dug doesn't even reach 12,500m, and the deepest anyone has ever gone underground only reaches 4,000 meters.
/rant. Sorry, the more I think about it, the more excited I become about the premise of the story. WANT. MOAR. MUST. RESIST. URGE. TO. READ. THE. MANGA.