Linux file naming issue on Network drive

Summery
So I am running into an issue with my Linux server I just built (see parts list below). I installed xubuntu nightly build on it at first and now it has been updated to Xubuntu 20.04 LTS. I then went and set up a samba file share using the 6tb drive. The setup went smoothly except for an error stating that the netbios name was too long but I could still access the drive from my other systems. I then got all my media transferred to the 6tb drive which worked pretty well. Then I decided to install jellyfin and this is where the issues started.
https://jellyfin.org/

Jellyfin requires a specific naming clature so that it can pull the media info from imbd and other sites. My media was not labeled correctly so I went through the very time consuming process of manually re labeling or creating new sub folders to consolidate all of my media so it shows up properly. Most of this works fine but now if I try to access my network drive from a different pc some of the media will show up with incorrect names, show that their is no media their or show up with folders instead of files and state that they are not a directory. However when I go back to my server everything is labeled correctly and I can access all the files fine.

one example of a file miss naming was Spider man Homecoming
on my server it will show up as Spider Man Homecoming (2017)
however on my network drive that access the same folders it shows up as SLUN7X~U

Troubleshooting steps taken

The first thing I thought was maybe this is just a windows 10 glitch so I got on my HTPC which was running Ubuntu 18.04 and access the network drive but it shows the same file names.
I also tried accessing the network drive from my laptop which was running Linux Mint but showed the same file names.

Then I tried renaming the file to the exact same name that was displayed on my server but it said that the file already existed so its like it was reading the file correctly but not displaying the correct information.

I then thought that it might be the thunar file manger what was causing this issue so I took a live ISO of xubuntu and ran that on my laptop and accessed my network drive from that but it also showed the same name.

Then as a final step since I run my server headless and access it with a program called Nomachine https://www.nomachine.com/ maybe it was causing some issues with the naming on the virtual screen with an hdmi dongle since I don’t have a monitor connected to it. I plugged a monitor into the server and checked the files names but they showed up correctly their as well.

The only other thing I can think of that their might be an issue with the samba version I am running.
Samba version
Version 4.11.6-Ubuntu

Thunar version
1.8.14 (xfce 4.14)

If anyone has any idea on how to fix this I would be glad to hear them it is kind of a strange issue.

https://pcpartpicker.com/list/VcPY27

Well it looks like I figured out part of my issue. I needed to add another line to my samba global.conf file. The code I needed to enter was mangled names=no. This fixed the naming problem now i just have to figure out why files are ether not appearing on my windows machine or they are acting like folders that are not accessible on my Linux machine. They give the error this is not a directory. Which is true because they are just files not folders.

I did and update on my Xubuntu machine which upgraded samba and then on my windows 10 pc the files that were once just missing show up as folders with the error handle is invalid. I looked up that it would be good to added these 2 lines to my samba conf global section (see below)
unix extensions = no
wide links = yes
This however would ether remain the same or make the non accessible files just vanish again after a samba restart.

I had a samba issue with VLC on android where files would not show up or not be accessible when they did show up.

What fixed it for me was requiring smbv2 as the minimum protocol in the server config in the global section: min protocol = SMB2

The vlc app does have an option to force smbv2, but using that option did not actually make the app use v2.

That might be worth a try if you don’t have any other ideas.

Just tried this and restarted samba and now on my windows 10 machine the files come up as folders again and give a new error stating that the directory name is invalid, which would make sense since they are files not folders.
I should also state that this seams to happen randomly on my network drive some files are perfectly fine and others files are showing up as folders. I think this has something to do with naming scheme but other files I renamed and they show up perfectly fine.

Is your naming scheme flirting with the 255 character limit for filepaths in windows. mount/cifs also has the same limit.

That might be the case more of the folders that have files in them are usually separated by a season folder like season 1 season 2 etc and more of those are not accessible or they show up as files instead of folders. But then I have other files that are separated by season with really long files names that come up fine.

I also tried taking an episode of a show and copying it up to the main folder but when I tried to access it on the network drive it would still show up as a folder instead of a file.

Have you seen this bug report regarding samba on 20.04 and regular files being presented as directories? https://bugs.launchpad.net/ubuntu/+source/samba/+bug/1872476

Edit: At the end of the thread, it looks like a “fix” was applied to 4.11.6 this morning: https://bugs.launchpad.net/ubuntu/+source/samba/+bug/1872476/comments/44

That is my exact issue I will look into this. I did update xubutnu this morning and just restarted and in the updates it did show that samba was getting updated but it still showed as folders maybe I don’t have the right repo I will look into this.

Darn the update did not work for me. I tried updating my system and I also tried downloading the ppa that was on the list and adding this line to my samba conf file client min protocol = CORE but they still show up as folders. Maybe the update that he pushed hasn’t hit xubuntu 20.04 yet. Ill wait a bit and see what happens on the next cycle of updates.

Sorry for the necro, but I am having what appears to be the exact same issue, however my system (samba server and the samba client) are both up to date. And it seems to be just a few files that are presenting as folders. Did you ever find the solution to your problem?

Edit 1: Only files copied to the server on sunday seem to be effected. And if I ssh into the server and delete the file and recopy it to the server, the file no longer is presented as a folder.

I never figured out how to fix it. I did find out that if I named files through the network drive on a different machine and not directly on the server then the files work perfectly fine. Also I did see on some of the threads that having the most up to date version of samba fixed their issue. I run xubuntu 20.04 but even that distro doesn’t have the most up to date version I think they are on samba version 4.13 and ubuntu ships with 4.10 or something.