bedHedd's ntfs to btrfs adventure: log/guide

[bedHedd]

steps for how to convert a windows ntfs drive to linux btrfs

1. navigate to the ntfs2btrfs repo
2. download and install it for your OS. This guide will use Fedora. Open a new terminal

   •  dnf install ntfs2btrfs
     

     • in root/sudo
3. get the name of the ntfs partition to convert write it down if needed

   •  lsblk
     

   • NAME                            MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
     sda                               8:0    0   3.7T  0 disk 
     ├─sda1                            8:1    0     1K  0 part 
     └─sda5                            8:5    0 931.5G  0 part 
     ...
     

   • In this case I know that it is on the sda drive because it is the only drive with >3tb. if you need more information with the drive name, you can use the -f command to get the label and UUID. Alternatively if you use the gnome-disk-utility to check with a graphical interface
4. back up your drive data. I recommend rescuezilla the interface is a lot more approachable than clonezilla
5. start ntfs2btrfs (in root/sudo), recall the name of the partition and replace $partition_name with the name from the previous step

   • ntfs2btrfs /dev/$partition_name
     

     • in root/sudo
6. wait for it to finish. the output should look like this

   •  Using Zstd compression.
      Using CRC32C for checksums.
      Processing inode 153629 / 153629 (100.0%)
      Mapped 91054 inodes directly.
      Rewrote 0 inodes.
      Inlined 30767 inodes.
      Updating directory sizes
      Calculating checksums 219684372 / 219684372 (100.0%)
     

7. install the winbtrfs driver following the instructions in the readme

additional steps/troubleshooting

The following steps are what I used to finish setting up/troubleshoot my 4tb drive

how do I expand my partition

1. start by getting the partitions and available drives using lsblk in a terminal session

   •     NAME                            MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
     sda                               8:0    0   3.7T  0 disk 
     └─sda1                            8:1    0   1T  0 part /mnt/sda1
     
     

2. based on the output we want to expand sda1 to the entire sda drive
3. If your drive is > 2tb and is MBR, you’ll need to change the partition type to GPT
   1. ensure that gdisk is installed the command will change for your package manager

      • dnf install gdisk
        

        • in root/sudo
   2. run gdisk in root/sudo

      • gdisk /dev/sda
        GPT fdisk (gdisk) version 1.0.10
        
        Partition table scan:
          MBR: MBR only
          BSD: not present
          APM: not present
          GPT: not present
        
        
        ***************************************************************
        Found invalid GPT and valid MBR; converting MBR to GPT format
        in memory. THIS OPERATION IS POTENTIALLY DESTRUCTIVE! Exit by
        typing 'q' if you don't want to convert your MBR partitions
        to GPT format!
        ***************************************************************
        

   3. within the gdisk prompt, type w and hit enter. This is what you should see

      • Command (? for help): w
        
        Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
        PARTITIONS!!
        
        Do you want to proceed? (Y/N):
        

   4. respond to the command with y and hit enter. When it finishes, it will return the following

      • OK; writing new GUID partition table (GPT) to /dev/sda.
        The operation has completed successfully.
        

4. with the table set to GPT, lets continue with parted
   1. ensure that parted is installed the command will change for your package manager

      • dnf install parted
        

        • in root/sudo
   2. run parted in root/sudo

      • parted /dev/sda
        

      • GNU Parted 3.6
        Using /dev/sda
        Welcome to GNU Parted! Type 'help' to view a list of commands.
        

   3. confirm that you have the correct drive with print within the parted shell

      • (parted) print  
        

      • Model: ATA T-FORCE T253TY00 (scsi)
        Disk /dev/sda: 4097GB
        Sector size (logical/physical): 512B/512B
        Partition Table: gpt
        Disk Flags: 
        
        Number  Start   End     Size    File system  Name              Flags
         1      1049kB  1000GB  1000GB  btrfs        Linux filesystem
        

   4. if this looks like the correct drive, run the following command within the parted shell

      • (parted) resizepart 1 100% 
        

whoops I deleted my partition

1. install testdisk for your OS. In my case it is Fedora

   •  dnf install testdisk
     

     • in root/sudo
   • the command will return

     TestDisk 7.2, Data Recovery Utility, February 2024
     Christophe GRENIER <[email protected]>
     https://www.cgsecurity.org
     
     
     TestDisk is free data recovery software designed to help recover lost
     partitions and/or make non-booting disks bootable again when these symptoms
     are caused by faulty software, certain types of viruses or human error.
     It can also be used to repair some filesystem errors.
     
     Information gathered during TestDisk use can be recorded for later
     review. If you choose to create the text file, testdisk.log , it
     will contain TestDisk options, technical information and various
     outputs; including any folder/file names TestDisk was used to find and
     list onscreen.
     
     Use arrow keys to select, then press Enter key:
     >[ Create ] Create a new log file
      [ Append ] Append information to log file
      [ No Log ] Don't record anything
     

2. using the arrow keys hover over the [ Create ] Create a new log file option and hit enter to select it

3. using the arrow keys hover over the disk that you accidentally deleted the partition on and hit enter to select it

4. you’ll be greeted with this menu

   • TestDisk 7.2, Data Recovery Utility, February 2024
     Christophe GRENIER <[email protected]>
     https://www.cgsecurity.org
     
     
     Disk /dev/sda - 4096 GB / 3815 GiB - T-FORCE T253TY00
     
     Please select the partition table type, press Enter when done.
     >[Intel  ] Intel/PC partition
      [EFI GPT] EFI GPT partition map (Mac i386, some x86_64...)
      [Humax  ] Humax partition table
      [Mac    ] Apple partition map (legacy)
      [None   ] Non partitioned media
      [Sun    ] Sun Solaris partition
      [XBox   ] XBox partition
      [Return ] Return to disk selection
     
     
     
     Hint: None partition table type has been detected.
     

     • using the arrow keys hover over to [Intel ] Intel/PC partition and hit enter to select it. If you are using a different partition type, make sure to check if the other partition types are relevant

5. after selecting the partition type, you’ll be prompted with this menu

   • TestDisk 7.2, Data Recovery Utility, February 2024
     Christophe GRENIER <[email protected]>
     https://www.cgsecurity.org
     
     
     Disk /dev/sda - 4096 GB / 3815 GiB - T-FORCE T253TY00
          CHS 498074 255 63 - sector size=512
     
     >[ Analyse  ] Analyse current partition structure and search for lost partitions
      [ Advanced ] Filesystem Utils
      [ Geometry ] Change disk geometry
      [ Options  ] Modify options
      [ MBR Code ] Write TestDisk MBR code to first sector
      [ Delete   ] Delete all data in the partition table
      [ Quit     ] Return to disk selection
     
     
     
     
     
     Note: Correct disk geometry is required for a successful recovery. 'Analyse'
     process may give some warnings if it thinks the logical geometry is mismatched.
     

     • using the arrow keys hover over to [ Analyse ] Analyse current partition structure and search for lost partitions and hit enter to select it.

6. after selecting the partition type, you’ll be prompted with this menu

   • TestDisk 7.2, Data Recovery Utility, February 2024
     Christophe GRENIER <[email protected]>
     https://www.cgsecurity.org
     
     Disk /dev/sda - 4096 GB / 3815 GiB - CHS 498074 255 63
     Current partition structure:
          Partition                  Start        End    Size in sectors
     
     *=Primary bootable  P=Primary  L=Logical  E=Extended  D=Deleted
     >[Quick Search]  [ Backup ]
     

     • using the arrow keys hover over to [Quick Search] and hit enter to select it.

7. let testdisk scan for the partitions

8. if the data is recoverable, you’ll see this menu

   • TestDisk 7.2, Data Recovery Utility, February 2024
     Christophe GRENIER <[email protected]>
     https://www.cgsecurity.org
            
     Disk /dev/sda - 4096 GB / 3815 GiB - CHS 498074 255 63
          Partition               Start        End    Size in sectors
     >* Linux                    0  32 33 121601  25 24 1953519616 [MassStorage]
      P HPFS - NTFS          121601  25 25 124032 144 16   39061504
      P HPFS - NTFS          124032 144 17 131344 239 31  117473280
     

     • the first entry is the partition I am trying to recover, so using the arrow keys hover over to >* Linux 0 32 33 121601 25 24 1953519616 [MassStorage] and hit enter to select it.

context

while working on the previous to expand my drive, I wanted to remove the ntfs partition. I didn’t realize that as part of the migration, the ntfs partition wasn’t cleared from the disks, so when I tried to remove it I accidentally deleted the btrfs partition.

background

I previously asked the community for a suggested file system that can be shared between windows and Linux. After reading through some of the suggestions and some additional research, I came to the conclusion that btrfs was the move.

why btrfs?

why btrfs specifically

I currently use Fedora 39 as my work/general browsing system and have a separate drive for windows to play games with friends. When Windows 10 reaches EOL, I plan to switch to linux for gaming as most of my library is on steam. Therefore, I need a file system that is currently compatible with both operating systems and will make the migration to Linux easier

why not NTFS

Although NTFS3g helps bridge the compatibility gap. I would prefer to a file system that is open and well supported on linux

why not exFAT

Initially, I was going to go with exFAT as, it has better compatibility between the two systems. However when I read the post about how exFAT is more prone to data corruption/loss, I changed my mind,

Linux file system that can share between windows and linux

https://old.reddit.com/r/DataHoarder/comments/18cix5r/is_exfat_the_best_option_for_cross_macwindows/kcb5kht/ hmmm now this is swinging the pendulum back to btrfs Looking at btrfs. This repo has the driver for btrfs which links to this repo for converting a ntfs to btrfs app. I’m going to go with btrfs because it has journals so if the drive …

I didn’t read too much more but it seems that it is due to exFAT not having journals
https://old.reddit.com/r/editors/comments/6k3u4k/whats_the_problem_with_exfat/djj82k9/

[ThatGuyB]

@bedHedd

exFAT not having journals

Correct. exfat is an extension of fat32 to support more than 4GB files. It is a non-journaled file system. Journaled file systems include ext4, xfs, ntfs, btrfs, zfs, jfs and more.

When using windows (like, idk, 2014 to 2016 or so, when I still wanted to share data between the 2, before a complete split-brain approach), I preferred to use ext4 and I think I used DiskInternals (which is absolutely proprietary). Back then, ntfs3g was in pretty bad shape (in my experience, although I was a noob back then).

This guide sounds a bit insane to me, if you ask me, but I only glanced at it. You are deleting your partition table and create a new layout, then trying to recover the data from the deleted partition, back into the new partition on the same disk. This can only lead to data loss, if the wrong blocks get filled, unless, somehow, magically, TestDisk gets the blocks of the existing data and copies them in empty blocks on the btrfs partition.

TestDisk would be useful when you want to slap the old data onto a new disk. Again, I’m very ignorant of the inner workins on TestDisk, but my somewhat educated guess is that, even if TestDisk was aware of which blocks on the disk the data resided in, it wouldn’t be able to control the target btrfs blocks for copying the data.

[bedHedd]

@ThatGuyB

You are deleting your partition table and create a new layout, then trying to recover the data from the deleted partition, back into the new partition on the same disk.

I accidentally deleted my partition tables. Twice actually

@bedHedd

context

while working on the previous to expand my drive, I wanted to remove the ntfs partition. I didn’t realize that as part of the migration, the ntfs partition wasn’t cleared from the disks, so when I tried to remove it I accidentally deleted the btrfs partition.

@bedHedd

I also borked the partitions when I used clonezilla as well, so I had to do the same steps for testdisk. I couldn’t document it because I didn’t have a text editor to document my notes in, so it’s a blessing in disguise as I was able to write a guide

[bedHedd]

I had some issues today trying to write to my drive

ls -ld /mnt/sda1
drwxr-xr-x. 1 root root 498 Aug  4 16:31 /mnt/sda1

The instructions in this guide helped

Fedora Server - External Hard Drive is mounted as root

You first need to change the permissions on the mount point and the device mounted there. Then it should work. Do the following from the command line.

1. sudo mount <device> /path/to/mountpoint

2. sudo chown -R <your user:your group> /path/to/mountpoint
   If the device is not mounted when this command is run then ownership will revert to root when it is mounted.

3. ls -ld /path/to/mountpoint

4. ls -l /path/to/mountpoint
   Steps 3 & 4 are to confirm the chown properly worked.

5. sudo umount /path/to/mountpoint
   this is to enable the test of the next step

6. sudo mount -a
   This is to test the entry made in /etc/fstab and verify it will properly mount the device. Note that after doing step 2 you should need NO special options in the /etc/fstab entry. I use only defaults as the option on a device I mount as noted above. The fstab line should read something like UUID=xxxxxxx-xxxx-xxxx-xxxxxxxx /path/to/mountpoint ext4 defaults 0 0 with the UUID the actual UUID of the file system on that device and the filesystem type (field 3) the actual type of file system you are using. If you are using btrfs then the options (field 4) should be adjusted for that type file system.

7. ls -l /path/to/mountpoint
   This is to confirm that the chown done in step 2 above remains active and that it is ready for use at the next reboot.

one difference I noticed is that when I ran step 6. I got this output

sudo mount -a
mount: (hint) your fstab has been modified, but systemd still uses
       the old version; use 'systemctl daemon-reload' to reload.

I ran the suggested systemctl daemon-reload and reran step 7 and the output matched step 2

[NightMoose]

How has this been holding up for you?

[bedHedd]

Outside of the issue in the last update, haven’t had any major issues in windows or fedora