In my day job I regularly come across large text files which I need to inspect, edit, manipulate and so forth. By “large”, I mean something which is maybe 10 or 20GB - many millions of lines. I’m looking at a DOS batch script right now in fact, which is 9.4GB and has around 72 million lines but usually it’s data which is about to get pumped into a database - not always in ASCII either.
I often end up having to use HxD - a popular Windows hex editor because it can handle a large text file, allow me to search the file and make manual edits. Sometimes I just need to see what the text is in line 34,000,000 at positions 108-109 and change them.
However, editing in HxD is not ideal especially when record lengths are not consistent. I spent a lot of time looking for Windows Text editors which can handle extremely large files and I found Emurasoft’s “EmEditor”, which works but is relatively expensive.
I now need to find a Linux equivalent - a text editor which permits me to open a very large file, search, scroll up and down, replace some text, find the 81st million line and so forth without giving up immediately with a “file too large” failure.
I know I can use sed, awk and numerous other tools to change files, but it’s tricky to know what to change when you can’t actually see the text on the screen.
There is vlf mode for emacs. https://github.com/m00natic/vlfi It’s a minor mode that opens files in batches. You can customize the batch size. It supports searching and scrolling.
Well, I’ll be buggered. You’re completely correct.
I last tried to do this a year or so ago, and if memory serves I got a core dump or a message equivalent to “too large” when feeding a relatively small 2GB file which I needed to look at. In my head I’d dismissed vim as it puked.
Problem solved then! Thnaks for pointing out the obvious candidate does pretty much what I need after I’d dismissed it.
Going back over my notes I see I tried this on an AIX machine, which in hindsight would have been using an ancient version of vim, or possibly even vi.
I think you’re going to need to have a system with enough memory to open the file.
For example, if it’s a 20GB text file, you’re going to need that much available memory if you’re going to edit it. Keep in mind your os has a memory footprint. I ran into this problem when I had to use a hex editor to edit a VMDK that was 30gb. There wasn’t enough available memory to load it into the buffer.
Hopefully I’m wrong, but if I’m not you’ll need to load the file by section. Grep is your friend here, and check out the split command.
If that vim was built for 32 bits then a 2 GB limit makes sense. That’s 31 bits worth of size. Programmers often use signed integers and the sign bit makes 32 bits.
My experience is that AIX comes with actual vi (have no recent exposure to AIX though, but I’d be very much surprised if that had changed), unlike most Linux distributions where vi is a symlink to vim. So depending on how you called it you might have been using actual vi, rather than vim (assuming vim was installed at all).
As far as I can tell this is the case on every OS that strives to be fully POSIX compliant, which, as far as I can tell is all the big UNIXes (AIX, Solaris,…) and most, if not all, of the BSDs.
Vim has a “vi” mode, but I’m not sure if that is enough to pass POSIX, but I also don’t really see the point given the size of vi, and that most vim users wouldn’t particularly enjoy it behaving like actual vi very much
i was thinking the same. Also, maybe pipe less or tail? grep? Or do what I do and write a 3 line Python function with deque or a generator function to get what you need…