Text editor to open big (giant, huge, large) text files
[+1023] [2] Yuvi
[2008-10-01 20:22:35]
[ windows xml editor text-editor large-files ]
[ https://stackoverflow.com/questions/159521/text-editor-to-open-big-giant-huge-large-text-files ]

I mean 100+ MB big; such text files can push the envelope of editors.

I need to look through a large XML file, but cannot if the editor is buggy.

Any suggestions?

(166) Actually, text files of 100+ MB or even 1+ GB is not as uncommon as you may think (i.e. log files from busy servers). - Anders Sandvig
(15) Sneakyness: And not exactly text. I think the requirements of reading text files and reading binary files differ somewhat. You might pass it through base64 or uuencode, though. - Joey
Back in 1995 I used WinWord to open 64MB files on a 16MB machine. I'm sure it would do just as well 15 years later. - Gabe
To generate random text files instead of binaries, use this: cat /dev/urandom | tr -dc 'A-z' | head -c 1000000, where the last number after -c is the number of bytes in the file. - indiv
Microsoft Office Access can actually read and parse very large xml files, but will only make sense of it if the xml format fits with something that it can translate to a table. - Niels Brinch
if using vim :set binary superuser.com/questions/364012/… - daydreamer
(2) This should be at least a similar question or even linked as it was asked 18 months prior... stackoverflow.com/questions/102829/… - ONDEV
(1) I was also looking for the answer to this exact question in order to read some huge log files that I've generated! - HorseloverFat
Here's my fallback: GigaEdit (heliwave.com/GigaEdit.html). Nothing fancy, but small, portable, free and opens massive files in an instant. - Carl
(1) @BlairHippo I feel the same way, I'm almost nervous when asking a question because chances are high that someone will say "Close this, it should go in WhateverExchange instead" - Rodolfo
@Sneakyness this can be used to generate large files in a matter of seconds as well. grep -r "someText" . > bigfile assuming that there are some files in your dir that contain matching lines with the search criteria. Of course, you would need to stop grep forcefully as this it will make it enter in a endless loop :) - user624558
amolnpujari.wordpress.com/2012/03/31/reading_huge_xml-rb its so simple to deal with large xml in ruby - Amol Pujari
To view files, I recommend to use this online viewer - readfileonline.com - you don't have to install any programming interface, it works in every device and OS. - Ivan Kuckir
On a windows machine with powershell > Get-Content C:\Scripts\Test.txt -totalcount 3 - dhysong
winasm.net/free-small-fast-text-editor.html Free and very fast - Bernhard
You can try this online jenson.in/demos/open_giant_files_in_browser.php - Jenson M John
First ask yourself this: do you actually want to edit a file >1GB in size, or do you just want to view it quickly, and be able to edit other, "normal" files? In the latter case you'll have a much better choice of log viewers and text editors. - rustyx
[+1243] [2008-10-01 20:25:19] Nick Ryberg [ACCEPTED]

VS Code [1] (Windows, macOS, Linux) - Free and Open Source with a nice GUI. Edited a 3.6 GB JSON file, loaded in a minute. You must have enough RAM to load the files.

• glogg [2] (Windows, macOS, Linux) – Confirmed to handle multi-GB files. Its main feature is regular expression search. Has tabs, reads files directly from disk, can watch/follow files, and allows user to mark lines.
• LogExpert [3] (Windows) – "A GUI replacement for tail." Supports file following, searching, filtering, configurable highlighting, plugins, and external tools.
• Large Text File Viewer [4] (Windows) – Minimalist and has very small executable size. Supports split view, text theme customization, regex search, and file following.
• Lister [5] (Windows) – Even more small and minimalist. It's one executable, barely 500 KB, but it still supports searching (with regexes), printing, a hex editor mode, and settings.

Free editors:

• Vim and Emacs (Windows, macOS, Linux) – Classic Unix editors. Steep learning curve, but brutally efficient. They have settings that can be tuned to make them even faster.
• Large File Editor [6] (Windows) – Opens and edits TB+ files, supports Unicode, uses little memory, has XML-specific features, and includes a binary mode.
• HxD [7] (Windows) – A hex editor, not a text editor; but it's amazingly fast and useful.
• GigaEdit [8] (Windows) – Supports searching, character statistics, and font customization. But it's buggy – with large files, it only allows overwriting characters, not inserting them; it doesn't respect LF as a line terminator, only CRLF; and it's slow.

Builtin programs (no installation required):

• less [9] (macOS, Linux) – The traditional Unix command-line pager tool. Lets you view text files of practically any size. Can be installed on Windows, too.
• Notepad [10] (Windows) – Decent with large files, especially with word wrap turned off.
• MORE [11] (Windows) – This refers to the Windows MORE, not the Unix more. A console program that allows you to view a file, one screen at a time.

Web viewers:

• htmlpen.com [12] – Can open and syntax-highlight TB+ files. Allows editing, except for very large files. Supports searching, regexes, and exporting.
• readfileonline.com [13] – Another HTML5 large file viewer. Supports search.

Paid editors:

• 010 Editor [14] (Windows, macOS, Linux) – Opens giant (as much as 50 GB) files.
• SlickEdit [15] (Windows, macOS, Linux) – Opens large files.
• UltraEdit [16] (Windows, macOS, Linux) – Opens files of more than 6 GB, but the configuration must be changed for this to be practical: Menu » Advanced » Configuration » File Handling » Temporary Files » Open file without temp file...
• EmEditor [17] (Windows) – Handles very large text files nicely (officially up to 248 GB, but as much as 900 GB according to one report).

And finally, have you tried opening the large file with your regular editor? Some editors can actually handle reasonably large files. In particular, Notepad++ [18] (Windows) and Sublime Text [19] (Windows, macOS, Linux) support files in the 2 GB range.

[1] https://code.visualstudio.com/
[2] https://glogg.bonnefon.org/description.html
[3] https://github.com/zarunbal/LogExpert
[4] https://web.archive.org/web/20140908181354fw_/http://swiftgear.com/ltfviewer/features.html
[5] https://www.ghisler.com/lister/
[6] https://www.liquid-technologies.com/large-file-editor
[7] https://mh-nexus.de/en/hxd/
[8] http://www.heliwave.com/GigaEdit.html
[9] https://en.wikipedia.org/wiki/Less_(Unix)
[11] https://ss64.com/nt/more.html
[12] https://htmlpen.com
[14] https://www.sweetscape.com/010editor/
[15] https://www.slickedit.com/products/slickedit
[16] https://www.ultraedit.com/
[17] https://www.emeditor.com/
[19] https://www.sublimetext.com/

(55) VIM, or Emacs... pick your poison, both will handle any file you throw at them. I personally prefer Emacs, but both will beat notepad without so much as a hiccup. - Mike Stone
(25) Emacs has a maximum buffer size, dependent on the underlying architecture (32 or 64 bits). I think that on 32 bit systems you get "maximum buffer size exceeded" error on files larger than 128 MB. - Rafał Dowgird
(74) I just tried Notepad++ with a 561MB log file and it said it was too big - barfoon
I regularly open ~600mb files with gVIM ... - Nippysaurus
(1) I've been asked in the past to edit a couple of plain text files in the multi-GB range, which our users tried to edit with MS Word... well, most of you will know what happened. Just opened it in vim and searched and replaced with the user sitting next to me in a matter of seconds (after that huge file was finally read in of course). - boxofrats
(9) @Rafal Interesting! Looks like on 64bit it is ~1024 petabytes. The reason has to do with the fact that emacs has to track buffer positions (such as the point) - baudtack
(1) I'll second gVim for huge files. I just edited a 950 MB text file with no problems (but it took a while to open and save). When I tried that same file in Notepad2, Windows became concerned with the size of my pagefile, and started resizing it. - Christian Davén
(72) But be careful, vim will only work as long as the files in question have enough line breaks. I once had to edit a ca. 150 MB file without any line breaks, and had to resort to gedit because vim couldnt handle it. - Benno
(5) If you are going to use (g)vim then to improve performance you may want to turn off some features such as syntax highlighting, swapfile and undo. See vim.wikia.com/wiki/Faster_loading_of_large_files, vim.wikia.com/wiki/VimTip611 and vim.org/scripts/script.php?script_id=1506. - Dave Kirby
I wonder if 5GB text files exist .. :-O .. if you don't mind, may I know .. (in practical world) where we are forced to use/edit these bulky text files.. (alternate way would have been to break the file and make a few of it.. usually larger files, of any file-type, make system cry to give performance) - InfantPro'Aravind'
@Rafal: emacs buffer size can be boosted with emacs 23. I don't recall offhand how to do it. - Paul Nathan
(7) I tried all, gVim sucked in that it didnt even tell you it was loading a file - took forever to load the file (only 200k, 5 million lines). SlickEdit opened the entire file in about 3 seconds. Getting the trial license was a PIA tho. Thank you for listing these. - schmoopy
Emacs definitely has a buffer size problem on 32-bit. - Joseph Garvin
(2) I want an editor that mmap()s the file and reads only the parts I am looking at... even gvim seems to load the whole thing into memory first, and even resizing the window freeezes it while it thinks... - Joe Koberg
(1) I had a problem with a single character encoded by 6MB of code. Notepad++ and Netbeans couldn't handle it, but the 010 Editor did it easily! ;D - Jayme Tosi Neto
gVim takes forever to load a 2GB file, and then seeking within it is painfully slow too. It seems to load the entire file into RAM. Maybe the large file plugin would help but the default doesn't seem optimal. - rogerdpack
(3) Wasn't able to get any of these to work on a 1.6GB file, especially gVim or any other Vim for windows that I could find. Had to use filesplitter to break it into 100MB chunks then I used EditPlus to view them. The old Edit.com (DOS) can handle large files (a few hundred MB) but is not available in 64bit windows. - Dustin Davis
(3) 010Editor successfully opened my 3.3GB MySQL database dump. - Zane
Does 010 Editor has XML formatting option? - FIre Panda
The 32-bit version of Vim 7.3 crashes at about 2G if the swap is on and just gives errors about an incorrect line count if swap is off. The 64-bit version however works ok; I'm currently "browsing" a 7.5G file and I can't resize the window with the mouse, seeking is a bit slow, but it works. (81M lines, a log of all memory allocations in an application.) - dash-tom-bang
(1) @docgnome Aw, crap. What about my 1025 petabyte file? Ugh. I guess Emacs is so stuck in the past that it can't edit it. What does it think the date is, 1997? - JavaAndCSharp
(2) Used a combination of Large Text Viewer and HxD. LTV for nice text view and line seeking, and HxD for actual editing + search and replace. - Pixel Elephant
I had a 2+ gigabyte sql script that I tried to open with gVim. It folded like a pair of 2's, then sat down and cried. Then I tried 010Editor, which shined like a CHAMP. Scrap gvim, use 010. - HerrimanCoder
(2) I use Hxd to open large files. It works very will for me and has many other features. I can open and veiw my entire hard drive with it( ~200 GB) in less than a second. Scrolling through files and editing them is very smooth as well. - user2097804
(2) "But be careful, vim will only work as long as the files in question have enough line breaks." @Benno, That's a configuration setting in vim, not a limitation. You can change it this way: :set display+=lastline. It is kind of weird to not have that as the default, though. - Aaron R.
If you have been using LargeTextFileViewer, switch to glogg! LTF is smaller, but glogg seems to work much better. - Martin Ba
(1) If you only want to view the file contents, then I suggest a very nice tool online: readfileonline.com it works on all modern browsers. - oabarca
(1) Large Text File Viewer was able to open a 22GB file without any issues. - Aravind R. Yarram
Is there a way to see the file with the correct line breaks in HxD?? I see weird "..." as separator and not even at the end of the line.thanks - Millemila
(1) Using Large Text File Viewer, I just opened a 30gb file with no issue. Finding a specific keyword took about 5 minutes though. - PaulBGD
gVim would open my 1.6 GB file but I couldn't really do much with it... I understand it's a big task, but I needed to do a find and replace about 150 million instances. gVim would only make it about half way. - Mike
(2) Emeditor opens fast "up to a 248 GB limit (or 2.1 billion lines)" emeditor.com/text-editor-features/large-file-support/… For csv files there is also Delimit ("Open data files up to 2 billion rows and 2 million columns large!") delimitware.com - miodf
When using gVim, use this plugin to automatically disable slow features on large files: github.com/vim-scripts/LargeFile - Denilson Sá Maia
gVim choked on an 8GB file and 'stopped working'. - Paul Zahra
I couldn't get 010 editor to search across a large 20G text file... it just choked and froze until I killed it - Earlz
See LargeFile plugin to Vim. - Hans Ginzel
You can browse, edit and search/replace in text files of almost any size with XML ValidatorBuddy. In addition you get syntax-coloring for XML documents. The editor lets you select the encoding too. - Clemens
LogExpert fails on long lines. If a line is ~8000 chars, it will get cut off and the following line will also be cut off, much shorter. HxD worked fine. - Dan
I used 010 Editor to open a 42.7 Gb file (yes, Wikipedia XML file 27). It worked (and the file has 634,957,038 lines)! I'm using Windows 10 64-bit on a laptop with 16 Gb RAM, and v6.0.3 of 010 Editor. I could also search for a string that was at the end of the file (although 010 took almost 10 minutes to finish the search). - Andrew Jens
(1) As @user2070775 suggested, readfileonline.com is fantastic for a quick view. - bmende
(1) Large Text View Viewer still exists here. - ruffin
(1) I use myself Editpad lite (free for non commercial use) which can edit files larger than 4 GB, even if your PC only has a few GB of RAM. Also, the maximum length of a single line is not limited, which is a problem with many editors claiming to support "unlimited" file sizes. - Eric Bole-Feysot
(1) windows cmd more is ideal for looking and confirming file structure. - clg4
Sublime Text 3 works beautifully for large files; I don't have the rep to add it as answer but just tested with a 1.6G log file. It even shows a loading bar while reading the file, feels fluid to use unlike the others mentioned here. Its free and also has a portable version! - Joseph
(1) I don't know why I can't reply to the OP. The best options in Windows are: EmEditor, Delimit, Editpad Pro and Texpad, all commercial and all can read and write and do many things with files much bigger than memory. Emeditor and Delimit offer the option to see csv files as a spreadsheet. I've being having problems with EmEditor, it keeps reloading the file every few seconds preventing you from working.You can also try SlickEditor and HippoEdit. - skan
010 Editor was perfect for my situation where I had to edit a 8GB mysql dump that failed and I had to resume, I also removed some GBs from log tables, this saved me a lot of time! - Ultranuke
(1) Also total commander's "view file" menu option (F3) is very good at this. What it does, I think, is virtual scrolling; not loading all the content. - myuce
LargeTextFileViewer worked for me for 2 GB of text. Emurasoft's EmEditor claims to open up to 248GB, might be worth a look if other solutions did not work for you. - gaborous
(1) LiquidSoft has a horrible installer which left me with a non-closable dialog. - Rüdiger Schulz
010 Editor worked like a charm :D - GabrielBB
I have tried logexpert and worked well for 1GB text file. The others were not free or didnt worked for me - Rodol Velasco
Liquid Large File Editor does great job and is free (Community Edition): liquid-technologies.com/large-file-editor - Ivan Akcheurov
I tried LogExpert 1.5.5493. It always crash after opening a large txt file for less than 2 mins. The txt file is about 2.5G. - smwikipedia
(1) Mixed bag experience with 010Editor on a 19GB JSON trace file. Takes several seconds to "scan for linefeeds", before showing the file. Crashed doing searches, and seems to look for all matches before moving to the first "closest" match (since took a while to show a match, with a full list of matches at that time). Otherwise still allowed me to examine my super-large file, so not a complete waste. FWIW. - ddevienne
Note that Liquid Studio requires you to sign up even for the "free" Community Edition (e-mail, name etc.) which is a bit dodgy. LogExpert tries to load the entire file into memory, so with a 30 GB file and 12 GB RAM it didn't work for me. - Sina Madani
Liquid Studio text search speed is mere 23 MB/s on my computer! Ridiculous. In comparison EmEditor speed is 312 MB/s and is limited by CPU speed. I have a SSD. Benchmarked on 06/19/2018. - Desik
Update: glogg has search performance in between Liquid Studio and EmEditor. EmEditor is still the fastest. - Desik
Can you edit files with LogExpert? - HelloGoodbye
On 64-bit Windows, the standard Notepad.exe handled opening and searching in a 450 MB csv file perfectly while Sublime Text 3 hung forever and VS Code said it's too large. - Violet Giraffe
Glogg is fantastic. I can easily spot trends and filter down to a log token. Thanks for the recommendation. - Shane
(1) I think the VS Code is a great option. Free and Open Source. Just added to the list. - Faheem
1
[+176] [2009-08-27 20:25:59] Roboprog

Tips and tricks

less

Why are you using editors to just look at a (large) file?

Under *nix or Cygwin [1], just use less [2]. (There is a famous saying – "less is more, more or less" – because "less" replaced the earlier Unix command "more", with the addition that you could scroll back up.) Searching and navigating under less is very similar to Vim, but there is no swap file and little RAM used.

There is a Win32 port of GNU less. See the "less" section of the answer above.

Perl

Perl is good for quick scripts, and its .. (range flip-flop) operator makes for a nice selection mechanism to limit the crud you have to wade through.

For example:

$perl -n -e 'print if ( 1000000 .. 2000000)' humongo.txt | less  This will extract everything from line 1 million to line 2 million, and allow you to sift the output manually in less. Another example: $ perl -n -e 'print if ( /regex one/ .. /regex two/)' humongo.txt | less


This starts printing when the "regular expression one" finds something, and stops when the "regular expression two" find the end of an interesting block. It may find multiple blocks. Sift the output...

logparser

This is another useful tool you can use. To quote the Wikipedia article [3]:

logparser is a flexible command line utility that was initially written by Gabriele Giuseppini, a Microsoft employee, to automate tests for IIS logging. It was intended for use with the Windows operating system, and was included with the IIS 6.0 Resource Kit Tools. The default behavior of logparser works like a "data processing pipeline", by taking an SQL expression on the command line, and outputting the lines containing matches for the SQL expression.

Microsoft describes Logparser as a powerful, versatile tool that provides universal query access to text-based data such as log files, XML files and CSV files, as well as key data sources on the Windows operating system such as the Event Log, the Registry, the file system, and Active Directory. The results of the input query can be custom-formatted in text based output, or they can be persisted to more specialty targets like SQL, SYSLOG, or a chart.

Example usage:

C:\>logparser.exe -i:textline -o:tsv "select Index, Text from 'c:\path\to\file.log' where line > 1000 and line < 2000"
C:\>logparser.exe -i:textline -o:tsv "select Index, Text from 'c:\path\to\file.log' where line like '%pattern%'"


The relativity of sizes

100 MB isn't too big. 3 GB is getting kind of big. I used to work at a print & mail facility that created about 2% of U.S. first class mail. One of the systems for which I was the tech lead accounted for about 15+% of the pieces of mail. We had some big files to debug here and there.

And more...

Feel free to add more tools and information here. This answer is community wiki for a reason! We all need more advice on dealing with large amounts of data...

[1] http://en.wikipedia.org/wiki/Cygwin
[2] http://linux.die.net/man/1/less
[3] https://en.wikipedia.org/wiki/Logparser

(7) +1, I recently had some really huge xml files (+1 gigabyte) that I needed to look at. I'm on windows and both vim, emacs, notepad++ and several other editors completely choked on the file to the point where my system almost became unusable when trying to open the file. After a while I realized how unnecessary it was to actually attempt to open the file in an -editor- when I just needed to -view- it. Using cygwin (and some clever grep/less/sed-magic) I easily found the part I was interested in and could read it without any hassle. - wasatz
cygwin's less works for viewing a file > 2GB sweet - rogerdpack
(8) you don't need cygwin for less, you can also use it under windows: gnuwin32.sourceforge.net/packages/less.htm - ChristophK
I couldn't figure out how to edit files with less but after getting the taste I used vim to edit a bulky JSON object which made the other editors (Brackets, Textmate <- note I'm running on OS X) throw up. - nuala
(2) This XML editor here has also a large file viewer component and does provide syntax coloring also for huge files. The files are not loaded completely into memory so a multi-GB document shouldn't be a problem. In addition this tool can also validate those big XML documents ... In my opinion one of the best approaches to work with huge XML data. - lichtfusion
I'm going to "disavow" the logparser.exe thing that "the community" added to my answer, as I would prefer to stick to POSIX tools, rather than some Microsoft-only thing, which also apparently takes a longer command line to invoke than my other examples. - Roboprog
I tried the less packaged in mingw shell, and it complains about memory on a 400MB file. No good. - goat
less is great as long as the lines aren't too long. I'm in this thread because less (linux) is choking badly on debug log files that contain BIG serialized XML and I need something faster. - Andy Brown
(7) OK so I just fixed my own issue. less with word wrap is slow. less -S without word wrap is lightning fast even on large lines. I'm happy again! - Andy Brown
(6) Great answer. I want to note that if you have Git for Windows installed, you probably have Git bash as well, which includes less. - transistor1
Well, this thing has become a bit of an "Alan Smithe Production". My original answer said nothing about IIS logparser, since I avoid MS-Windows whenever possible. Happy Wiki-ing, I guess :-) - Roboprog
The locked questions are some of the best questions on StackOverflow. - kd4ttc
2