share
Super UserIs there a text editor for very big files?
[+91] [24] Lea Cohen
[2009-09-03 05:38:09]
[ windows software-rec text-editing large-files ]
[ http://superuser.com/questions/34749/is-there-a-text-editor-for-very-big-files ] [DELETED]

I have a very big text file (10 GB). Both Notepad [1] and Notepad++ [2] tell me that the file is too big for them to open. What editor can handle such a large file? It's a log file.

(1) Most text editors put the entire files into RAM, so unless you have over 10GB of RAM, you're going to have a hard time finding a good text editor that does this. That said, I'm positive one exists. - Sasha Chedygov
(21) That's one heck of a text file - Col
(6) @NoCanDo - I would guess a log file of some kind, or a data file for processing. - ldigas
Definitely not kidding... And yes, it's a log file. - Lea Cohen
We had multi-GB data files when I worked at a billing statement printer. The files would come in "print format", designed to be printed on continuous feed paper. We had to parse and put the data on the web and create nice statements. Credit unions were the worst. The files plaintext, mostly spaces, highly compressible. - Nathan DeWitt
(1) Is there any unuseful information in your file that can be parsed out of it to make your file smaller? - ralford
Possible duplicate: superuser.com/questions/4044/… - Hello71
This should be moved to Software Recommandations. - Cristian Ciupitu
[+43] [2009-09-03 06:11:17] Natrium [ACCEPTED]

If you only need to read the file, I can suggest Large Text File Viewer [1].

[1] http://www.swiftgear.com/ltfviewer/features.html

(2) Sorry -1. This wasn't a very good editor. It froze up when opening a 7MB file. - DJTripleThreat
(19) +1. Worked for a 90GB xml file. - timbooo
@DJTripleThreat "This program was designed for viewing large (>1GB) text files." Source: swiftgear.com/ltfviewer/features.html - Connor W
(7) oh yeah? Notepad was designed to play 5.4TB HD videos as well. I can say that too. Doesn't mean it will do that though. - DJTripleThreat
+1 worked great for some multi GB files for me. - mindless.panda
Large Text File Viewer has bugs. It freezes if you click 10^4 page down when it didn't load these lines yet. Also it does not load file completely. You have to browse in file to load. - Berk Demirkır
I looked up the same thing a while ago, when working with a 35-million line file. It worked pretty well. - Eli
What's the analogue of Large Text File Viewer for Mac OS X and Linux? - Problemaniac
(1) I won't downvote this answer because Large Text File Viewer does open very large files, BUT this is definitely not the best solution. Based on this answer I used Large Text File Viewer for viewing large log files (excess of 2 Gb) and although it opened them, searching was very slow. I then tried V - The File Viewer mentioned in an answer lower down and that was extremely fast... searches for text in a 2Gb file in a matter of seconds. V definitely gets my vote. - BruceHill
Works perfectly! - rogcg
1
[+39] [2009-09-03 05:43:17] nik

Stack Overflow ref Text editor to open big (giant, huge, large) text files [1].

Vim [2] is a personal favorite.

  • However, if you are working with timestamped entries in log file, it's easier to filter for a date range and look at smaller files.
  • With other kinds of larger files, you could cut out specific line-number ranges (tools like sed [3] and grep [4]) to smaller files and analyze individually
  • If it's some specific translation or searching to do, again sed, grep, etc. are good tools.
[1] http://stackoverflow.com/questions/159521/text-editor-to-open-big-giant-huge-large-text-files
[2] http://en.wikipedia.org/wiki/Vim_%28text_editor%29
[3] http://en.wikipedia.org/wiki/Sed
[4] http://en.wikipedia.org/wiki/Grep

VIM is better than a regular text editor for big files, but it takes a while to open them anyway. I remember it taking like 30sec for a 200mb file (and looking like hanged during). - Gnoupi
@Gnoupi, Yes I understand. The SO link had all the options and the rest of my answer described how I'd do it. - nik
@Gnoupi - VIM is an excellent editor, but it is a source code editor, and not specifically made for opening huge files. It can open very large files, but that's just because it is well designed. But for opening files of this size, I'd still rather go for something made for that purpose. - ldigas
It's unusual to see someone recommend sed on Windows. If anyone doesn't know, Unix-like tools for Windows are available several ways. CygWin and MSYS provide special command prompts. MinGW (which underlies MSYS) just provides the tools without a special command prompt, but doesn't necessarily provide everything you'd want. GNUWin32 just provides the specific tools you ask for. - Steve314
2
[+18] [2009-09-03 05:45:09] The How-To Geek

UltraEdit [1] is your best bet - I've opened files that were a few gigabytes in size with it. They also have some tips on opening large files [2].

It's not free, however.

Otherwise, you can use the Unix split [3] utility from the command line - should be included in Cygwin or the Linux command line tools for Windows [4]. You'll have much better luck dealing with the files if you've split them up into smaller chunks.

[1] http://www.ultraedit.com
[2] http://www.ultraedit.com/support/tutorials_power_tips/ultraedit/large_file_handling.html
[3] http://en.wikipedia.org/wiki/Split_%28Unix%29
[4] http://unxutils.sourceforge.net/

I tried UltraEdit, but it wasn't able to open the file - Lea Cohen
You might have to set the option to work without temporary files: Options - File Handling - Temporary Files gives you the relevant choices (check the second checkbox). - Tim Pietzcker
(3) I have used Ultraedit to open files that were nearly a hundred gigabytes large with no problems whatsoever. It works well. - BBlake
Thanks. I had a big file I needed to work with and already have an ultraedit license.. Didnt think to try this after NotePad++, PSPad, vim failed me. - Yablargo
3
[+12] [2009-09-03 06:17:37] harrymc

The best one I tried was V - The File Viewer [1], when I needed to analyse a text file of more than 1 TB. It was by far the fastest. It's not free, though ($20).

[1] http://www.fileviewer.com/

If I could upvote this 10 times I would! I first tried Large Text File Viewer (the accepted solution to this question) and it was disastrous... searching was SLOW. V - The File Viewer on the other hand is fantastic!! This answer definitely need more upvotes!! :) - BruceHill
4
[+11] [2009-09-06 14:15:00] ldigas

EmEditor [1] is a very nice editor all around, but also build with opening very large files in mind.

From their site, and I quote: " EmEditor Text Editor is a lightweight, yet extendable, easy-to-use text editor for Windows. EmEditor is very customizable, and it supports Unicode and powerful macros. EmEditor now supports very large files - up to 248 GB or 2.1 billion lines! EmEditor is certified for Windows Vista! The x64 Edition is also available! The portable version is also available for USB drive setup! "

[1] http://www.emeditor.com

This should be the answer. EmEditor has a large file controller which loads big files in background. I opened 10GB log file with this program within 60 seconds. Perfect! - Berk Demirkır
(1) This was the only program that could reliably open my 32GB CSV files. - Robert H
This is great! I couldn't open just a 3.~ GB SQL dump with any of these other programs mentioned, but this one opened it immediately (probably because it didn't wait for it to load it all into memory first). Thanks~ - James
5
[+10] [2009-09-07 12:33:53] SztupY

Total Commander [1]'s lister can open very large files for reading. For editing you might also use some kind of Hex editor (like the free HxD [2]), they can usually open and edit large files.

[1] http://en.wikipedia.org/wiki/Total_Commander
[2] http://en.wikipedia.org/wiki/HxD

+1 for Lister. Immediatly opens any file, whatever it's size. - Juan Calero
I used Lister successfully to search within a 10 Gig XML file. - Marcel
6
[+7] [2009-09-03 06:48:43] Diomidis Spinellis

The Unix sed (stream editor) command can handle files of an arbitrary large (even infinite) file size, because it applies a script of editing commands to the stream of lines that pass through it. You can easily get it for Windows through the Cygwin [1] distribution.

For instance, if you want to delete all files containing the string foo you write in the command prompt:

sed /foo/d <inputfile >outputfile

To substitute foo with bar write:

sed s/foo/bar/ <inputfile >outputfile

More complicated constructions are possible. Someone has even written a script to have sed solve the Towers of Hanoi puzzle [2].

[1] http://www.cygwin.com/
[2] http://sed.sourceforge.net/grabbag/tutorials/hanoi.htm

You can also edit the file in place using sed -i. To save a backup with the .bak extension use sed -ibak - Nathan Fellman
The Cygwin version depends on cygwin libraries, so I would rather recommend the statically linked version from GnuWin (gnuwin32.sourceforge.net/packages/minised.htm) or from the UnixUtils package (unxutils.sourceforge.net) - ccpizza
MSysGit also contain sed, just as a side note. - Cawas
7
[+7] [2009-09-06 12:56:28] Kev

I've had luck with TextPad [1] on files 2 GB in size with less ram than that, it takes a while to load up, but it works.

[1] http://www.textpad.com/

(3) It feature list implies that it does load the file into memory, even if it is virtual memory: "It can handle file sizes up to the largest contiguous chunk of 32-bit virtual memory." - dlamblin
textpad works brilliantly with large files. Plus - It's a great editor. - seanyboy
+1 for textpad [ ] - Critical Skill
+1 This works!! - DJTripleThreat
(3) It cannot handle files with larger size than avaible memory. - Berk Demirkır
8
[+5] [2010-09-17 11:37:35] Ali Adams

I have developed a Unicode editor called GigaEdit [1] specifically to solve this issue of opening unlimited size files instantly by blocks of 1 MB with scrollbar to navigate effortlessly between blocks and a multi-threaded search feature that displays a result as they become available and allows you to go to matches instantly F3 or Shift+F3 even across blocks transparently.

WARNING: There is no UNDO after saving so make sure you know what you are doing.

[1] http://www.heliwave.com/GigaEdit.html

9
[+3] [2009-09-07 18:02:30] Reuben

For looking through large files (but only for read-only purposes), I use a Windows port of the UNIX less utility. It generally loads up any large file very quickly, and is pretty quick at searching for text patterns (once you get past its learning curve).

(Oh, and the other catch is that you have to run it within a command prompt window. It might not work if you were looking for something a bit more visually friendly.)

I don't recall where I found my current copy of it, but a quick web search shows a few possible places to download one, e.g.:


10
[+3] [2009-12-10 15:10:04] Pauli L

I would recommend VEDIT [1]. It is about the only editor that is fast enough to realistically edit huge files. Its engine has been written in highly optimized assembly language.

No matter how big the file, VEDIT loads it in a fraction of a second (typically about 0.1 second). Most editors load the whole file in memory, which limits the file size and slows down opening the file. VEDIT only loads a part of the file at a time, and does not create a huge temporary file when opening the file (temporary files are created only when needed and as big as needed). In addition, VEDIT uses very little of computer memory. When I had a 300 MB file open, Task Manager reported 4.5 MB memory usage for VEDIT.

If you are just viewing the file and not editing, opening the file in read-only mode (with Fast Browse Mode enabled) makes VEDIT even faster. For example, you can scroll to any location in the file instantly, without any delay.

The standard (32-bit) version of VEDIT can handle files up to 2 GB. VEDIT Pro64 can handle files of any size.

To edit files larger than 2 GB on a 32-bit editor, you can split the file into smaller parts. You could use a separate tool for that, but VEDIT has built-in function to split and combine such files. For huge log files, splitting might be good idea anyway. You can then open all the parts for editing simultaneously and you can perform global searches among all of them, etc.

[1] http://www.vedit.com/

11
[+3] [2010-10-31 08:59:24] twinsen

I had trouble loading a 5.5 GB binary file in a whole range of editors. Two I found that worked, and opened the file instantly:

HxD is my preferred one, since it is free, supports viewing and editing and is a stand alone. My large files are generally binary, so a hex editor is fine. But for viewing large text files, Altap Salamander has a better viewer.

[1] http://en.wikipedia.org/wiki/Altap_Salamander
[2] http://en.wikipedia.org/wiki/HxD

12
[+3] [2010-10-31 09:34:37] TusharG

I'll suggest gVim [1] which can open such a huge file.

[1] http://www.vim.org/download.php

gVim is really good if your file fits in memory, still quite good if it fits in swap memory, but not so great if you have a really huge file and small memory, such as viewing Wikipedia XML dump files on a netbook. In that case an editor that can load just one big chunk of your file into memory at a time is better. Not sure which one is best though ... - hippietrail
13
[+2] [2009-09-06 11:36:34] bbryyz

You can also try Bare Tail

http://www.baremetalsoft.com/baretail


That seems to be just a GUI version of GNU CoreUtils' tail command. gnuwin32.sourceforge.net/packages/coreutils.htm - dlamblin
14
[+2] [2009-10-20 03:59:44] Dracoder

I edited a file of 7 GB with PilotEdit [1]. But it may take you an hour to open/save a 10 GB file. After you open the file, you can edit it quickly.

[1] http://www.pilotedit.com/

15
[+1] [2010-05-06 23:59:53] jfmessier

Under Windows, I use EditPad Pro [1] from Just Great Software [2] (there is a lite version too).

EditPad is the best editor I found. TextPad is also very good. Both now support UTF-8 file encoding.

[1] http://www.editpadpro.com
[2] http://www.just-great-software.com/

16
[0] [2009-09-06 21:15:29] nirzar

Try TextPad [1]. It's the best.

  • Huge files can be edited, up to the limits of 32-bit virtual memory. See Specifications for the actual limits.

And then per the [Specifications]

  • It can handle file sizes up to the largest contiguous chunk of 32-bit virtual memory.
  • The number of concurrent edits is limited by available Windows resources.
[1] http://www.textpad.com/products/textpad/features.html

Link and or explanation of how it supports big files is needed. - dlamblin
17
[0] [2009-09-06 21:27:41] dlamblin

I used to like pepper [1] for this purpose. It's a little hard to find the abandoned windows port, and harder still to get it registered, though I managed about 2 years ago to contact Hekkelman about it. One of the features is that it does not load the whole file into memory.

[1] http://eriksrailnews.com/pepper/

That is a standard feature in this kind of editors. Opening from size till size, or fron line# till line#, right ? - ldigas
18
[0] [2010-03-31 02:56:15] ggonsalv

If you only want to view a section of a file, then you can use the following command in a DOS window.

type reallylargefile.txt | more

This will display file at a screen full at a time.


(1) or just more reallylargefile.txt - newenglander
19
[0] [2010-07-30 17:58:02] EJP

EmEditor [1] is basically built for huge files. It's never balked at anything I'm thrown at it!

[1] http://www.emeditor.com/

(1) If you want to advertise your product, I recommend you do so by contacting the team. You can buy advertisements to put on side, or even better, sponsor a tag (like text-editor). The point is, if you really want to advertise, that's the best way. Because if you keep doing it by poking all related questions and adding link to your product, you will be seen as spammer. And this rarely ends good. That said, have a good day, enjoy your stay around here, provide useful answers. - Gnoupi
20
[0] [2011-05-02 16:18:28] Mahesh Lavannis

Use FileSplitter [1] to break the file into smaller files, and then use your favorite text editor!

Description: The File Splitter is a free Windows program that will split any file into pieces of any size and later combine the pieces to form the original file.

[1] http://www.dekabyte.com/filesplitter/

21
[0] [2012-01-01 00:21:39] Harry Johnston

Assuming you only need to read the file, Microsoft's Windows 2003 resource kit [1] includes list.exe.

[1] http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=17657

22
[0] [2012-09-06 20:52:39] Nick Gotch

One approach is to break the giant file into smaller files. You can do that with Chainsaw [1]. It takes a file of any size and outputs a bunch of smaller files from it.

[1] http://www.softpedia.com/get/System/File-Management/GS-Chainsaw.shtml

23
[0] [2013-05-29 10:08:25] Kevdog777

One very good editor for large files is Programmer's File Editor [1]. I use it to ope files bigger than 15GB.

[1] http://download.cnet.com/Programmer-s-File-Editor-32-bit/3000-2352_4-10002157.html

24