Tuesday, March 17, 2015

Solaris: how to edit/vi huge files (GBs)


Every now and then you end up with task to read or edit file that is huge. By huge I mean the size of the file at least 1GB, more likely several GBs.
You usually notice huge file by the following message when you try to open it using vi editor:

Not enough space in /var/tmp

The default directory /var/tmp for the vi editing buffer needs space equal to roughly twice the size of the file with which you are working, because vi uses extra lines for buffer manipulation.
This is how you choose another directory for vi's editing buffer:

$ cd /where_your_file_is_located
$ vi
:set directory=/tmp
:e filename

You can replace /tmp with any other directory containing enough space as a editing buffer. The filename is the name of your large file.
In case you just need to read file and do not plan to edit it, you can use one of the commands that use fewer resources like less, head or tail.
Other option would be to split the file either size-wise or line-wise.

$ split -l 10000 xyz.txt

 
Each of new files will have 10000 lines. Or you can try


$ split -b 250m xyz.txt xyz.txt.split

which will create 250MB chunks out of the original file.

If you prefer to use vim over vi, you might try to press Ctrl-C when loading your file. Vim tries to read in the whole file initially to do things like syntax highlighting, number of lines in file and so on. Ctrl-C will cancel this and will only load what's needed to display on your screen.

No comments:

Post a Comment