ConservingMemory

A program that uses more memory than is physically available will swap to disk. Swapping takes nearly forever, and must be avoided. In some cases it will cause all programs running on that computer to run very slowly and the computer to appear dead.

You can tell how much memory your Stata job is taking with the stata -memory- command or you can estimate it from the product of the number of observations times the size of each plus a little extra. Floats are 4 bytes, doubles 8, strings as many bytes as characters, etc. If you are using any medium or larger dataset you should keep in mind the approximate size so that you never start your job on a machine with insufficient available memory. The "showload" Linux command will show the amount of available memory on each machine in the NBER cluster. "top" will show the memory your program and others are using. Try to leave some (a lot, actually) for other users.

There are many ways to conserve memory in Stata.

• You can drop variables or observations in the -use- statement or anywhere else.
• You can specify a storage type in the -generate- statement. For example: generate stateid byte = ... generate income float = ... Since the default storage type is double, specifying byte saves 7/8s of the space.
• -compress- will examine each variable and copy it to the smallest storage type that will not lose any precision. So it won't convert most non-integer doubles to floats, even if the economic significance is only at the float level. You can convert one or more doubles to a more compact representation with: recast float x y z,force recast int x y z,force recaset byte x y z,force The -force- option is required if some precision may be lost in the conversion. Some documentation claims that the -compress- command defaults to -coalesce- which will save storage space for repetitive strings, but we do not have much experience with this and other documents say no.

  Compress is the absolute minimum any medium or larger Stata program should do to save memory. Unless you have SSNs stored as doubles or otherwise have data that requires double precision, these two commands may save quite a bit space with no additional bookkeeping of variables:

  recast float *,force compress
• -egen,group- will recode strings into integers. Each unique string is give an integer label. So the 50 state names could be recoded from str14 to byte, a saving of 13 bytes. Also see Recoding in Stata/Efficient.
• Perhaps you can do without some observations. See Sample Selection.
• Perhaps you can do men and women in separate regressions?
• Fixed effects should not require additional memory. The Stata supplied SSC routines for fixed effects will mostly copy all the data to doubles before running the regression. This will at least triple the memory consumed and can easily make a doable problem infeasible. See Fixed Effects, Multiple Fixed Effects and TSLS for thriftier routines. Even if none of those appeal, there are still Factor Variables which save space (if not CPU).

There is some information about memory usage in the Stata manual and in this post by Bill Gould.

Note that R has no float datatypes with less than 8 bytes, so a very large problem might be possible in Stata and not in R.