Faster Recoding
The typical Stata job involves considerable recoding of variables. There
are many ways to recode variables and how this is done can have a
substantial effect on runtime. Consider the conversion of the FIPS state
code to the IRS state code. Stata includes the -recode- statement for just
this task:
<![CDATA[
recode fips = ( 1 = 1) ( 2 = 2) ( 4 = 3) ( 5 = 4) ( 6 = 5)
( 8 = 6) ( 9 = 7) (10 = 8) (11 = 9) (12 = 10) (13 = 11)
(15 = 12) (16 = 13) (17 = 14) (18 = 15) (19 = 16) (20 = 17)
(21 = 18) (22 = 19) (23 = 20) (24 = 21) (25 = 22) (26 = 23)
(27 = 24) (28 = 25) (29 = 26) (30 = 27) (31 = 28) (32 = 29)
(33 = 30) (34 = 31) (35 = 32) (36 = 33) (37 = 34) (38 = 35)
(39 = 36) (40 = 37) (41 = 38) (42 = 39) (44 = 40) (45 = 41)
(46 = 42) (47 = 43) (48 = 44) (49 = 45) (50 = 46) (51 = 47)
(53 = 48) (54 = 49) (55 = 50) (56 = 51),generate(irs);
]]>
That statement takes about 11 seconds per million observations. Given that
the source values are all small integers, it is quite possible to substitute an
array assignment:
<![CDATA[
matrix define fips2irs = (1,2,.,3,4,5,.,6,7,8,9,10,11,.,
12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,
33,34,35,36,37,38,39,.,40,41,42,43,44,45,46,47,.,48,49,50,51);
generate irs = fips2irs[1,fips];
]]>
which takes only .18 seconds per million observations, or a factor of 60
speedup. A series of 51 replace...if statements
<![CDATA[
...
generate irs = 1 if fips==1
replace irs = 2 if fips==2
replace irs = 3 if fips==4
...
]]>
would fall in between those times, at 1.6 seconds/million values but is
maximally flexible.
Still another way to recode would use a sort and merge with a translation
dataset, which is .65 seconds in my test dataset, but would be highly
variable.
In some cases the -group- option on the -egen- statement will be practical and is
also quite fast. It doesn't let you choose the new values (they are consecutive
integers) but it may be used to rapidly and easily compress long strings into single
bytes. See also -encode- or -multencode- (SSC), In my tests -encode- is nearly
instantaneous - less than a second for converting a million strings with a thousand
possible values.
Note that if you want regular quantiles such as deciles or percentiles, there are
fast one-line commands.
At least one of the techniques mentioned above should suit any
recoding task, and even the most flexible methods are likely to be an
order of magnitude faster than recode.