upData {Hmisc} | R Documentation |
cleanup.import
will correct errors and shrink
the size of data frames created by the S-Plus File ... Import
dialog or by other methods such as scan
and read.table
. By
default, double precision numeric variables are changed to single
precision (S-Plus only) or to integer when they contain no fractional
components.
Infinite values or values greater than 1e20 in absolute value are set
to NA. This solves problems of importing Excel spreadsheets that
contain occasional character values for numeric columns, as S-Plus
converts these to Inf
without warning. There is also an option to
convert variable names to lower case and to add labels to variables.
The latter can be made easier by importing a CNTLOUT dataset created
by SAS PROC FORMAT and using the sasdict
option as shown in the
example below. cleanup.import
can also transform character or
factor variables to dates.
upData
is a function facilitating the updating of a data frame
without attaching it in search position one. New variables can be
added, old variables can be modified, variables can be removed or renamed, and
"labels"
and "units"
attributes can be provided. Various checks
are made for errors and inconsistencies, with warnings issued to help
the user. Levels of factor variables
can be replaced, especially using the list
notation of the standard
merge.levels
function. Unless force.single
is set to FALSE
,
upData
also converts double precision vectors to single precision
(if not under R), or to integer if no fractional values are present in
a vector.
Both cleanup.import
and upData
will fix a problem with
data frames created under S-Plus before version 5 that are used in S-Plus 5 or
later. The problem was caused by use of the label
function
to set a variable's class to "labelled"
. These classes are
removed as the S version 4 language does not support multiple
inheritance. Failure to run data frames through one of the two
functions when these conditions apply will result in simple numeric
variables being set to factor
in some cases. Extraneous "AsIs"
classes are also removed.
For S-Plus, a function exportDataStripped
is provided that allows
exporting of data to other systems by removing attributes label,
imputed, format, units
, and comment
. It calls
exportData
after stripping these attributes. Otherwise
exportData
will fail.
The dataframeReduce
function removes variables from a data frame
that are problematic for certain analyses. Variables can be removed
because the fraction of missing values exceeds a threshold, because they
are character or categorical variables having too many levels, or
because they are binary and have too small a prevalence in one of the
two values. Categorical variables can also have their levels combined
when a level is of low prevalence.
cleanup.import(obj, labels, lowernames=FALSE, force.single=TRUE, force.numeric=TRUE, rmnames=TRUE, big=1e20, sasdict, pr, datevars=NULL, datetimevars=NULL, dateformat='%F', fixdates=c('none','year'), charfactor=FALSE) upData(object, ..., rename, drop, labels, units, levels, force.single=TRUE, lowernames=FALSE, moveUnits=FALSE, charfactor=FALSE) exportDataStripped(data, ...) dataframeReduce(data, fracmiss=1, maxlevels=NULL, minprev=0, pr=TRUE)
obj |
a data frame or list |
object |
a data frame or list |
data |
a data frame |
force.single |
By default, double precision variables are converted to single precision
(in S-Plus only) unless |
force.numeric |
Sometimes importing will cause a numeric variable to be
changed to a factor vector. By default, |
rmnames |
set to ‘F’ to not have ‘cleanup.import’ remove ‘names’ or ‘.Names’ attributes from variables |
labels |
a character vector the same length as the number of variables in
|
lowernames |
set this to |
big |
a value such that values larger than this in absolute value are set to
missing by |
sasdict |
the name of a data frame containing a raw imported SAS PROC CONTENTS CNTLOUT= dataset. This is used to define variable names and to add attributes to the new data frame specifying the original SAS dataset name and label. |
pr |
set to |
datevars |
character vector of names (after |
datetimevars |
character vector of names (after |
dateformat |
for |
fixdates |
for any of the variables listed in |
charfactor |
set to |
... |
for |
rename |
list or named vector specifying old and new names for variables. Variables are
renamed before any other operations are done. For example, to rename
variables |
drop |
a vector of variable names to remove from the data frame |
units |
a named vector or list defining |
levels |
a named list defining |
moveUnits |
set to |
fracmiss |
the maximum permissable proportion of |
maxlevels |
the maximum number of levels of a character or categorical or factor variable before the variable is dropped |
minprev |
the minimum proportion of non-missing observations in a category for a binary variable to be retained, and the minimum relative frequency of a category before it will be combined with other small categories |
a new data frame
Frank Harrell, Vanderbilt University
sas.get
, data.frame
, describe
,
label
, read.csv
, strptime
,
POSIXct
,Date
## Not run: dat <- read.table('myfile.asc') dat <- cleanup.import(dat) ## End(Not run) dat <- data.frame(a=1:3, d=c('01/02/2004',' 1/3/04','')) cleanup.import(dat, datevars='d', dateformat='%m/%d/%y', fixdates='year') dat <- data.frame(a=(1:3)/7, y=c('a','b1','b2'), z=1:3) dat2 <- upData(dat, x=x^2, x=x-5, m=x/10, rename=c(a='x'), drop='z', labels=c(x='X', y='test'), levels=list(y=list(a='a',b=c('b1','b2')))) dat2 describe(dat2) dat <- dat2 # copy to original name and delete dat2 if OK rm(dat2) # Remove hard to analyze variables from a redundancy analysis of all # variables in the data frame d <- dataframeReduce(dat, fracmiss=.1, minprev=.05, maxlevels=5) # Could run redun(~., data=d) at this point or include dataframeReduce # arguments in the call to redun # If you import a SAS dataset created by PROC CONTENTS CNTLOUT=x.datadict, # the LABELs from this dataset can be added to the data. Let's also # convert names to lower case for the main data file ## Not run: mydata2 <- cleanup.import(mydata2, lowernames=TRUE, sasdict=datadict) ## End(Not run)