>>>>> Juli <julianeleusch...@web.de>
>>>>> on Sat, 12 Sep 2015 02:32:39 -0700 writes:
> Hi Jim, thank you for your help. :)
> My point is, that there are outlier and I don´t really
> know how to deal with that.
> I need the dataframe for a regression and read often that
> only a few outlier can change your results very much. In
> addition, regression diacnostics didn´t indcate me the
> best results. Yes, and I know its not the core of
> statistics to work in a way you get results you would
> like to have ;).
> So what is your suggestion?
Use robust regression, e.g.
MASS::rlm() {part of every R installation},
or a somewhat better and more sophisticated version.
lmrob() from package 'robustbase' {yes, shameless promotion}.
Further:
1) Removing outliers is not at all the best way to deal with such
problems (intuitively, because it is a *dis*continuous method).
Rather they should be downweighted (continuously, as it
happens with methods used in rlm() or lmrob() see above)
2) Removing outliers in *multivariate* setting, if you want to do
it in spite of 1) by using univariate treatment {each column
separately as you do here} is often completely insufficient. E.g.
the bivariate outlier in
xy <- cbind(x= c(2,1:9), y=c(8,1:9)); plot(xy)
cannot be found by looking at 'x' and 'y' separately.
3) If, in spite of 1) and 2) you are considering univariate
treatment, using mean() and sd() for detecting univariate outliers
has been proven to be insufficient more than 50 years ago (*1), and
if one looks closer into the literature (say "L_1") even
considerably longer ago.
Using median() and mad() instead, is one possibility (*2) of
what you should do. Hampel's rule (*3)
proposes declaring outliers for the observations outside
the interval median(x) +/- 3.5*mad(x)
*1 Tukey, J. W. (1960) A survey of sampling from contaminated distributions.
In Contributions to Probability and Statistics,
eds I. Olkin, S. Ghurye, W. Hoeffding, W. Madow and H. Mann,
pp. 448–485. Stanford: Stanford University Press.
*2 Another (less robust, but still infinitely better than mean/sd) approach
uses median() and IQR() which is
basically/approximately what boxplots do to identify outliers.
*3 Frank R. Hampel (1985)
The Breakdown Points of the Mean Combined With Some Rejection Rules,
Technometrics, 27:2, 95-107
[ http://dx.doi.org/10.1080/00401706.1985.10488027 ]
See also section
"1.4b. How Well Are Objective and Subjective Metbods for
the ReJection of Outliers Doing in the Context of Robust
Estimation?",
page 62 ff od
of
Frank R. Hampel, Elvezio M. Ronchetti, Peter J. Rousseeuw and Werner A.
Stahel
(1986) Robust Statistics: The Approach Based on Influence Functions.
John Wiley & Sons, Inc.
______________________________________________
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
