My students are working with several SPSS dataset provided by the
European Social Survey. If you register your name, you can download it
too. This is the 2004 data, for example:
http://ess.nsd.uib.no/ess/round2/
I cannot give you the European Survey dataset, but you can download it
for free if you like, and then you could run these commands to
re-produce this weird pattern described below.
library(foreign)
d2 <- read.spss("ESS3e03_2.por")
warnings()
str(d2$HAPPY)
d2 <- as.data.frame(d2)
str(d2$HAPPY)
d2 <- read.spss("ESS3e03_2.por",to.data.frame=T)
warnings()
str(d2$HAPPY)
Here's my info for this example:
sessionInfo()
R version 2.10.0 (2009-10-26)
x86_64-pc-linux-gnu
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=C LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] foreign_0.8-38
The weirdness that follows is the difference between
d2 <- read.spss( ... , to.data.frame=T)
and
d2 <- read.spss ()
d2 <- as.data.frame(d2)
The former causes all data to become <NA> but the latter seems mostly OK.
library(foreign)
d2 <- read.spss("ESS3e03_2.por")
warnings()
There were 12 warnings (use warnings() to see them)
Warning messages:
1: In `levels<-`(`*tmp*`, value = c("CENTRUMP", "", "FIDESZ", ... :
duplicated levels will not be allowed in factors anymore
2: In `levels<-`(`*tmp*`, value = c("CENTRUMP", "", "FIDESZ", ... :
duplicated levels will not be allowed in factors anymore
3: In `levels<-`(`*tmp*`, value = c("Refusal", "Don't know", ... :
duplicated levels will not be allowed in factors anymore
4: In `levels<-`(`*tmp*`, value = c("No second language mentioned", ... :
duplicated levels will not be allowed in factors anymore
5: In `levels<-`(`*tmp*`, value = c("Sans dipl", "Non dipl", ... :
duplicated levels will not be allowed in factors anymore
6: In `levels<-`(`*tmp*`, value = c("\"Ej avslutad
folkskola/grundskola\"", ... :
duplicated levels will not be allowed in factors anymore
7: In `levels<-`(`*tmp*`, value = c("Armed forces", "Legislators,
senior officials and managers", ... :
duplicated levels will not be allowed in factors anymore
8: In `levels<-`(`*tmp*`, value = c("Armed forces", "Legislators,
senior officials and managers", ... :
duplicated levels will not be allowed in factors anymore
9: In `levels<-`(`*tmp*`, value = c("K", "K", "Frederiksborg Amt", ... :
duplicated levels will not be allowed in factors anymore
10: In `levels<-`(`*tmp*`, value = c("P", "L", "Kesk-Eesti", ... :
duplicated levels will not be allowed in factors anymore
11: In `levels<-`(`*tmp*`, value = c("Galicia", "Principado de Asturias", ... :
duplicated levels will not be allowed in factors anymore
12: In `levels<-`(`*tmp*`, value = c("Stockholm", "", "Sydsverige", ... :
duplicated levels will not be allowed in factors anymore
str(d2$HAPPY)
Factor w/ 14 levels "Extremely unhappy",..: 9 7 9 11 9 6 9 4 13 8 ...
d2 <- as.data.frame(d2)
str(d2$HAPPY)
Factor w/ 14 levels "Extremely unhappy",..: 9 7 9 11 9 6 9 4 13 8 ...
That appears valid. On my first effort, I had tried to get the data
frame in a single shot with read.spss
d2 <- read.spss("ESS3e03_2.por",to.data.frame=T)
There were 15 warnings (use warnings() to see them)
warnings()
Warning messages:
1: In xi >= z[1L] | xi <= z[2L] | xi[xi == z[3L]] :
longer object length is not a multiple of shorter object length
2: In xi >= z[1L] | xi <= z[2L] | xi[xi == z[3L]] :
longer object length is not a multiple of shorter object length
3: In xi >= z[1L] | xi <= z[2L] | xi[xi == z[3L]] :
longer object length is not a multiple of shorter object length
4: In `levels<-`(`*tmp*`, value = c("CENTRUMP", "", "FIDESZ", ... :
duplicated levels will not be allowed in factors anymore
5: In `levels<-`(`*tmp*`, value = c("CENTRUMP", "", "FIDESZ", ... :
duplicated levels will not be allowed in factors anymore
6: In `levels<-`(`*tmp*`, value = c("Refusal", "Don't know", ... :
duplicated levels will not be allowed in factors anymore
7: In `levels<-`(`*tmp*`, value = c("No second language mentioned", ... :
duplicated levels will not be allowed in factors anymore
8: In `levels<-`(`*tmp*`, value = c("Sans dipl", "Non dipl", ... :
duplicated levels will not be allowed in factors anymore
9: In `levels<-`(`*tmp*`, value = c("\"Ej avslutad
folkskola/grundskola\"", ... :
duplicated levels will not be allowed in factors anymore
10: In `levels<-`(`*tmp*`, value = c("Armed forces", "Legislators,
senior officials and managers", ... :
duplicated levels will not be allowed in factors anymore
11: In `levels<-`(`*tmp*`, value = c("Armed forces", "Legislators,
senior officials and managers", ... :
duplicated levels will not be allowed in factors anymore
12: In `levels<-`(`*tmp*`, value = c("K", "K", "Frederiksborg Amt", ... :
duplicated levels will not be allowed in factors anymore
13: In `levels<-`(`*tmp*`, value = c("P", "L", "Kesk-Eesti", ... :
duplicated levels will not be allowed in factors anymore
14: In `levels<-`(`*tmp*`, value = c("Galicia", "Principado de Asturias", ... :
duplicated levels will not be allowed in factors anymore
15: In `levels<-`(`*tmp*`, value = c("Stockholm", "", "Sydsverige", ... :
duplicated levels will not be allowed in factors anymore
> str(d2$HAPPY)
Factor w/ 13 levels "Extremely unhappy",..: NA NA NA NA NA NA NA NA NA NA ...
Oh, heck, all the values are missing!! Somehow, putting
"to.data.frame" inside the read.spss causes a different outcome than
using as.data.frame after reading in the data.
The symptoms of this in R-2.9 are a little different, but the
conclusion the same. Help?
In case you are a student who wants to work with this data, I can
share to you the large script that I have been accumulating so that
you might "play along". It turns out to be surprisingly difficult to
"recode" these factor variables that have levels like "none", "1",
"2",..."9", "total".
## Paul Johnson
## November 13, 2009
## A question arose in the lab. A student asks "I want
## to compare the answers from two different editions
## of the European Social Survey.
## I will add this to Stuff Worth Knowing later, but
## I can share this tutorial to you right now.
## From this website:
## http://ess.nsd.uib.no/ess
## Download those European Social Survey Datasets into a directory.
## In a terminal, use the unzip command:
## unzip ESS3e03_2.spss.zip
## unzip ESS2e03_1.spss.zip
## Then run the following in R.
library(foreign)
d2 <- read.spss("ESS3e03_2.por",to.data.frame=T)
d2 <- read.spss("ESS3e03_2.por")
warnings()
### You can try to go into a data frame in one
### step, that's an option in read.spss. But
### we saw warnings, and wanted to be careful.
d2 <- as.data.frame(d2)
d2$whichSurvey <- 2
d3 <- read.spss("ESS2e03_1.por")
d3 <- as.data.frame(d3)
d3$whichSurvey <- 3
namesd2 <- names(d2)
namesd3 <- names(d3)
commonNames <- intersect( namesd3, namesd2)
combod23 <- rbind(d2[ , commonNames], d3[, commonNames])
save(combod23, file="combod23.Rda")
## Error
##Warning messages:
##1: In `[<-.factor`(`*tmp*`, ri, value = c(NA, NA, NA, NA, NA, NA, NA, :
## invalid factor level, NAs generated
##2: In `[<-.factor`(`*tmp*`, ri, value = c(NA, NA, NA, NA, NA, NA, NA, :
## invalid factor level, NAs generated
##3: In `[<-.factor`(`*tmp*`, ri, value = c(1, 1, 1, 1, 1, 1, 1, 1, 1, :
## invalid factor level, NAs generated
## That worries me a little bit. The warnings did too.
## Inspect a few lines in the result.
combod23[1:4, ]
## fix doesn't work for me, did not bother to investigate.
##> fix(combod23)
##Error in edit.data.frame(get(subx, envir = parent), title = subx, ...) :
## can only handle vector and factor elements
## That means some data from hell came into this thing.
## I suspect that combod23 is OK.
## The memory use on this exercise is huge! Try to help it
rm (d2)
rm (d3)
## But I worry. I have 2 ways that I use to try to figure this
## out. One is to open the dataset in a clone of SPSS called
## "PSPP". Actually, the executable is "psppire".
##
## The other thing I do is open the same data again in
## a numeric format, and compare the 2 combined data frames
## This is also a useful exercise because it helps you
## understand what a "factor" is in R.
dn2 <- read.spss("ESS3e03_2.por", use.value.labels = F)
dn2 <- as.data.frame(dn2)
dn2$whichSurvey <- 2
dn3 <- read.spss("ESS2e03_1.por", use.value.labels = F)
dn3 <- as.data.frame(dn3)
dn3$whichSurvey <- 3
## Might be smart to compare
# dn2$HAPPY[1:50]
# d2$HAPPY[1:50]
namesdn2 <- names(dn2)
namesdn3 <- names(dn3)
commonNNames <- intersect( namesdn3, namesdn2 )
combodn23 <- rbind(dn2[ , commonNNames], dn3[, commonNNames])
save(combodn23, file="combodn23.Rda")
table( combod23$HAPPY, combodn23$HAPPY)
## In summary, whenever I want to use a variable from
## the combined data frame, I would probably compare
## against combodn23 just to feel safe.
## Note, after when you come back to work on this project again, you
## might as well just reload the saved copies of combod23 and
## combodn23.
## load("combod23.Rda")
## load("combodn23.Rda")
## That will put you at the current spot, no need to redo the merge
## Now, about "recoding". If you just want numerical
## data, you might consider using combodn23.
## But if you want some factors and some numberical
## variables, then you might need to recode to reclaim
## values.
## HAPPY turns out to be an interesting example of a
## PAIN IN THE ASS because in SPSS, it is scored from
## 0 to 10, but they give value labels only for scores
## 1= Extremely unhappy
## and
## 10= Extremely happy
##
## And the SPSS column has no labels for values 1-9.
## If SPSS gave NO labels at all, then this would come
## into R as a numeric variable. BUT, because there are
## 2 levels named, then R makes a factor out of it.
## When R turns it into a factor, you
## end up with a nutty looking factor, which has
## levels you don't really appreciate.
levels(combod23$HAPPY)
# [1] "Extremely unhappy" "1" "2"
# [4] "3" "4" "5"
# [7] "6" "7" "8"
#[10] "9" "Extremely happy" "Refusal"
#[13] "Don't know" "No answer"
## Create a new variable to play with
combod23$HAPPY2 <- combod23$HAPPY
## Change Extremely Unhappy to text "0"
levels(combod23$HAPPY)[1] <- "0"
## Change Extremely Happy to "10"
levels(combod23$HAPPY)[11] <- "10"
HELL <- levels(combod23$HAPPY)
### Look at HELL
HELL
combod23$HAPPY2[combod23$HAPPY %in% HELL[12:14] ] <- NA
##CHECK RESULT
table(combod23$HAPPY, combod23$HAPPY2)
## Eliminate the unused levels from HAPPY2
combod23$HAPPY2 <- factor(combod23$HAPPY2)
### Same is found with
## combo23$HAPPY2 <- combo23$HAPPY2[ , drop=T]
## Use the "factor trick" to
## reset the variable back to numeric:
combod23$HAPPYN <- as.numeric(HELL)[combod23$HAPPYN]
##CHECK RESULT
table(combod23$HAPPY, combod23$HAPPY2)
## CHECK by comparing against numeric data from spss
table(combodn23$HAPPY, combod23$HAPPYN)
## Next, a student asks "how can I make that same recode
## on a lot of variables?" I'm going to have to leave
## that one unanswered. I think the answer will probably
## be to get a list of variables, then use "lapply" to
## do the same thing to each variable in turn. But
## I have not written up a simple, understandable example
## yet
## After the data is all recoded and homogenized, then we
## could run any analysis we want, and throw in the variable
## "whichSurvey" to see if there is a difference beteween the
## two models.
## Example, choose your y and x1 and x2, then
## mod <- lm(y~ (x1+x2)*whichSurvey, data=combod23)
## or if you think the difference is just in the intercept:
## mod <- lm(y~ x1+x2 + whichSurvey, data=combod23)
