Author Topic: Who writes the multiple choice questions that appear while classifying galaxies?  (Read 5855 times)

JeanTate

  • OotD posters
  • Hero Member
  • *****
  • Posts: 2976
    • View Profile
I'm getting increasingly annoyed by them.

I think the underlying intention may be good, but the implementation is, IM(ns)HO sadly lacking in the 'critical review' department.

For example, I just 'failed' one such, to the effect of 'which of the 3 galaxies, in the images, contain many bright blue stars?'

What really annoyed me about this was not so much that the galaxies chosen were not ideal (a very nice, face-on spiral, a giant elliptical with many globular clusters, and a near edge-on (giant) S0 or Sa with a prominent dust lane and lots of GCs) - though there's good ground to criticize here - nor that the waveband of the images was not identified (it was a B&W, with no indication of whether they are all in the same band, to pick on just one shortcoming), but the sloppy use of 'many'.

By any reasonable definition of 'many', a giant elliptical will contain many bright blue stars, at least in terms of their absolute number, and an S0/Sa will too (obviously a blue spiral will, as well)*.

In short, the questions - and answers - are so ambiguous, so ill-defined, as to make the quiz more a test of guessing the intent of the person(s) who wrote them than one's knowledge and understanding of contemporary astronomy/astrophysics.

Can someone point me towards the Customer Service Department please? I wish to register a complaint.  :P

* it may even be possible to make a case that a sufficiently large giant elliptical contains more bright blue stars than a small, dwarf spiral!

UPDATE: I just got the same question; it actually says "very few", as in very few bright blue stars. Which is ever worse, because you'd expect none of these galaxies to have 'very few bright blue stars', as in 'very few' = 'maybe only a dozen or so' ... I mean, if you answered 'very few' to the question 'how many of your close relatives have died of cancer?' when, in fact, a dozen or so had, no one would think you'd answered correctly, would they? Now it's clear what the author intended, but I feel this sort of sloppiness just makes understanding of astronomy more difficult, not easier.
« Last Edit: March 03, 2012, 02:15:26 am by JeanTate »

Budgieye

  • Hero Member
  • *****
  • Posts: 3860
    • View Profile
I am still trying to decided what I did wrong on my answer about mass and luminosity of three stars A B and C, all on the main sequence. I can't recall the exact wording anymore.

JeanTate

  • OotD posters
  • Hero Member
  • *****
  • Posts: 2976
    • View Profile
Ah yes, I vaguely remember that one too.

And you bring up a much more important point than mine; namely, what's the purpose, or intention, of this quiz? If it's to help educated zooites, then it fails pretty badly in the pedagogy department I think: how can you learn something from being told your answer is wrong, when you are given no reference material, no explanations, and can't even review the question?

If it's some sort of experiment, or research, to understand how much (basic) astronomy zooites have internalized, then it also - pretty obviously - fails.

But maybe it's a psychology, what-influences-classification thing? Something like, 'on average, zooites who were told they answered a universe quiz question wrongly went on to classify more accurately than those who were told they'd answered correctly'.

sebastiencormier

  • Newbie
  • *
  • Posts: 1
    • View Profile
JeanTate,

I am part of the research team that is developing the Galaxy Zoo survey.  We apologize for the frustration you're experiencing and we are very grateful for the feedback, it will help us improve the survey.  Let me give you an idea of the research project:

The current survey items are being administered for a preliminary study as part of an ongoing research program with Zooniverse.  Note that the current questions being used have been field-tested with other populations of learners and have been validated by astronomers. The survey is designed to be cognitively accessible to the majority of the population being surveyed, most of whom do not have your level of astronomical content understanding. The two questions you mentioned (which are under review) are part of a cluster of questions on the same subject.   By applying different statistical analysis methodologies (i.e. classical test theory, item response theory, and factor analysis) to the large numbers of responses we have received we are able to assess the performance of individual items in the survey and determine how well different combinations of responses allow us to discriminate the understandings and reasoning abilities of different populations of Zooniverse participants.  In fact, our preliminary data shows that zoo-ites with higher participation have a deeper conceptual understanding of the underlying astronomical concepts of galaxies.

We are currently using statistical analysis of the data from the existing survey, feedback from zoo-ites such as yourself, and data from use of these questions in many other education settings, to create a new and improved version of the survey.

Again, thank you for your feedback,

-Sébastien

JeanTate

  • OotD posters
  • Hero Member
  • *****
  • Posts: 2976
    • View Profile
For anyone interested in what's been happening, Sébastien and I have been exchanging PMs. In one of mine, I elaborated on some of my concerns/complaints; if any of you would like a copy, I would be happy to send it to you.

egalaxy

  • OotD posters
  • Hero Member
  • *****
  • Posts: 2212
    • View Profile
ooh sounds like a great idea! perhaps, we can add some more suggestions! send me a copy. oh by the way JeanTate:

just some humor not meant to offend anyone.

twobob

  • Jr. Member
  • **
  • Posts: 70
    • View Profile
Am I the only one around here that finds this sort of information gathering by stealth insulting. By all means do your survey, but you must state your intentions clearly.

bigjohnton80

  • Newbie
  • *
  • Posts: 2
    • View Profile
JeanTate,

I am part of the research team that is developing the Galaxy Zoo survey.  We apologize for the frustration you're experiencing and we are very grateful for the feedback, it will help us improve the survey.  Let me give you an idea of the research project:

The current survey items are being administered for a preliminary study as part of an ongoing research program with Zooniverse.  Note that the current questions being used have been field-tested with other populations of learners and have been validated by astronomers. The survey is designed to be cognitively accessible to the majority of the population being surveyed, most of whom do not have your level of astronomical content understanding. The two questions you mentioned (which are under review) are part of a cluster of questions on the same subject.   By applying different statistical analysis methodologies (i.e. classical test theory, item response theory, and factor analysis) to the large numbers of responses we have received we are able to assess the performance of individual items in the survey and determine how well different combinations of responses allow us to discriminate the understandings and reasoning abilities of different populations of Zooniverse participants.  In fact, our preliminary data shows that zoo-ites with higher participation have a deeper conceptual understanding of the underlying astronomical concepts of galaxies.

We are currently using statistical analysis of the data from the existing survey, feedback from zoo-ites such as yourself, and data from use of these questions in many other education settings, to create a new and improved version of the survey.

Again, thank you for your feedback,

-Sébastien

bigjohnton80

  • Newbie
  • *
  • Posts: 2
    • View Profile
I must have "passed the two quiz's i took...but i am still wondering why there is no place to check our answers...I cant have made the correct choice on all of my answers can I? I only have one course in Astronomy and that was in the 1970's... :o
anywhere I can check my answers?

AdamKeto

  • Newbie
  • *
  • Posts: 1
    • View Profile
I am also looking for an option to check my answers,but I am not able to find it.Sometimes it gets really very annoying.

wide format ink

Lovethetropics

  • Hero Member
  • *****
  • Posts: 11196
  • I hunt Irregular galaxies And Hyper-Velocity Stars
    • View Profile
<Guys have faith on the Zookeepers ideas,if they don't work they will change it, trust me on this  ;D

 *and find lots of asteroids  ;D

JeanTate

  • OotD posters
  • Hero Member
  • *****
  • Posts: 2976
    • View Profile
I recently agreed to partake in the weekly astronomy quizzes that pop up on Galaxy Zoo during classification, and I have one suggestion for them:

Could the correct answers be displayed once the quiz is complete? I'm embarrassingly uneducated on a lot of the questions but I'd still love to know whether I'm getting them correct, so that if I'm not I can go research them. (Not that I don't tend to research them anyway, but it's more likely to stick in my brain if I know I got it wrong the first time.)

A stand-alone thread, Suggestion for weekly quiz, in Suggestions and Comments.

Here's another, the first post in an older stand-alone thread, quiz, in the same section:

being asked to participate in a quiz, I am happy to participate, however ..... there is no way to find out if one's answers were correct or not, or what the correct answers indeed were. it would be more educational if that information could be supplied somehow, no?
I too would really appreciate knowing the answers to the quiz!  I'm not an astronomer (but an architect), I just am interested, so all the stuff about wavelengths and red and blue is totally lost on me!
A bit frustrating to not even be given a score...
Maybe once the research is done we can be told the answers?
Thanks

And from GZ Talk, in a thread called Quiz answers!, in the Help Board. It was started "6 months ago", and the posts go right up to "4 days ago":

Quote from: chucksensei
I want answers to the quizzes!

Some of the questions I don't know the answer to, nor where I can quickly find the answer! I want to know them.

Education. It's the candle in the dark of ignorance, superstition, and the abyssmal dark of religion.
Quote from: faiq
I also want to know them :-(
Quote from: bajik
Like wise.
Quote from: jonathanfox5
Just to add my support for this. I too would find it useful to have the answers so that I can work out what I got incorrect and therefore go research it.
Quote from: Shaedlaer
It could be useful if we could possibly see answers of other users (or statistic of those possibly even better). Then we perhaps could learn to do it better over time. The way it is currently we don't get feedback at all unless we post a discussion asking if what we're seeing is correct;)
Quote from: geert3
why not simply show a short explanation for each question asked at the end of the quiz? Or at least indicate 'correct/incorrect' when having answered. What is the point of asking the questions otherwise?
Quote from: suelaine
I think that I would have received an F+ on all of these questionnaires, but I am very interested in what the answers are and why. Perhaps a tall order
Quote from: rfguisti
I am very much interested in not only contributing to this project but also learning from it. Having the answers for the quizzes, preferrably with explanations (or links to explanations) would be awesome.

Something else that would be really really cool is a tutorial. Before classifying galaxies, there should be a tutor that would tell you what to look for when classifying. This might also improve the quality of the classification in tenfolds.

And also in the GZ Talk Help Board, a thread entitled simply Quizzes ("3 months ago"):
Quote from: turelli
Where can I find the results of my quizzes?

tks

(I'm sure there's more, but I think the message comes through, loud and clear  ::) ).

And here's jules, Zooniverse Workshop 2013 – The Unconferences Day 1, not even a week old yet:
Quote from: jules
So what we like is ...
Educate us! We like to learn. Some volunteers are teachers. So promote and explain your educational aspects

And from her wrap-up: "So another Zooniverse workshop has been and gone. This one seemed more volunteer-focused to me. The desire to educate, engage and encourage volunteers shone through."

Trick question: what's the opposite of "bacon"?



Katsee

  • Newbie
  • *
  • Posts: 1
    • View Profile
Hey guys, just wanted to butt in and copy a part of my post on the PH forum regarding this quizz issue:

"I understand it might be frustrating to not know how you did, but since it's not really a test meant to help you get better (at least not directly!), but one meant to assess the level of what's already there... It's kinda like testing PH in a pool if you will, the pool represents the zooite population, and the quizz is the PH strip. If there was a way to warn the pool that its PH was being assessed, and it was a.. conscious pool (this analogy is getting worse by the second... ), it could falsify its results (google the answer to get good PH... sneaky, sneaky pool!).

I guess what I mean is it's not an individual exam, individual data isn't the focus (even if you answer them all correctly, you'll be a point on a diagram in a future paper, ultimately..). Perhaps it would be a good idea to have a disclaimer somewhere on site that states what the quiz data will be used for, like a forum post with a short description, but once you understand what it's for, giving a record of how well you did on quizzes would risk affecting the veracity of the data in the long run. It's always hard to gather data on humans; as humans who like to analyze data, we should be in a good place to feel for social scientists/information researchers.. their object of study can talk back to them, get suspicious of their science objective.. plus they don't need outside sources to mess up their data, their data is already totally capable of messing with itself! 8)"

Once again, I understand some might feel uneasy knowing they're being probed, but if it helps any, we're all getting probed together... *coughs* and it's to our benefit: we'll get better zoos out of it, best tailored to our science-attentive needs.

And the opposite of bacon is fear, wasn't it the two poles in that movie Donnie Darko? Fear and Bacon (Love)?

JeanTate

  • OotD posters
  • Hero Member
  • *****
  • Posts: 2976
    • View Profile
Update: zutopian noted that there's an interesting paper published, on the first GZ quiz: Measuring the Conceptual Understandings of Citizen Scientists Participating in Zooniverse Projects: A First Approach, by Edward E. Prather, Sébastien Cormier, Colin S. Wallace, Chris Lintott, M. Jordan Raddick, and Arfon Smith (the link takes you to an Astronomy Education Review page, from there you can easily get the full PDF).

There's some discussion of this over in GZ Talk, here.