Note: these posts are copied over from the ‘mathbucket’ section of my old tumblr blog and I haven’t put much effort into this, so there is likely to be context or formatting missing.
The Cognitive Reflection Test came up in the SSC Superforecasters review. I’ve seen it a couple of times before, and it always interests me:
- A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?
- If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets?
- In a lake, there is a patch of lily pads. Every day, the patch doubles in size. If it takes 48 days for the patch to cover the entire lake, how long would it take for the patch to cover half of the lake?
I always have the same reaction, and I don’t know if it’s common or I’m just the lone idiot with this problem. The ‘obvious wrong answers’ for 2. and 3. are completely unappealing to me (I had to look up 3. to check what the obvious answer was supposed to be). Obviously the machine-widget ratio hasn’t changed, and obviously exponential growth works like exponential growth.
When I see 1., however, I always think ‘oh it’s that bastard bat and ball question again, I know the correct answer but cannot see it’. And I have to stare at it for a minute or so to work it out, slowed down dramatically by the fact that Obvious Wrong Answer is jumping up and down trying to distract me.
I did a maths degree. I have a physics phd. This is not a hard question. Why does this happen?
I know I have a very intuition-heavy style of learning and doing maths. For the second two I have very strong cached intuitions that they map to, whereas I’m really lacking that for the first one for some reason. I mean, I can visualise a line 110 units long, and move another 100-unit line along it until there’s equal space at each end, but it’s not some natural thought for me.
Now, apparently:
The CRT was designed to assess a specific cognitive ability. It assesses individuals’ ability to suppress an intuitive and spontaneous (“system 1”) wrong answer in favor of a reflective and deliberative (“system 2”) right answer.
Yeah so that definitely isn’t getting tested for me. My System 2 hates maths and has no intention of putting in any effort on this test, but luckily System 1 has internalised the ‘intuitive and spontaneous’ answer for two of the questions for me. I will fail the first question unless my equally strong ‘the answer can’t be that obvious’ intuition fires, but that one makes me seriously worried about my answer to 3. as well.
My inability to internalise the bat and ball thing might be a quirk of my brain, but I’m sceptical of this test in general. It’s extremely vulnerable to having the right cached ideas.
drossbucket 
Random SSC lurker that saw your post.
Here are my thoughts on the bat and ball question:
The bat and ball question primes the test taker to try to split the total into two parts by listing two things, a bat and a ball. Then it gives you a thing that looks like the size of one part, triggering heuristics to use subtraction for the remaining part, without hinting that the result of subtraction needs to be divided in half (one half for the ball, one half for the bat).
Consider this alternative problem:
A centered piece of text and its margins are 110 columns wide. The text is 100 columns wide. How wide is one margin?
Same numbers, same mathematical formula to reach the solution. But less misleading because you know there are two margins, and thus know to divide by two after subtracting.
On the topic of thinking by heuristics, I write code and design algorithms based on heuristics, with careful deliberation saved only for the details that really need it. It works well because I’ve been doing it for a long time. Use of heuristics probably varies with familiarity with an area. Slow deliberation when an area looks unfamiliar, until heuristics that perform well enough are developed. Then the heuristics are used to the extent that they work, with deliberate thought filling in what heuristics can’t do.
Heuristics are useful, but it’s important not to start them with a bad path. The CRT questions attempt to set a bad starting path, but that only works on people who do not already recognize the situation and heuristically pick their own starting path.
So I agree that the CRT is vulnerable to having the right background.
LikeLiked by 1 person
Thanks for the comment – I really like your version with the paper margins! As well as making it clear that there are two margins, in your version the 100-unit object is something tangible. In the bat-and-ball version, the 100-unit object is a more abstract quantity, the difference in price between the bat and the ball. As you say it’s much more tempting to slap those 100 units onto one of the existing tangible objects, the bat, leaving 10 units for the ball.
And of course the nice round numbers help make the wrong version even more appealing! There’s a surprising amount to unpack in such a simple problem. I still think that with this problem it’s harder to guard against the obvious wrong answer than the other two – still not quite clear on what heuristic I can pick up that will allow me to just intuit the correct result without thinking.
I agree with your comment on heuristics too. I’m sort of a lazy thinker who’s normally too keen to jump to heuristics, with the dangers of jumping to a wrong path that that implies. Absorbing new heuristics is the fun part though!
LikeLike
I have no degrees in anything, so this might not make much sense or use terminology in strange ways, or just be completely alien.
Maybe abstraction level is the key concept here. A thing that is derived from relative values of more tangible things is more abstract. The bat is tangible (AL 1), the ball is tangible (AL 1). Their sum is AL 2. If that sum was used as part of something, that something would be AL 3.
If an AL N value is composed of two AL N-1 values added together, subtracting one naturally yields the other. Similar operation reversal rules apply to the other mathematical operations.
The bat-and-ball problem gives you two AL 2 values: bat plus ball, and bat minus ball. Neither AL 1 value is directly revealed, so subtraction yields an AL 3 value that happens to be double the AL 1 value that solves the problem. The solution path is not steps that always reduce the abstraction level.
The margins version gives you one AL 1 value (text) and one AL 2 value (text plus margins (maybe this should be considered AL 3, since margins is AL 2)). Subtracting the text results in the AL 2 value of the margins added together. Neither margin is known, but they are known to be equal, so dividing by two gives the AL 1 value of one margin. Each step on the solution path reduces the abstraction level of the things being handled.
Does this analysis make sense for the other problems? Can their parts be described by abstraction levels, and how the solution paths change those abstraction levels?
5 machines (AL 1), 5 minutes (AL 1), 5 widgets (AL 2 (machines and time combined)). Widgets are tangible, so the natural reaction is to assign them AL 1, and when two of three AL 1 values in a problem are multiplied by 20, it’s natural to do the same to the remaining AL 1 value.
Time is not actually affected by either the number of machines or the number of widgets because it is an AL 1 value. A heuristic that recognizes that leads to the correct solution path.
Problem 3, the lily pad problem, does not involve abstraction level confusion. 48 days counts as AL 1, “whole lake” is AL N (the exact value of N is irrelevant), “half the lake” is AL N-1, “time to cover half” is AL 1. This problem relies on operation confusion, using the wrong operation relating the AL N and AL N-1 value to reach the AL 1 solution value.
After writing that out, abstraction levels seem to work for describing problem 1, but for 2 and 3 abstraction levels seem kind of contrived and unhelpful. So maybe it’s not a good way to describe the parts of a problem and find a solution.
LikeLike
Kyzentun (doesn’t look like I can reply directly to a comment that nested): interesting analysis. I’ll have to think about it more some time, but I think you’re right that abstraction is a key part of the problem for question 1 but doesn’t particularly help with the other two.
I’m increasingly convinced that these questions aren’t really a natural set of anything much, and I’m not sure how they were picked in the first place. Maybe I’ll have to, like, read the actual paper sometime. I probably have no more time to think about this thing for the next couple of weeks though!
Slightly off topic, your abstraction levels idea reminded me of Bret Victor’s Up and Down the Ladder of Abstraction essay, which is well worth reading if you haven’t already!
LikeLike
I just had an easy time with #1 which I haven’t before. What I did was take away the difference so that all the items are the same (subtract 100), evenly divide the remainder among the items (divide 10 by 2) and then add the residuals back on to get 105 and 5.
The heuristic I seem to be using is to treat objects as made up of a value plus a residual. So when they gave me the residual my next thought was “now all the objects are the same, so whatever I do to one I do to all of them”.
LikeLike
Ah, interesting that you’re thinking about this again. I have been on and off too, and did a bit of background reading, and intend to write a proper blog post about it (not sure when, I never seem to get it to the top of the pile).
Yours is a bit different to what I’m now doing with the bat and ball, but related. I’ve finally got it into my head to visualise the *difference* as 100, so that’s the concrete thing I’m imagining (as a row of blocks 100 units long). Then I just move the row of blocks so that there’s the same amount of space (5 units) at each end.
So I’m not specifically imagining ‘object = value + residual’, but I am doing something similar with dividing the remainder evenly between the items.
(I mentioned this method in the post, but it didn’t seem natural then. I think it’s mostly that familiarity has finally taught me to reify the difference between the bat and the ball and remember to assign 100 units to it, instead of falling into the trap of assigning 100 units to the bat because it’s a concrete thing.)
Kyzentun above does something similar with a page and two margins – in that case it becomes a natural question, because ‘page without margins’ is actually a pretty concrete thing, it’s the bit you write in.
LikeLike
It sounds like you require a little more cognitive coupling than me. I have to visualize the quantities laid out across a space but I don’t have to reify each quantity as an tangible object. I assume that as I get comfortable with a particular way of translating a problem into a form I can solve (turning differences into blocks of text and remainders into margins) I gradually drop the more tangible aspects of it, until I forget the metaphor that I am using to the point that it just feels automatic.
That each abstract quantity needs to be assigned to a tangible object in the problem domain, is a valuable principal that I should remember when designing things.
What is it like to solve the following problem?
Glove costs $5 more than hat, hat costs $0.25 more than pin, total is $5.75
what is the price of a pin?
Oh! and this one too.
At work we have 16 ton feed tanks divided into 4 equal-volume sections for easy estimation of how much feed they currently contain. The top three sections are cylinders but the bottom section is a downwards pointing cone.
From the side it looks like this
|__|
|__|
|__|
.\../
_\/_
If each of the cylindrical sections are four feet tall how tall is the cone?
a) 4 feet
b) 6 feet
c) 8 feet
d) 12 feet
LikeLike
> It sounds like you require a little more cognitive coupling than me.
Possibly, yeah. I don’t *always* need to reify quantities like that, but in this case there’s a very loud wrong answer jumping up and down distracting me, and that one *is* concrete, so it tends to win.
I had a go at your questions. For the first one I ended up having to get a pen and paper and crank the algebra out. The answers ended up being fairly nasty and not whole numbers of cents, so I’m not really surprised. How did you solve it?
The first thing I thought of was what I assume was the ‘obvious wrong answer’ (500 glove, 50 hat, 25 pin). That one assigns the 500 to a concrete object, the hat, but does correctly assign 25 to a difference. Not sure how I thought of it, just seemed like the obvious thing to try with those numbers. I then tried 525 and 550 but they were obviously too small and too big respectively.
For the second one I first got distracted by the ascii art and completely misread the question. The question I answered was ‘If each of the sections in the picture is 4 feet tall how tall is the cone?’, for 8 feet. I realised that that was too basic to ask and actually read the question.
I then thought ‘one of the cylinders is 4 feet tall, cones have a third of the volume of a cylinder the same height, so the cone will have to be 12 feet tall’. ‘Cone is a third of a cylinder’ is just one of those facts embedded in my head from school maths, so I just regurgitated it without thinking.
Have I got these right, or am I still misreading something?
LikeLike
The cone one isn’t actively misleading, I just get surprised every time by the fact that half the height of the feed tank has only one fourth of the volume. Also it surprises me that if I cut off the bottom half of the cone it has only one eighth of the volume of the whole cone. It sounds like you have a much better grasp of it than I do. (I just realized that I can think of a cone as a distorted tetrahedron in the same way that I can think of a circle as a triangle and a cylinder as a triangular prism (at this point I went and cut up a block of cheese to confirm that three tetrahedrons go into a triangular prism))
The items for money problem, I can using your process to visualize the differences as two equally spaced objects with a a margin to the left, a margin between them and a margin to the right.
The way I actually did it is I imagined setting each item on a stair step where distance from the floor is the price of that item. $5.00 is the height difference between the second and third step and and $0.25 is the difference between first and second. So the glove must cost $5.25 (second riser plus third riser) more than the pin and the hat costs $0.25 more than the pin. Added together this gives $5.50 so there is only $0.25 cents left for the three sections of rise that correspond to the price of a pin. 25 divided by three is 8 1/3 (because 8 goes into 24 3 times)
so $5.33 1/3, 33 1/3, and 8 1/3.
When you do moderate amounts of arithmetic (involving about 8 numbers) in your head, how do you keep track of which number is which?
Also I want to hear a step by step of what it is like for you to think about physics, because I think that is something that I haven’t experienced.
LikeLike
Sorry I’ve been so slow to reply to this one! I was procrastinating on actually having to do some thinking to understand your stair step process 🙂
Was going to do it now, but I am really remarkably stupid this evening for some reason so it will have to wait a bit. I’ve responded to the rest though.
> It sounds like you have a much better grasp of it than I do.
I actually don’t have a great intuitive grasp of it, and it felt weird to me too. My gut says that if you slice a cone into thirds, the top two thirds are way too big to be crammed around the side of the bottom third to make a cylinder. I drew a picture and if anything it made it feel *more* weird – I drew a cross section of the cone, so just an isoceles triangle, and it’s obviously not true for a plane triangle.
The ‘one third’ fact is really well embedded from school at a rote memorisation level, but I’m not sure how it was justified to us. Maybe it wasn’t!
> at this point I went and cut up a block of cheese
Haha, I need do something like this too. ‘Deform to a triangle’ is a good idea.
> When you do moderate amounts of arithmetic (involving about 8 numbers) in your head, how do you keep track of which number is which?
Um, I just don’t, tbh! Three or four is a stretch for me unfortunately. I’m not quite sure what I do even in that case, but generally I’m not great at mental arithmetic and haven’t developed much in the way of strategies.
> Also I want to hear a step by step of what it is like for you to think about physics
Hm, is there anything in particular you were thinking of? I’m not sure ‘thinking about physics’ feels like any one thing to me, and coming from a maths background I’m not sure I really picked up ‘thinking like a physicist’ to a great extent anyway.
LikeLike
I can’t really tell you what problem in particular I would like you to walk through, because what I’m hoping for is something as distant as possible from my own experiences; I want to vicariously experience what it is like to think as you do. I want to know how you wrestle problems into a form that your intuition applies to.
I don’t really care whether the intuitions are physical or mathematical as long as they are yours.
I’m not surprised that you find mental arithmetic difficult because, for me at least, it is all about creating a network of associations rich enough to keep track of what each of the naturally barren numbers is supposed to be doing. And if you have been relying on thing’s natural set of associations all the numbers would get mixed together.
For me “cognitive decoupling” doesn’t seem to be the ability to ignore particularities, nor the ability to do without them, but the skill of hallucinating vivid new ones into existence.
Your list of ways to think about derivatives is the kind of thing that is vital to my way of thinking. Each one is a metaphor that mediates between mathematical derivatives and another domain. And so when I think through a problem I construct a long chain of metaphors. The primary difficulty I have is keeping that of what is represented by each entity in the metaphor. When I lose that, everything immediately falls apart into meaninglessness.
Because of this solving the ball and bat problem with algebra feels like cheating because you don’t have to wrestle with the associations at all, I think this is because when you performed algebraic manipulations you weren’t aware of what entities were being manipulated in the original domain. If you could line up a row of pennies and show how each operation corresponded to giving actual pennies to a particular object then I would be satisfied that it was a way of understanding.
As an example of how I think, this is how I explained gyroscopes to myself.
I imagine a big horizontal bicycle wheel.
I take the closest part of the rim and focus on that (because it can stand in for all parts of the rim)
I imagine that it traces out a circular path as the wheel spins. This static path can now represent the motion.
I imagine that the chunk has an initial motion vector one unit long and pointing to the right. This can represent the particular path and motion.
I stop the wheel and imagine that I had given that chunk of rim some downwards momentum instead. The wheel would flip over a horizontal axis parallel to me and the chunk of rim would again travel in a circle, but this time vertically.
Both these situations are exactly what I expect and there is no surprise in what the wheel does. But what if I combine them.
I imagine that I have the wheel spinning horizontally, and I add a piece of downwards momentum to the chunk of rim closest to me.
Now the momentum vector has both rightwards and downwards components, so the chunk of tim would take a diagonal path causing the whole orbit to tilt to the right but not tilting the point of impact.
This is exactly what we see in actual gyroscopes so now I have a way to think about them
LikeLike
> I’m not surprised that you find mental arithmetic difficult because, for me at least, it is all about creating a network of associations rich enough to keep track of what each of the naturally barren numbers is supposed to be doing. And if you have been relying on thing’s natural set of associations all the numbers would get mixed together.
> For me “cognitive decoupling” doesn’t seem to be the ability to ignore particularities, nor the ability to do without them, but the skill of hallucinating vivid new ones into existence.
This is *really* helpful, thanks! Now I actually feel quite stupid for not realising this myself.
Nostalgebraist had a good reply to my cognitive decoupling post, challenging me to explain what a ‘decoupled’ mode of thinking should consist of, and I had real trouble replying. I agreed with him that it wasn’t *just* following formal rules (though it probably still correlated with a propensity to do that) but couldn’t work out what more perceptual thing could be going on. ‘Hallucinating vivid new ones into existence’ could be the key.
And definitely it’s something I don’t do much of myself, which fits. It does feel slightly wrong/’like cheating’, similar to how pure algebraic manipulation feels wrong – it’s not using the intrinsic properties of the structure.
Maybe I should get over myself and start learning to do this…
I want to think about the rest of your comment, and the other loose ends in this thread that I haven’t got back to, but I probably won’t get to it for a while. I’m interested in a few other things and my attention is currently split between too many side threads, so I’m getting nowhere fast.
> I can’t really tell you what problem in particular I would like you to walk through, because what I’m hoping for is something as distant as possible from my own experiences
… so, because of the too-many-side-threads problem I probably can’t do this directly for a while, but I do have a monthly email newsletter thing where I talk about physics much more than on my blog, would that be of interest? If so I have your email so can send it. It’s not really focussed on explaining my thinking process, so it might not be ‘low level’ enough for what you want, but some of that probably comes out anyway.
LikeLike
Oh yeah, something else I was thinking about! I ran into a problem at work last week that I couldn’t deal with *at all* (while my three colleagues sitting nearby had no problems). It was really frustrating, and seemed to rely strongly on whatever skills it is I don’t have, but I’m not sure quite how.
I got a new laptop and dock delivered and had to work out how to connect up my existing two monitors to it using the cables, adapters and ports I had available, which were a confusing mix of VGA, DisplayPort, DVI and HDMI. So the formal problem would be a list of statements like:
The dock has two of port types 2 and one of port type 3
Monitor 1 has port types 1 and 2
Cable 1 can connect port type 3 to port type 3
Adapter 1 converts port type 1 to port type 4
… and so on …
and eventually both monitors have to be connected to the dock.
There were a couple of possible solutions but I was incapable of thinking of them, and in the end someone else solved it for me. (I would have been able to solve the problem by trial and error, or by writing the problem down explicitly, but the others were able to just look at it and work out what to do.)
Part of the problem was that I’m just not very familiar with cable types. I’ve only done this once before ages ago, and I think I just had VGA and DVI. So just remembering which was which was a pain. But even given this… I think it relies on more ability to remember and distinguish things than I can muster. This might tie in to what you said about mental arithmetic.
LikeLike
Yes! Your ideas are intriguing to me and I would like to sign up to your newsletter.
Also I don’t mind if you drop threads, or repond late or never because I don’t mean to be piling more obligations on you. It is normal in conversation to only respond to half of what someone says.
I feel like I agree with you in rejecting anything that feels like it is arbitrary, like memonics. But my reason for feeling that these “enhanced associations” are not arbitrary might just be an overextension of the “if it’s stupid and it works then it’s not stupid” principle (if it’s a hack and you prove it always works then it’s not a hack, it’s a technique).
Also as an aside the “5 wigets per 5 machines in 5 minutes” isn’t automatic for me, I have to break it down into sources sinks and converters just like I would have to use to figure out how to attach cables.
LikeLike