Crackpot time 3: speculations will turn out well?

In 2017 I wrote two posts about my about my experiences with attempting to do physics outside of academia, which I called Crackpot Time 1 and Crackpot Time 2. At the time I was trying to reconnect to a more expansive, free-ranging energy that I had lost during the hyperfocus on technical details required for Ph.D. work. I was enjoying the ‘crackpot’ label as a kind of tongue-in-cheek pointer to the style of thinking I was trying to cultivate. I wanted to directly attack any topic that looked interesting, without fussing about whether the topic was ‘too ambitious’, or ‘too difficult’, or ‘not my field’. Small details like a total lack of relevant expertise didn’t matter.

I had a lot of this kind of energy in 2017, which was a very good year intellectually for me. I went to two deeply unusual and inspiring physics workshops that immediately raised my ambitions for what it would be possible for me to do in my spare time alongside a full time job. At the same time I was starting to take my side interest in mathematical intuition more seriously, and get oriented reading some phenomenology for the first time, so it was an intense time where I felt like the horizon was opening up fast in all directions. I started this blog and cranked out a bunch of short, unpolished but enthusiastic blog posts to try and make some sense of my thoughts.

I’ve been meaning to write another Crackpot Time update ever since, but just… never have. Partly that’s because I started a monthly newsletter practice in 2018 that took over some of the same role. But also it’s the standard inspiring workshop problem: the inspired feeling eventually wears off and then you then have to do the hard bit, which is doing the actual work. This is less immediately exciting and doesn’t autogenerate breathless updates about how amazing everything is, so they stopped appearing. I’ve finally decided to crank one out anyway, even if it’s effortful and uninspired.

At the beginning of 2020 I got this fortune cracker for Chinese New Year. Perfect fortune for a crackpot, right?

I’m now trying to evaluate whether speculations did in fact turn out well. It’s weirdly hard to decide. I’m normally at least somewhat confused by my progress – trying to do independent work in a complicated domain is slow and ambiguous at the best of times – but I think this is the most confused I’ve been in a long time. Long 2020 has obviously been enormously strange for everyone, and then on top of that I’m in a hard-to-interpret stuck phase. This is my best attempt to explain what I’ve been up to, and where I’m at now.

Focus and accountability

I’m not going to try and go over everything I’ve done since 2017, nobody cares including me, but I’ll do a few quick catch-up paragraphs to get me to the beginning of 2020. I had two good strategic ideas at the start of 2018. The first was to pick a very specific topic to focus on. My natural tendency is to dissipate my energies going partway down some interesting rabbithole before getting distracted by something else, and only end up with a very vague high-level understanding of anything. Useful for getting a sense of the territory, useless for making any sort of meaningful contribution to it.

To counteract that, I picked a single 8-page paper, A toy model for quantum mechanics by S. J. van Enk, as my focus for the whole year. I had some sense that this particular paper would be a good anchor for me, and that turned out to be correct. The core toy model is very concrete and easy to play around with, but touches on a number of ideas in quantum foundations that interest me – negative probabilities, the phase space formulation of quantum physics, the Spekkens toy model. There are also potential intriguing connections to my favourite recurring fascination, retrocausal interpretations of quantum physics. Having to stick close to the anchor paper meant I could explore aspects of these big topics without disappearing off into uselessly ungrounded speculation.

The second good idea was to use a monthly email newsletter as an accountability mechanism, inspired by this post. This wasn’t a Substack or anything, just a bog standard email that I sent out to a handful of people. I’d ramble a bit about what I’d done in the month, and that gave me a bit more incentive to stay on track. I stuck fairly closely to the area of this paper for the whole of 2018 and didn’t stray much further in 2019 either. This gave me far more focussed knowledge than I’d managed to pick up before working on my own.

At the beginning of the year I wrote the following:

My plan for 2018 is to go beyond just learning some physics in my spare time and to do ‘something novel’, interpreted broadly. ‘Novel’ in this case doesn’t have to mean original research (though that would definitely count) – I’m thinking of a wider conception of what counts as a novel contribution, in the style of Chris Olah and Shan Carter’s Research debt essay (I wrote some comments on it here).

I’ve never been too fussed about whether anything I do is original in the sense required for an academic physics paper, as a completely new technical contribution to the field. But my ambitions are higher than just passively making notes from a textbook. I want to follow my own curiosity trail through a subject, write down what I notice on the way, and highlight ideas and connections that currently aren’t available in digested blog post form. The sort of work that Olah and Carter call ‘research distillation’ in the essay linked above.

This took longer to spin up than I was initially hoping for, and I spent most of 2018 just learning background and writing notes. I finally got going in 2019 and had a few thoughts on negative probabilities from a somewhat novel angle, which produced a couple of posts and a mildly popular twitter thread. So that takes me up to 2020, and the fortune cookie.

Long 2020

In early 2020 I had a tedious hour+ two-bus commute to work and sometimes skimmed some interesting-looking papers on my phone. Otherwise I wasn’t getting much done, because my energy was sapped by the stupid commute. I decided to have a twitter break in February to claw back whatever time I could, which worked fairly well. Some time near the end I spent a Saturday holed up in a corner of Bath University library where I had an idea for a very basic toy model that was quite limited by itself but maybe extensible in some interesting way. I was excited to figure out what it could do and started fiddling around with that for the next week or two.

I got back on twitter on March 1 to discover everything had been replaced by coronavirus panic, which was a big shock to me because I had almost completely ignored it until then. So I started catching up on panic, and the toy model went out of my head for the next couple of months along with everything else that wasn’t covid. I no longer had the bus commute, but I also couldn’t think properly, so that didn’t help much.

After a couple of months my brain came back online at least partially, but the toy model was completely dropped. (I still haven’t managed to pay it any consistent attention, it’s a loose thread at the moment.) Instead I remembered the papers I’d been reading on the bus. I’d been learning about Abramsky and Hardy’s logical Bell inequality work, and I realised that I could use the tools from this to finish off a half-baked idea for a post on Bell’s theorem that I had, connecting a classic popular-science explanation to the version you’d find in a text book. The logical Bell inequality techniques made a natural bridge between the two, and over the summer I was able to use this idea to extend my scrappy notes into a full post that I was pretty happy with. I was finally managing the kind of distillation work I’d been thinking about at the start of 2018.

After that I was on a roll, and found a second use for the logical Bell techniques. In my 2019 posts on negative probabilities I used a very simple toy model created by Dan Piponi as an illustrative example. I picked it because it was simple, but I was also intrigued by its relation to quantum physics – it’s structurally similar to qubit phase space, but the specific numbers are different. In a sense it’s even further from classical physics, with the negative probability being more negative than anything allowed in quantum physics.

I’d noticed before that this was interestingly parallel to a much more well-known case of something being ‘worse than quantum physics’, the Popescu-Rohrlich box, but thought it was only a vague similarity. Once I had the logical Bell tools I realised that there was an exact numerical analogy. I couldn’t find this described anywhere else, so I started writing it up.

Unfortunately this took long enough that it took me into the long depressing UK lockdown winter. The news was a constant stream of miserable statistics from the new covid variant mixed in with increasingly batshit US election nonsense, the weather was dark and grey, and working from home was getting more and more tedious. I eventually managed to finish the ‘worse than quantum mechanics’ stuff and get it out as two blog posts, but that overstressed my limited ability to care about things and once I published the posts I lost interest. I made some very half-hearted attempts to find out more about whether this was actually novel, and when this wasn’t completely straightforward I just dropped it. That was some time around February and I still haven’t picked it up again.

So… now what?

I’m writing this up now because I suddenly have a lot of free time. I’ve just quit my job – last day was last Friday – and haven’t lined up another one. I’m planning at least a couple of months off before I start thinking seriously about getting a new job. So this would be the perfect time to pick this up again. I’m not too bothered if I can’t get my attention back round to physics, because I have other weird projects that I am still keen to work on, but it does seem like a shame to just drop all this stuff. I’m not going to push it though.

The thing I’m feeling most is the lack of social support. I’m not naturally plugged in to a community of people in quantum foundations who are thinking about similar topics, so it can be difficult to keep motivation. David MacIver has a great newsletter post on Maintaining Niche Interests, where he talks about struggling with the same problem:

“Nobody actually wants to know” is a bit unfair. It’s more like… there are people who are interested, but they are both less interested than I am in the subject, and also I don’t talk to them much. The people who I talk to on a regular basis are not interested, because this is mostly not their field.

I feel it even more keenly in comparison with some of my other interests that I talk about on this blog and newsletter and on Twitter, where I do have some sort of community. I can talk about some pretty niche topics – Derrida, Vygotsky, the Prussian education system – and get meaningful informed responses from other people. Book recommendations, suggestions for related areas to explore, that sort of thing. It’s not the same as being in a densely-networked in-person research group, but it goes a surprisingly long way.

The pandemic has definitely made it worse. I do normally get some sense of shared community from the physics society I’m in, which organises workshops and meetups (including the two really inspiring ones I went to in 2017). But it’s very much a community built around meeting in person, rather than around producing large quantities of English-language text on the public internet. We’ve tried a few online calls and talks, but it’s not the same.

Even without the pandemic, though, I struggle with this. I’m just not very good at collaborating when it comes to physics. A lot of this is rooted in defensiveness – I’m just weird for a physicist, kind of slow and mediocre technically and with an odd thinking style, highly focussed on examples and weak on abstraction. I go into any interaction worrying that I’m going to look stupid and expecting to not be able to get my point across, which makes it even harder to get my point across, which… you get the idea. It’s difficult. I think I could make good incremental progress on this in the same way I made progress on blogging, but getting the right supportive environment to start the feedback loop going is tricky. Physics culture is not known for providing what I want.

In the mean time I’m going to keep plugging on with other projects and not force anything. After all, it’s been a strange enough year that I should probably feel happy that I did anything at all. Hopefully my interest in physics will return soon and I can get a better sense of whether speculations have turned out well.

The shitpost-to-scholarship pipeline

I’m at @ssica3003‘s Sensemaker Workshop today, and thought it would be fun to get a blog post out while I’m here, so I dug out this draft I wrote back in August for the newsletter. I wasn’t sure I liked it much at the time, but reading back it’s better than I remembered and works as a first stab at the the idea, at least. Hopefully I can get it a little further down the pipeline. There are some questions at the end, so let me know if you have any thoughts.

shitpost_basicAnyway, the rough idea is… there’s an extraordinary explosion of creative idea generation going on online. And there’s this fascinating kind of pipeline where people will start feeling out the vague beginnings of an idea through twitter threads and dumb throwaway posts and blog comments and email conversations, and then if something looks promising they’ll discuss more and pull in bits of other people’s ideas, and gradually build up to more thought out, polished work.

I’m excited about this culture for a lot of reasons. It’s a kind of online version of the casual, unobservable ‘dark matter’ part of academia, the part you can’t access by looking at published work – all the throwing around wild claims over coffee in the common room and in the pub on Friday evenings, bits of ‘yeah, that paper is unreadable, but this is what they’re really talking about’ insight from people in the know, standing round the whiteboard trying to figure something out, group meeting gossip, and the like. And it’s an incredibly vivid and alive version, at a time when large parts of normal academia have become rigid and bureaucratised and plain boring. This seems important to me: it’s the shitposting engine that produces the raw generative power that can drive more focussed work further down the line.

There’s a kind of wild energy; people aren’t afraid to go after big topics. We’ve got ourselves free of the constraints of the academic pimple factory:

An example of Little History is an essay by Matt Might (clearly a Marvel superhero in a counterfactual universe) titled The Illustrated Guide to a Phd. Go read it. It’ll only take a minute. It frames the sum of all human knowledge as a big circular bubble, and your PhD as a little pimple on the surface of it. I’ll call this the Mighty Diagram. It gets passed around in graduate student circles with depressing frequency.

Instead of a dent in the universe, you get a pimple on a uncritically proceduralist conceptualization of the frontier of knowledge as the sum of all the peer-reviewed academic literature in the world.

What makes this essay utterly horrifying is that it is actually an accurate description of what a PhD is; it calibrates academic career expectations correctly and offers an accurate sense of perspective on the peer-reviewed life. I suspect Matt Might sincerely intended the essay as a helpful guide to academic survival, but its effect is to put aspiring scholars in their place, rather than help them find a sense of place in the universe. It’s a You Are Here map for your intellectual journey at the end of a PhD, you disgusting little pimple, you. Kneel before this awe-inspiring edifice of knowledge that you’re lucky to be allowed to add a pimple to.

This rings very true with my own experience of academia, and the mindset it got me into. I personally found that after a couple of years out of there my thinking kind of cleared and became more expansive, and I was able to have good ideas again.

Unfortunately, this pipeline only goes so far. Currently, I think we’re in something like this situation:

shitpost_fullIt currently tends to dump ideas out somewhere around the ‘insight porn’ point – ideas that you read, think ‘oh that’s clever’, hit the like button, maybe comment on or talk about for a bit, then completely forget a week later. In the best case, a fragment of the idea or a bit of new jargon escapes into the local thought soup and can be combined with other ideas that are currently percolating. Sometimes this can be quite a powerful effect on its own. But there are a lot of places that academia still goes to that just can’t be reached in this way.

One of my favourite examples of this dynamic is Sarah Perry’s theory of mess. This is a genuinely great idea, and it’s not just a vague ‘insight’ – it’s an initial sketch of a satisfying explanatory theory of what mess is, complete with some very convincing examples and thought experiments (put a kaleidoscope filter on your mess, and it’s no longer mess!). But as far as I can tell, it got the same treatment as everything else that goes down the pipe – we all liked it and moved on. No real discussion (that I know of) of how to test it, or what has already been done in this line, or probing to see where it might fall down. Does it work? Who knows! On to the next idea!

Now, there’s an obvious explanation for why this happens. Most of us are not doing this as a full time job. We’re fitting this into the spare time we get, alongside paid work or other responsibilities. So we’re only really interested in doing the enjoyable parts of the idea generation process. Chucking around ideas is easy and fun, whereas checking whether they actually work is hard and boring. It’s not a big surprise that people prefer easy and fun work to hard and boring work.

There’s a lot of truth to this, but I think it’s slightly too cynical, in that it both makes the first part of the pipeline sound too easy and the second half of the pipeline too hard. Chucking around ideas is easy, but to be able to do that we need to have some good ones to chuck around, and that’s not exactly trivial. We have some advantage in being able to go after very broad, vague, ambiguous, undeveloped topics, and slowly clear fog. There’s no pressure to quickly get to a point where we can publish something. And at the same time, polishing up ideas is hardly some unrelenting tedious grind. Calculating can be fun, testing can be fun, writing up can be fun. If your eventual aim is to publish in traditional academia then there are some definite unfun parts, like altering your conversational blog post style to fit a more academic register, but this is only one part of the process.

My own experiments

For me, at least, it just feels unsatisfying to leave ideas at the insight porn stage. There’s a natural pull in the direction of getting further down the pipeline, rather than a tedious sense of duty.  I’ve been playing around with some haphazard experiments of my own, and I think I’ve got past the insight porn stage too with some of them, but nowhere near as far as I’d like. I’ll go through a couple of examples.

A few years ago, I wrote a tumblr post called stupid bat and ball, title all lower case, 700 words of low-effort writing not far above the shitpost level. I wasn’t really expecting it to go anywhere further. But it did contain a small core of insight – the bat and ball question of the Cognitive Reflection Test is different to the other two questions in some respect, and so the questions don’t really form a natural set. When I got the wordpress blog I reposted it, and eventually it attracted some really good comments that probed the mechanics of the bat and ball question much more deeply than I had. So I realised that this idea probably was worth investigating and that I should up my game a bit, and I started reading some of the literature. I discovered that the bat and ball question came first and the others were picked ‘to be like it’, with no elaboration of the process for picking them, which confirmed my suspicion that not much work went into question validation. And I found a fascinating follow-up paper showing how ridiculously sticky the wrong answer is.

The comments to this post pushed things further again, coming up with more detailed explorations of how the difficulty relates to the way the problem maps numbers to an abstract quantity (the difference in price), but fools you into mapping it to a concrete one (the price of the bat). @_awbery pointed out that this abstract/concrete confusion is completely missing from the other two questions, where all the quantities map to concrete objects. And anders devised a set of ‘similar questions’ that turn up the level of abstractness one step at a time. These comments point towards something like rat-running experiments for the Cognitive Reflection Test, getting an understanding of how the tools we’re trying to use actually work before using them to make inferences about abstractions like ‘cognitive reflection’. I do think a potentially valuable contribution could be made here. 

But… I’m not really the person to do it. (Even if I cared more about this specific question than I do. I’d pretty much used up my remaining store of shits-to-give on writing the blog post, and didn’t even have enough left to engage with the comments as fully as I’d have liked to.) Doing psych research without fooling yourself sounds like an absolute minefield even if you know what you’re doing, and I have no expertise at all. 

So I guess in this case I quit the pipeline at the level of having a sort of slapdash lit review with some pointers to interesting ways to take it further. Not the most impressive result. But the interesting bit for me was the distance I travelled from the original tumblr post, which I’d put no effort into at all, and the way the project took on a life of its own, with other people helping to propel this considerably further than I’d ever thought to take it myself.

My other example is all my thinking about negative probability in the last year and a half. Although it sounds superficially like a kind of a crackpot topic, there are deep links to quantum mechanics on phase space, and I’ve been using my fascination with this as a serious starting point to learn all kinds of interesting things in quantum foundations/quantum information. I’ve been experimenting with the discipline of using a single paper as my focus, and this has been incredibly helpful for keeping me on track, and damping down my normal habit of wandering from subject to subject too quickly to pick up anything useful. 

I’m more serious about this project than the bat and ball one – it actually connects to an enduring deep interest rather than something I blundered into by accident. Again, I’m not yet as far down the pipeline as I want to be, but I’ve got past the vague insight level. My last couple of posts explored an intriguing decomposition of the Wigner function for a qubit that I found myself, and that I can see some potential use for in interpreting negative probabilities. Since then had quite a few more ideas that I want to investigate, and I’ve started to link things into a more coherent picture. There’s also a lot more I could be doing in terms of making contact with people in academia and asking questions (something I’m rather bad at). I can definitely see how to push further. 

It’s still really funny to me that I’m cheerfully crashing about between cognitive psych and quantum foundations, with a few clueless forays into reading Derrida for good measure. Whereas in academia I’d have felt daring if I tried to pivot from burst to continuous sources of gravitational waves from neutron stars, or something. Obviously this is too scattered for me to get anything done, and I need to get better at idea triage. But there’s something really psychologically healthy about this mindset of just taking a direct run at whatever I feel like, instead of thinking ‘oh, that’s outside my field, I can’t think about that.’ I want to keep this even as I hopefully learn to focus my efforts more usefully.


Right, I want to push this out now so I can stop being antisocial at the workshop. I’ll end with some questions:

  • What are examples of people navigating the whole shitpost-to-scholarship pipeline successfully on the public internet? I’m particularly interested in people who are trying for academia-style focussed research on specific object-level questions, rather than big-picture synthesis or popularisation.
  • Is there any kind of institutional support out there, or is it all just individual weird nerds pursuing individual weird research programs?
  • Has anyone written about this well already? For a start, Venkatesh Rao had a couple of excellent threads here and here on a similar topic. It’s a much more pessimistic take, which actually fits my current drizzle-soaked winter-brain opinions better than these cheery ramblings from last summer – for example, in most of my experiments I haven’t managed to get much further than this sort of ‘reading published literature and blogging a few derivative observations’ stuff. I’d like to hear about anything else relevant that people have liked.


Crackpot time 2: cargo culting hard

I’m back from another weird foundations-of-physics workshop in the middle of nowhere, this one even smaller and more casual than the last. Also much more relaxed, schedule-wise, so there was plenty of time to think idly about various rubbish in my head.

Last time I was inspired to write my crackpot plan, so it feels like a good time to revisit it a bit, but mostly this is just a large braindump to get various things out of temporary memory before I lose them.

Continue reading

Research debt, double distilled

I said a few weeks ago that I was going to talk more about this article by Chris Olah and Shan Carter on the idea of research debt, but every time I went back to it I felt that the original article made the point so clearly and elegantly that there was very little I wanted to add. (Other than ‘I want this thing in physics too please!’)

So I started to pull out my favourite bits, intending to do a very lazy quotes-and-comments sort of post, and realised that with a couple of additions they made a coherent summary of the original post on their own. In the process I discovered that there were a few things I wanted to say, after all.

If you just read straight down the blockquotes you get a double-distilled microversion of the original essay. Or you can also read the bits I’ve stuck in between.

For centuries, countless minds have climbed the mountain range of mathematics and laid new boulders at the top. Over time, different peaks formed, built on top of particularly beautiful results. Now the peaks of mathematics are so numerous and steep that no person can climb them all.

This has always saddened me. In Men of Mathematics, E. T. Bell labels Henri Poincaré as the ‘The Last Universalist’, the last person to be able to range freely across all fields of mathematics as they existed in his time. Now Bell did have a tendency to over-dramatise things, but I think this is basically right.

Probably this is unavoidable; probably the expansion wave has accelerated too fast, and those days will not return. I’m temperamentally susceptible to millenarian dreams of the return of the once and future universalists, but I accept that this is unlikely.

Still, there is a lot of compression that is within reach:

The climb is seen as an intellectual pilgrimage, the labor a rite of passage. But the climb could be massively easier. It’s entirely possible to build paths and staircases into these mountains. The climb isn’t something to be proud of.

The climb isn’t progress: the climb is a mountain of debt.

The analogy is with technical debt in programming, which is all the awkward stuff thrown to the side in an effort to get software into production quickly. Eventually you have to go back and deal with the awkward stuff, which has an unfortunate tendency to compound over time.

The insidious thing about research debt is that it’s normal. Everyone takes it for granted, and doesn’t realize that things could be different. For example, it’s normal to give very mediocre explanations of research, and people perceive that to be the ceiling of explanation quality. On the rare occasions that truly excellent explanations come along, people see them as one-off miracles rather than a sign that we could systematically be doing better.

People who are truly excellent at explaining research are probably rare. But ‘better explanations than we have currently’ seems like a very, very easy target to hit, once people are persuaded to put resources into hitting it.

I plan to finally start taking my own advice soon, and start putting whatever notes and bits of intuition I’ve gathered online. I’m not too convinced that they’ll be especially great, but the current floor is pretty low.

Research distillation is the opposite of research debt. It can be incredibly satisfying, combining deep scientific understanding, empathy, and design to do justice to our research and lay bare beautiful insights.

Distillation is also hard. It’s tempting to think of explaining an idea as just putting a layer of polish on it, but good explanations often involve transforming the idea. This kind of refinement of an idea can take just as much effort and deep understanding as the initial discovery.

Distillation is fundamentally a different sort of activity to the types of research that are currently well supported by academia. Distillers aren’t mountain climbers; they engage with their subject by criss-crossing the same ground over and over again, following internally-generated trails of fascination that can be hard to interpret from the outside. They want to understand!

An aspiring research distiller lacks many things that are easy to take for granted: a career path, places to learn, examples and role models. Underlying this is a deeper issue: their work isn’t seen as a real research contribution. We need to fix this.

Distillers generally have little interest in who can get to the top of the mountain fastest, and anyway it certainly won’t be them. In an environment that rewards no other activity, they tend to disappear quickly. They require different infrastructure.

None of this infrastructure currently exists, but it easily could do. Research distillation doesn’t intrinsically need to cost huge amounts of money. It’s not like we need to spend billions on a gigantic high-energy collider to smash our current explanations together. This is an area where transitioning from moaning about academia to actually doing something about it looks to be pretty straightforward.

It’s one of the nice sorts of problems where small efforts at the margins are already useful. It certainly helps if you have Google’s resources behind you, but you can also just polish up any half-decent notes you have lying around on a topic that’s currently poorly explained and put them online, and you’ve made a tiny contribution towards fixing the problem.

If you are excited to distill ideas, seek clarity, and build beautiful explanations, we are letting you down. You have something precious to contribute and we aren’t supporting you the way we should.

I’ve been saying ‘they’ throughout this post, but, I mean, it’s obvious why I care about this thing. This is my old tumblr ‘about me’ page:


It’s amusing self-deprecation, but unfortunately I also meant it a lot of the time. (I still believe the programmer bit, but I’m starting to have some optimism about improvement there too.) My standard line after finishing my thesis was ‘I love physics but I’m bad at research’.

I had a poorly understood but strongly felt sense of what I wanted instead, academia was clearly not going to provide it, and I just wanted to get out. ‘Research distillation’, however, is a reasonably close fit. (Maybe not an exact one. I feel the ‘criss-crossing existing territory’ approach goes deeper than just refining existing ideas, and is a valid route to original research in itself. But it’s an ecosystem I think I would have been able to cope with, and succeed in.)

So I’ll admit my enthusiasm for the idea of research distillation is mostly pure self-interest. But I’m pretty sure that a thriving ecosystem of distillers would also help academia. After all, you only criss-cross the territory for love of the subject. The external rewards are currently too poor for any other motivation to make sense.

Hackers and painters but not physicists?

There are two common routes that people go down after a physics PhD. The first is, of course, to stay on in academia and get a postdoc, and then hopefully another postdoc or two, and then hopefully a permanent job.

Many people fail one of these steps, and many others don’t fancy the whole process in the first place. So the second common option is to leave and look for a job that uses the same skills in some way. This could be in, for example, data science, algorithmically intensive areas of programming, quantitative finance, or industrial research and development. These are challenging jobs that require a lot of thinking, normally with long work hours attached. There isn’t much time and energy left to spare for learning physics, so mostly people don’t do that any more.

I wasn’t particularly good at my PhD, so playing the first game would have been a struggle. And there was enough that annoyed me about academia that I was pretty OK with leaving.

But I also didn’t like the look of the second game. I don’t want to do a challenging job that ‘uses my technical skills’ in some other field. I don’t care about my technical skills. They’re not even very good! I’m completely lacking in the kind of sharp, focussed intellect that excels at rapid problem solving in unfamiliar contexts. I will not pass your whiteboard interview.

I just want to think about physics. I have specific questions that are stuck in my head, and I want to work on those. I may not succeed in doing this very well or finding anything useful, but it’s not going to stop me thinking about them. Getting to use some decontextualised ‘problem solving skills’ elsewhere is not much of a consolation prize, because everything I care about is in the context itself. (This mindset actually makes most academic postdocs look quite unappealing too.)

So in my case it makes more sense to be relatively unambitious career-wise and try to free up time for learning physics. The main useful features of my current job are that it’s reasonably not horrible, has sane hours, and leaves me with mental energy to spare. Eventually I want to do better, and figure out how to cut down the hours I work.

This seems to be an unusual choice in physics. I can only think of two vaguely relevant archetypes: cranks of the classic ‘retired electrical engineer who’s just realised relativity is WRONG’ variety, and the occasional seriously impressive person who ends up the news for making important contributions to number theory while working in Subway. I feel embarrassed admitting to people what I’m doing, because I worry that the subtext they’ll pick up is either ‘I’ve deluded myself into thinking I’m an unrecognised genius’, or ‘I’ve gone full crackpot and no longer care what anyone thinks, also did you know that Einstein was wrong?’ Neither of those exactly sounds good, so I tend to talk about my plans like it’s just a big joke.

The annoying thing is that what I’m trying to do is not at all rare in other fields. Everyone understands the concept of people in the arts having day jobs, for example. The musician who works in a coffee shop to pay the bills is a standard cultural stereotype.

I like several things about this stereotype. For a start, it’s nice to simply have it available, so you can explain what you are doing to others quickly without looking too odd. (Possibly someone will give you a bit of ‘get a real job!’ grief for it, but that’s also part of the script! You already know how to play along in that exchange, so you escape a lot of awkwardness.)

There’s also no requirement for you to have any particular level of ability, so you escape the genius/crackpot dichotomy. You can be a brilliant musician, but it’s also OK to be sort of mediocre, or even downright awful, but love it anyway.

And best of all, it’s expected that your interests will be specific and contextual, that you’ll be driven by a particular obsession. If someone advises you to ‘apply your technical skills’ to writing advertising jingles, it’s a pragmatic suggestion for how to pay rent, not an indication that it could be a comparably fulfilling way to spend your time.

Maybe it’s not so surprising that the idea of people working on physics in their spare time is not well represented in the wider culture. Most people aren’t especially interested in physics, and might not even realise that you can actually care this much. But I’m not just talking about cultural representation. I’m confused about why hardly anyone seems to be doing it at all.

One very obvious point is that the financial situation is so much better than in the arts. All those jobs in data science and industrial research pay very well. This is not something I want to complain about! But it means there are strong forces pulling people into these careers, and away from having time for physics. This probably makes it harder to notice the idea in the first place.

I don’t think this is the whole story, though. In this respect it’s interesting to compare us against a different population of STEM nerds. Programmers are also well paid. But, like musicians, they still manage to have a robust culture of day jobs and side projects. It might not be such a sitcom stereotype, but within the field it’s not seen as so unusual for someone to spend the day at their dull enterprise Java job, and then go home and contribute to an open source project they really care about.

In fact, once you’ve thought of it, the financial situation makes the ‘day job’ idea easier, not harder. The most challenging and well-paid jobs may not leave you much extra time for thinking, but there are other less impressive-sounding options that still pay pretty well. Writing line-of-business applications at Large Dull Company may not be the most exciting way to spend your day, but that means you’re likely to have brainpower to spare. And if you want to reduce your hours further or have more flexibility with your time, there are reasonable pathways in consulting and contracting.

As with artists and musicians, you don’t have to be brilliant to fit this pattern. You just have to be fascinated enough with a specific idea to want to work on it even if nobody is paying you. Writing a basic CRUD app in your spare time is fine if there’s something in there that really interests you.

Paul Graham famously made the same comparison in his Hackers and Painters essay:

The other problem with startups is that there is not much overlap between the kind of software that makes money and the kind that’s interesting to write…

All makers face this problem. Prices are determined by supply and demand, and there is just not as much demand for things that are fun to work on as there is for things that solve the mundane problems of individual customers. Acting in off-Broadway plays just doesn’t pay as well as wearing a gorilla suit in someone’s booth at a trade show. Writing novels doesn’t pay as well as writing ad copy for garbage disposals. And hacking programming languages doesn’t pay as well as figuring out how to connect some company’s legacy database to their Web server.

I think the answer to this problem, in the case of software, is a concept known to nearly all makers: the day job. This phrase began with musicians, who perform at night. More generally, it means that you have one kind of work you do for money, and another for love.

So why is this missing from physics? One good reason is that working independently doesn’t make much sense for a lot of people if they want to still produce original research. If you enjoy being an experimentalist in a big collaboration, you’re out of luck. You can’t build a small version of the LHC in your shed and hope to keep contributing to high energy physics. Some areas of theory would also not work well, such as big numerical simulations, or fast-moving subfields developing highly technical methods that would be hard for an outsider to keep up with.

I think that still leaves a reasonable number of options, though. It’s the same for programmers, surely: you won’t have access to Google-sized datasets in your one-person side project, either. And I’m not necessarily even talking about original research anyway. In fact, I’m explicitly in favour of a much wider conception of what constitutes a useful contribution to physics. I talked about some of the things I value briefly in this ‘niches’ comment on David Chapman’s Meaningness blog:

What I’d like to see is more niches, in the ecological sense: more acceptable ways to be ‘successful’ in academia so that everyone isn’t stuck trying to shove each other down the same boring hill in their quest for the summit. Off the top of my head I would like all these things to be valued as highly as ‘high-impact’ research: teaching, reproduction of experiments, new ways of visualising or conceptualising existing results, communication of existing concepts to people outside of your speciality, programming new tools to make research easier, digging around in the historical archives of the field for interesting lost insights…

Many of these are completely feasible to work on as an individual outside of academia. So I don’t think this is the whole problem.

It’s true that a lot of people are in physics mostly for the problem solving element. In that case, applying your skills elsewhere for more money might look extremely attractive, as you aren’t really losing anything by solving the same kinds of problems in a different field. This could actually be enough to explain the situation completely – maybe there just are very few weirdos like me out there who care about specific questions in physics rather than broadly applicable techniques, but who also don’t try to stick it out in academia. But in that case I don’t really understand why the situation in programming is so different.

I do have to wonder, though, whether one reason that people don’t do this in physics is simply that, well, people don’t do this in physics. Nobody sees anyone else doing it (apart from a few crackpots and geniuses who are easily discounted) so they don’t think to try it themselves. This would be the most optimistic explanation, because it looks so easily changeable! Maybe if more of us continued doing physics outside of academia it would become a thing. Maybe there are already a fair number of people trying this, but they’re currently keeping rather quiet about it.

Mostly I’m just confused, though. Suggestions gratefully received!