EA - Please don't throw your mind away by TsviBT

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Please don't throw your mind away, published by TsviBT on February 15, 2023 on The Effective Altruism Forum.Dialogue[Warning: the following dialogue contains an incidental spoiler for "Music in Human Evolution" by Kevin Simler. That post is short, good, and worth reading without spoilers, and this post will still be here if you come back later. It's also possible to get the point of this post by skipping the dialogue and reading the other sections.]Pretty often, talking to someone who's arriving to the existential risk / AGI risk / longtermism cluster, I'll have a conversation like the following.Tsvi: "So, what's been catching your eye about this stuff?"Arrival: "I think I want to work on machine learning, and see if I can contribute to alignment that way."T: "What's something that got your interest in ML?"A: "It seems like people think that deep learning might be on the final ramp up to AGI, so I should probably know how that stuff works, and I think I have a good chance of learning ML at least well enough to maybe contribute to a research project."T: "That makes sense. I guess I'm fairly skeptical of AGI coming very soon, compared to people around here, or at least I'm skeptical that most people have good reasons for believing that. Also I think it's pretty valuable to not cut yourself off from thinking about the whole alignment problem, whether or not you expect to work on an already-existing project. But what you're saying makes sense too. I'm curious though if there's something you were thinking about recently that just strikes you as fun, or like it's in the back of your mind a bit, even if you're not trying to think about it for some purpose."A: "Hm... Oh, I saw this video of an octopus doing a really weird swirly thing. Here, let me pull it up on my phone."T: "Weird! Maybe it's cleaning itself, like a cat licking its fur? But it doesn't look like it's actually contacting itself that much."A: "I thought it might be a signaling display, like a mating dance, or for scaring off predators by looking like a big coordinated army. Like how humans might have scared off predators and scavenging competitors in the ancestral environment by singing and dancing in unison."T: "A plausible hypothesis. Though it wouldn't be getting the benefit of being big, like a spread out group of humans."A: "Yeah. Anyway yeah I'm really into animal behavior. Haven't been thinking about that stuff recently though because I've been trying to start learning ML."T: "Ah, hm, uh... I'm probably maybe imagining things, but something about that is a bit worrying to me. It could make sense, consequentialist backchaining can be good, and diving in deep can be good, and while a lot of that research doesn't seem to me like a very hopeworthy approach, some well-informed people do. And I'm not saying not to do that stuff. But there's something that worries me about having your little curiosities squashed by the backchained goals. Like, I think there's something really good about just doing what's actually interesting to you, and I think it would be bad if you were to avoid putting a lot of energy into stuff that's caught your attention in a deep way, because that would tend to sacrifice a lot of important stuff that happens when you're exploring something out of a natural urge to investigate."A: "That took a bit of a turn. I'm not sure I know what you mean. You're saying I should just follow my passion, and not try to work towards some specific goal?"T: "No, that's not it. More like, when I see someone coming to this social cluster concerned with existential risk and so on, I worry that they're going to get their mind eaten. Or, I worry that they'll think they're being told to throw their mind away. I'm trying to say, don't throw your mind away."A: "I... don't think I'm being told to t...