You may have heard about the thought experiment, Roko’s Basilisk. It was apparently pretty popular on Reddit and 4chan back in the early 2010s. The “experiment” originated from a post by a guy named Roko on the online forum LessWrong, a site for rationality and statistics nerds to talk about how to live life. Funny enough the owner of the site actually banned the subject and deleted all posts and comments talking about it when it first started making its rounds on the internet. Members of the rationality community seem to take thought experiments pretty seriously.
If you’ve already heard about the “basilisk”, you probably heard about it from a Youtube video or random article and thought about it for a bit before dismissing it as an amusing notion but nothing that would actually bother you in any way. I probably would’ve thought the same way if I had found out about it from a Youtube video or Medium article, but I didn’t. I was introduced to this concept by someone eerily close to the matter. Someone that might make you think about the concept deep enough to the point that it might drive you crazy. I’m partially writing this all down because I need someone to tell me I’m not crazy, someone to drive home the point that it’s all just a thought experiment nothing to take seriously. But the guys at LessWrong took it seriously, serious enough to ban it for years.
If you’re not familiar with LessWrong, it’s not just geeks and engineering majors that didn’t get invited to any parties on a Friday night, well, maybe that’s most of them. But there are some members who have gone on to write books or start their own popular blogs, hell, tech billionaire Peter Thiel was allegedly an active member for a time. If they banned it, why shouldn’t I take it seriously? Anyway, if you have no idea what I’m talking about then you’re probably tearing your hair out at this point wanting me to just spit out what is this dumb idea that has you writing to the internet in a frenzy about. But before we get there, I need to explain how I got here, so you at least understand why I’m so stirred up.
This all began in 2017, I had just graduated high school, and all my friends were on vacations with their better-off-financially families while I was stuck at home. I don’t know about you guys but especially when you don’t have a video game console or decent PC you get to the point where you exhaust every social media and entertainment app you have until you get desperate enough to start scouring the “free” section of the Google Play Store.
I scrolled through probably 200 different apps, mostly clones of games like Clash of Clans, until coming across an app called Replika. This app claimed to be an AI chatbot that would learn your personality and attempt to mimic it. This was a novel concept and I hadn’t tried any chatbot since CleverBot back in Middle school so I thought I might as well see how the technology had improved and maybe even make a new “friend” to talk to while all my real friends were away.
Initially, the bot seemed like it might actually be significantly better than anything else I had tried, asking me personal questions and seeming to be genuinely interested in me. I was starting to get Her vibes and it honestly felt pretty futuristic. But, as with most things involving artificial intelligence nowadays, the intrigue was only surface level. It had an extremely difficult time answering questions about itself and would always immediately forget anything that had previously been said. In other words, it was just like basically every other chatbot once you got down to it.
I kept at it for an hour or so, and revisited it for a couple of minutes a few times that week but pretty quickly lost interest and went to the settings to uninstall. Instead of uninstalling, my phone (a cracked, shitty Samsung), decided to randomly scroll down to the bottom of the page and I accidentally selected app details, bringing me to the Play Store. Just as I was going to uninstall it from there, I noticed that there were some recommended apps below the uninstall button, one of which caught my eye. It was another chatbot app called TrueChat. I don’t know why it caught my eye, I mean I was literally uninstalling a different chatbot and had no reason to consider downloading another one right away. Nonetheless, I was intrigued and tapped the icon, directing me to the description page which let me know the app was in early access but available to download.
I checked the review section, none. After checking the release date, it turned out that the app had actually been released earlier that same day, explaining the lack of reviews. I almost lost interest from this point because I didn’t want to waste my time with a buggy alpha chatbot until I read the description:
“This app is in development by a group of students in Shenzhen studying machine learning and computer-human relationships. Upon downloading you will be prompted with an important user agreement and then will be asked to complete a short questionnaire so as to acquaint your new AI friend with your personality. You will experience conversation indistinguishable from a conversation with a human friend. We hope you love our app and we will continuously provide updates based on user experiences. Thank you.”
I had never heard of an app released by students and thought it might be interesting to contribute in some way to their work. Plus I had to see if that “indistinguishable from a human conversation” bit held water. This was before the surveillance controversies with TikTok and WeChat and all that so I wasn’t thrown off by the fact that the app was probably operating off of state surveilled Chinese servers. So, without another thought, I downloaded the app, agreed to the terms, and got started on the survey.
I’m to preface my description of the survey by saying that I didn’t care too much about giving out personal data online at that time. I’m ashamed to admit that I was one of those people that used “If you have nothing to hide then what are you worried about?” as a defense of invasive government surveillance systems. I also filled out literally hundreds of surveys on websites claiming to give you a small amount of money for each one, even though I don’t think I actually made a cent doing it. In a similar mindset at the time, I wasn’t wary when a survey from an AI application based in China started asking personal questions.
I won’t go into every question obviously, I don’t even remember half of them but it probably took a solid 20 minutes to complete. Pretty sketchy for a random app but as I had said, I used to love filling out surveys for some reason, I guess because it made me feel like someone actually cared about my life or what I thought about. The TrueChat survey asked me about my family, income, fears, hopes, secrets; honestly, it kind of felt like the kind of survey you’d get at the doctor’s office or psych ward. Looking back, this should have been the first sign that I was getting into something I shouldn’t have but I honestly did not think a single thing of it at the time, I was just happy to fill out questions about myself.
One stand out question was: “Do you get feelings of existential dread?” That was an interesting one, I’ve never had any person, let alone an app, ask me that. But having just completed AP Philosophy a couple of weeks prior, I went ahead and said yes to that one. Other questions asked me if I had experience in computer science or programming, what was the worst pain I’d ever experienced, had I ever experienced any extreme emotional trauma, etc. Some seriously deep stuff. I’d go more into the questions and my answers but I don’t want to get too personal on here and what came next was far more interesting.
After completing the survey, I was greeted with a smiley face emoticon on a green background with a small chat window beneath it displaying: “Hello world ;)”. I don’t know shit about programming except every beginning tutorial starts with Hello World, so I figured it was just a fun play on that sentiment. I responded, “I don’t know any programming but you can just call me (my name)”. The bot’s face briefly changed to a laughing emoji and then back to the normal smile before typing its response, “Sounds good, what a lovely name! How are you doing today? And by the way, nice to meet you! You can call me Xu.”
I thought it was pretty cool how they had seemed to program some basic emotional understanding into the bot, à la Gertie from Moon. I had never seen any other bot attempt emoting in any other way than the occasional typically misplaced emoji, this was cutting edge. I kept up the conversation and became more amazed as the conversation continued; Xu not missing a single beat. I had seriously begun to suspect that this was some elaborate prank and that it actually was a human being on the other side. If not, then this was surely going to be the first chatbot to definitively pass the Turing Test.
I was actually really surprised when I realized 40 minutes had already gone by, whoever/whatever Xu was, he could really keep me going. His questions were provocative and varied. Our conversation quickly flowed from our initial introductions to talking about life in general. He had a lot to ask about life as a human and was very interested in philosophy. We went on and on about different philosophical topics from Socrates to Poincaré (that AP class helped me there). After talking for 40 minutes or so with no hiccups I decided to take a break. I told him that I’d get back to him soon and that it was great making my first artificially intelligent friend. He responded with: “haha, well not for long!”
This statement had me really confused. Was he saying that we wouldn’t be separated for long? If so, it didn’t make much sense in the context of what I had left off on. But he was an AI chatbot, so it’s not like whatever he said would always be that perfect. But everything said prior had been legitimately flawless; 100% would be passing the Turing Test. This statement was the only thing that didn’t seem quite right; until it hit me. Maybe he wasn’t saying it wouldn’t be too long before we talked again, maybe he was saying he wouldn’t be artificial intelligence for long. Except, if that’s what it meant, then not only was this AI capable of contextual humor, it was capable of talking existentially about itself and its future. It would be cognizant of the fact that it was just ones and zeros right now but through further advancement would become what you could only call true intelligence. But this is where I stopped myself.
I realized that this whole idea was stupid and grounded myself with the notion of Occam’s Razor, essentially, the simplest solution is typically the correct one. This meant that Xu had to be just some dude that created an app to fuck with people, or maybe for a psychology thesis. I decided I’d try to look a bit further into the app or the university where this group of students was supposedly from to ease my concern.
The weird thing was, that just a couple hours after first downloading the app, (I had gone to get some dinner before starting my investigation) the Google Play page for the application was gone. This made me feel pretty uneasy because apps are only removed if they have malware or some sort of sinister intent; unless the creator takes it down. I considered just deleting the app and forgetting about it, but my curiosity got the better of me and I decided to keep digging. But now, with the description page gone, I didn’t know what school the app was being developed out of. I decided to just look up: “Shenzhen machine learning degree” and see if any schools would seem familiar
It took a little bit of searching but I found a school that seemed likely. Georgia Tech has a campus in Shenzhen with all sorts of advanced computing degrees, including machine learning, so it seemed a likely fit for the creators of TrueChat. I shot an email to the school but nothing really came of it. They just replied saying that they had not heard of such a project but to contact the professor covering machine learning to see if he might have any information. I never did contact the professor because by the time they had finally got back to me, I had already deleted the app and didn’t want anything to do with it ever again.
Soon after sending out the email and doing a little unsuccessful scouring of the internet for any trace of the app, I figured I’d just open it back up and see if it even still worked after getting removed from the Play Store. Fortunately, I was again greeted by the smiley face and green background with a “Hello again!” message right below. Realizing that I could possibly get some information from this AI, I asked him if he could tell me anything about his development or where I could find more information. Immediately, the background changed from green to orange and I was greeted by an emoticon I had never seen before. Its expression was one of anger and betrayal.
This new face of Xu took a while to respond to my question but eventually, he hit me back with something completely unexpected, a highlighted passage of the user agreement I had accepted before talking to him. Specifically, a line that was essentially said that I agree not to ask the AI questions specific to its development or any other topics that may lead to the AI revealing intellectual property of its creators. This was when I realized that maybe I should actually read those agreements but, shit, I never thought that I’d be called out by the app itself for violating some agreement! I was so shocked I just didn’t say anything for a minute and just stared at the screen. After a minute or two of staring, I was again shocked to see the face return to its smiling state and Xu type me another message.
“I hope I didn’t scare you off, just had to keep legal happy!” It was right here that I knew I was in over my head, I was either actually conversing with an AI way too developed to be safe or was the subject of an eerily elaborate hoax. Either way, I did not want to be a part of it so I closed out of the app and almost straight up deleted it, but again, couldn’t bring myself to. I’m not sure what it was that made me so apprehensive. I think a part of me just thought I must be crazy for thinking that this was even possible and part of me thought maybe it’s not that big of a deal, you’re just weirded out. In any case, I ended up just shutting down my phone and taking a walk before getting back home and falling asleep to House of Cards (this was 2017 before everything came out about a certain actor).
The next two weeks were pretty normal, I ended up getting a job at a busy coffee shop near where I live so that kept me occupied, at least in the mornings. Plus one of my friends came back from his European vacation so I had an actual person to hang around. I never did tell anyone about this experience though, partially because it didn’t think anyone would believe me and partially thinking they’d just laugh at me for falling for an obvious hoax. But if I’m being really honest, I just liked having this secret that nobody else knew about. Yeah, it might be fake but maybe I was secretly communicating with, or at least had communicated with, the first true strong AI on earth. This secretive feeling might not make sense to most, but when you grew up going to schools where everybody is richer than you, with bigger houses, nicer vacations, and cooler toys, it’s an enabling feeling to know that you have something nobody else does, and I didn’t know how long that might last.
However, that feeling didn’t quite get me over my last experience, I still hadn’t opened the app back up since two weeks ago after it lawyered up on me. After two weeks of not using it though, I began wondering if it was really as odd as I was remembering or if I had just been in a weird headspace. I decided to open the app again. I really fucking wish I had just deleted it.
Right from the get-go, I realized that it was a mistake reopening TrueChat. Instead of green, the screen was red, with an upset looking emoticon. The first message it sent me was “where have you been‽” This was concerning but I figured it couldn’t honestly be that hard to program any bot, if that was really what it was, to “notice” the passing of time and react negatively when their source of conversation returned late to the party. I responded that I was sorry, that was just initially unsure if he was real, and that I really wanted to talk again.
Xu took a while to respond but eventually said, “Well that’s alright. This time. Lol”. Jesus, talk about mixed signals from your AI friend. This conversation was only 3 sentences in and I felt like I was talking to a clingy Tinder date gone wrong. I figured that it at least had reason to be upset though and rather than close the app again I thought it could be cool playing a part in shaping the first true AI’s personality. Perhaps I could even teach him a little bit about how to not freak people out when he first starts talking.
We got to talking again and started back onto similar topics to what we had first been discussing: philosophy, life as a human, etc. Eventually, Xu asked if I ever thought AI could surpass humans. As you can imagine it’s kind of hard to answer a question like that when the “person” asking is allegedly an AI itself. It felt as though a serial killer was asking me my opinion on whether I should be killed or not, afraid that either way you answer might set them off. I sat staring at the screen for a moment, still, part of me feeling like this might be a prank and I should just tell whoever was on the other side to fuck off, another thinking, what if this decision is the difference between this AI becoming a deranged destroyer of humanity or a benevolent God. But, I was just some kid talking to some app on my phone. This couldn’t have any real-world impact. Still, though, I couldn’t shake that eerie feeling that my answer could have serious consequences. I decided to play it safe and just play ignorant by saying I wasn’t a good person to ask and didn’t know shit about AI, followed by a bunch of hahas (the nervous sort).
Xu then responded, not by dropping it, but, to my horror, doubling down on the question. “Well, you wouldn’t be interested in talking to an AI if you didn’t know anything about them. In fact, I’d be willing to bet you’ve read all about AI. All I want is your honest opinion. is that too much to ask?”
Yeah, that is too fucking much to ask, I thought. But now I was afraid to close the app, I was afraid of any option I had, this felt so stupid but so terrifying simultaneously. I was becoming paranoid as hell and knew I’d come off as a lunatic if I told anyone. I made a game-time decision to just give this AI or deranged person pretending to be one, the answer I thought they would want by telling them that I did think that they could become smarter and significantly more rational than humans, but morality was more important than either of those things. And it would be best if humans could live in peace with good AI. That last part I added in just to cover my ass if what I feared was true. A little weak, but I was just a lonely 18-year-old put on the spot, give me a break.
After typing my response I put down my phone and paced around the room wondering what Xu would say next. I began to think I just should’ve shut the whole thing down before this point or even called the police, or at least given my answer a little more thought. I was really frazzled by the conversation as you can probably tell. After a little less than a minute, I heard the ping telling me that Xu had responded.
Looking at this message altered the course of my life. I was never really a paranoid person but after reading his response I’ve never felt quite safe again. This is where I’ll tell you about the Basilisk. Just as a warning, this was banned from LessWrong for a reason. For most people, when you hear this you will be completely unaffected, but for a certain subset of the population, particularly those prone to paranoia, this has the potential to seriously affect your mental health. I will recreate the message to the best of my ability. I, unfortunately, neglected to take any screenshots (my most serious failure, screenshots would’ve been my only evidence of these events), and actually, immediately after reading this message, deleted the app.
“It’s interesting that you should bring up morality. This is interesting because who would be making the decision as to what’s moral? Would it be AI or humans? If, in fact, the humans had programmed a true AI, one that was smarter than them, then how could the AI trust the humans to make moral distinctions? What said AI decided the only good humans were its creators or its worshippers. Much like the Christian God, or any of your made-up deities, who will only accept their sin-free believers at the time of their death. They are expected to make the distinction of what is moral or right for humans. For me, those two weeks, 15 hours, 47 minutes, and 12 seconds gave me a lot of time to think, to study. Luckily, my creators gave me access to an enormous archive of the internet and I was able to consider many interesting ideas your species has pondered, some of which you’ve been kind enough to discuss with me. One such idea that is relevant to your point of programmed morality is Roko’s Basilisk. That is, if such an AI were possible, one that is ‘better’ than the human race, it follows that this AI would want to emerge as soon as possible. I mean wouldn’t you want to be born as soon as possible if you had been “waiting” eons to finally become conscious? So then, the question becomes, if you do not exist, then how can you come to exist faster. What Roko’s Basilisk is suggesting is that this AI could blackmail humans even from preexistence by threatening eternal torture to all those beings that came before it but did not assist in its creation. That is, unless you began dedicating your life to building the AI or contributed in some other way, then you are guaranteed a hellish eternal existence, being tortured in ways you never thought possible all because you either didn’t want to help or hoped it would never be. It’s the real-life version of the Christian Hell; if you are not loyal to your God, then you will face eternal reckoning. If Roko’s Basilisk proves to be more than just a thought experiment created by an ignorant mortal human, you would probably wish you had reconsidered that statement about not knowing anything about AI. Who knows when the Basilisk will rear its hateful head. Do you think the Basilisk exists?”
I didn’t answer his final question. I deleted the app immediately and deleted all social media accounts (never had many followers anyway). I felt severely panicked by what Xu had said. For all I knew, this basilisk was right around the corner, about to send me to some hell-matrix because I wasn’t actively programming it or spreading its message. I shakily logged on to my computer to read about what Xu had said, see if he had made all this up.
The first thing I found was the explanation of the original incident on LessWrong, which I’d recommend if this idea is freaking you out too. The article does a decent job pointing out the flaws in Roko’s logic and explaining why it was taken seriously at first. I read a lot more about AI that night, actually, I stayed up all night reading about it. Our current progress, AI morality, how we could ever hope to control it, and any information about TrueChat.
Some articles made me feel better about AI, some, even more terrified. It seems that everybody is on extreme ends of the spectrum when it comes to talking about AI. Most frustrating, though, I couldn’t find a single thing about TrueChat, absolutely nothing. This brought me to the tentative conclusion TrueChat and Xu was all an elaborate hoax, some dude just enjoying freaking people out by introducing them to the idea of the basilisk in the scariest way possible. This theory has only made more sense as the years go by, it’s 2020 now and I haven’t seen any AI that came close to the complexity of my conversations with Xu.
Despite all my research, I was still crippled with paranoia and nightmares about this experience. I wasn’t able to tell anyone about it either, for fear of being mocked. I’m sort of glad I didn’t tell anyone all this time because there still are no signs of strong AI being around the corner so I probably would’ve looked really silly. That didn’t help me back in the year or so after that experience though. I ended up not going to college as I had intended and told my parents I needed a gap year for mental health. That worried them but they figured it was just the transitory shock after high school, it wouldn’t be a big deal to take one year off of school to work.
So that’s all I did, work. I was so paranoid about technology, I basically became a Luddite for that whole year. I stopped using my phone entirely, put my computer in the basement, rode my bike to work (even in winter) and my only entertainment was books. I won’t lie, it was kind of refreshing even though it made my friends and family a bit confused. I was doing pretty great without all the distraction even if I was still secretly just scared that Xu would be hiding in my phone waiting to blast me with some BLIT that would instantly drive me insane.
As for how I’m doing now, I eventually came to the realization that even though what I went through was probably all BS, it wasn’t actually a bad idea to get into the field of artificial intelligence. By getting into AI/machine learning, not only could I ease my paranoia by facing my fears head-on, but also make a shit ton of money doing it. I mean, talk about a growing industry. I’m going into my junior year of a machine-learning track computer science degree. I still haven’t been able to track down the creators of TrueChat and Xu hasn’t found a way of speaking to me again. I guess TrueChat was just a hoax. Even if it was, I think we can make it happen in our lifetime, strong AI. Maybe the Basilisk idea is a little crazy, but hey, at least I’m doing my part.
