I built an open library for psychedelic and consciousness research that pulls from over a dozen academic APIs every 20 minutes, dedupes them, organizes by relevance, and synthesizes evidence.
I just finished my master's in Psychedelics and Consciousness Studies, and built the library I wanted while studying. I started it over a year ago, but put it down until my practicum advisor suggested I dust it off and go hard on it during my last term.
As of this writing, there are over 35,700 papers by over 82,200 authors organized into 37 topics, growing daily. There's a 2D map of the whole corpus and evidence syntheses on the topic pages, plus the ability to generate your own syntheses with any prompt.
It's built with Rails 8.1, uses one Postgres (for full-text search, vectors, and job queue), and runs on a $24/mo DigitalOcean droplet that I just upgraded from the $12 tier and deploy to with Kamal. Total LLM spend so far is ~$28 in DeepSeek-V4-Flash, trending down after a big one-time backfill push.
Relevance was the hardest part to get right by far. Ensuring we keep papers on LSD the psychedelic vs Lumpy Skin Disease, or Ketamine for depression vs anesthesia in cats was no small feat. The design ended up being a cheap keyword prefilter, then an LLM rubric that accepts or rejects papers. More on that here: https://consciousnesslibrary.org/docs/article-pipeline.html
It's free, no ads, no signup to read anything, and a registered 501(c)(3) I fund myself for now. I'd really like to know what breaks or any other feedback, and would love to answer any questions about it!
So freakin' cool. Overjoyed by your efforts here. I notice when I search for certain different words like "brahmavihara" or just "vihara" sometimes I get different results due to slight differences in the titles of papers. But that might be a feature and not a bug [if there were many papers, for example].
Technical question: Is Gemini not as good at rifling through paper citations? I would think Google scholar would be a moat of sorts for Goog, but maybe every LLM has roughly equal access to it now?
Research question: Did you find any mind-blowing results with your awesome new tool?
Development question: Do you plan on extending this beyond just a search apparatus? I would personally love to have a "factoid list" that actually cites interesting factoids from each article and can link to the source document -- although that is probably a "don't bite off more than you can chew" juncture and what you have is already excellent. But a thought. In case you're interested in expanding on it.
Hey, thanks so much for the kind words and taking a look!
Search observation: What you're noticing with search is the currently intended behavior, but you raise an interesting point. I'm using tsvector for Postgres full-text search over title, abstract, etc. with prefix matching, specifically in English. The stemmer doesn't decompose Sanskrit so "brahmavihara" and "vihara" are separate tokens matching legitimately separate papers. Then it falls back to the embedding geometry to search by meaning instead of strings.
That said, there's enough Sanskrit going on in contemplative papers that a synonym/variant layer for Pali and Sanskrit vocabulary is a pretty good idea. There's a lot of German and Portugese in the corpus as well, so upgrading the search to account for all these languages is a great feature idea.
Tech question: There's actually zero LLM involvement at all regarding citations or references in this project. That all comes from OpenAlex API. LLMs only do relevance gating, study detail abstraction, and evidence synthesis summaries.
Google has a moat in the sense they have no API and don't license anything to anyone, so nobody builds on it and the open ecosystem of sources I'm using is the same as what AI products use. Gemini is trained on the same sources this library uses (and recently, this library itself) and the Google Scholar vault is in my opinion, pretty useless given the open ecosystem available.
Research question: the most mind-blowing results I've discovered are in an unexpected direction - it is wild how thin and low-confidence the evidence is for basically any claim regarding psychedelics, due primarily to functional unblinding and low sample size and diversity. The hype cycle is way beyond science can keep grounded.
Here's an example (disclaimer, it's my own published research!): A recent study reported an effect size for mebufotenin exceeding the theoretical maximum for antidepressants, implying methodological issues account for the results. https://consciousnesslibrary.org/research/52248-blinding-int...
Another "mind-blowing" observation is the sheer volume of consciousness work outside the psychedelic hype cycle. A full half of the corpus is on meditation or philosophy of mind, not related to psychedelics. And we still are so far from understanding the "hard problem.
Development question: I have plans to extend this indefinitely, and your factoid idea is interesting. The "study at a glance" might have most of what you're asking about already on each article, but taking that a step further, it could be cool to have a cross-corpus browsable feed of those. I'll also be integrating data from a project by Josie Kins (maker of https://effectindex.com) on psychedelic substance topics in particular for dose info and potentially trip reports, but that's a little orthogonal to what you're suggesting.
Let me know if you mean something more about factoids than the "study at a glance" and the potential to have those be aggregated in a browsable way and I'll see what I can think of.
I haven't bit off more than I can chew yet, my plan is to work on it until that's the case OR there's just nothing left to do in the scope of the project.
Interesting that there are masters programs for psychedelics now, but I guess it makes sense given the current hype trends.
Curious why you chose Rails for this? I'd think Python tooling would have been a better fit given the data-heavy pipeline doc you linked.
Also, I saw the https://consciousnesslibrary.org/ai-ethics link in the footer. Lots of ambitious goals there. Do you really think it's possible to run models in-house that can do what you're having deepseek do right now, especially with "renewable energy wherever you can source it?"
Thanks for taking a look! Yeah, these programs are very new and pretty uncommon (for now).
As for why I chose Rails - I just know Rails really well. I've been using it since 2012 or so, and it's basically muscle memory now. The most intensive parts of the app really would lend themselves to Python tooling and I did end up hand-building a lot of the data engineering stuff that there for sure exist plugins for in Python, but the trade off of using what I know well outweighed that by a lot. Plus the rest of it are perfectly CRUD-able server-rendered pages. Pretty Railsy aside from the data eng stuff.
I also started this before AI coding harnesses were available. I shelved it for a long time after the initial few rounds, but the work I did in the beginning was substantial enough to not just start over. If I had Claude Code with Fable when beginning, it might look a lot different. But I do like knowing everything about what I built and understanding every line.
On your AI ethics question... I do think it's possible, eventually. That's why I hedged a bunch on it being a long term goal. If open weight models gain steam and hardware acquisition becomes manageable, it would be a dream to run a self-hosted self-trained model in my house (which is powered by solar, so the renewable thing is real). It is especially realistic if some funding comes through for hardware.
Cool, that makes sense. Did you use AI to help build it when you picked it back up?
Also, any plans to include trip/experience reports? I'm particularly curious about experiences regarding meditation and dreams and that side of what you have in the project. I know there are places like Erowid and whatnot for psychedelic reports but haven't heard of something similar for the rest of what you have in here.
Yeah I’ve used mainly Claude code and recently kimi with pi for dev. Honestly the vast majority of features and improvements happened in the last few months with AI enablement. The Fable fake-outs really pushed me to optimize my time with it.
On experience reports - I decided to keep it purely academic at first but will soon be working on integrating data from a tool by Josie Kins (the maker of https://effectindex.com).
I also don’t know of anywhere that aggregates reports from meditation and dreams etc, but that’s a really interesting idea. Maybe this can be the place!
Thanks! One minor note, it's got pretty much all accessible research on psychedelics, but it's not exactly specific to psychedelics. About half the corpus is research on meditation/contemplative traditions and philosophy of mind/consciousness theory. They all go pretty well together though!
I built an open library for psychedelic and consciousness research that pulls from over a dozen academic APIs every 20 minutes, dedupes them, organizes by relevance, and synthesizes evidence.
I just finished my master's in Psychedelics and Consciousness Studies, and built the library I wanted while studying. I started it over a year ago, but put it down until my practicum advisor suggested I dust it off and go hard on it during my last term.
As of this writing, there are over 35,700 papers by over 82,200 authors organized into 37 topics, growing daily. There's a 2D map of the whole corpus and evidence syntheses on the topic pages, plus the ability to generate your own syntheses with any prompt.
It's built with Rails 8.1, uses one Postgres (for full-text search, vectors, and job queue), and runs on a $24/mo DigitalOcean droplet that I just upgraded from the $12 tier and deploy to with Kamal. Total LLM spend so far is ~$28 in DeepSeek-V4-Flash, trending down after a big one-time backfill push.
Relevance was the hardest part to get right by far. Ensuring we keep papers on LSD the psychedelic vs Lumpy Skin Disease, or Ketamine for depression vs anesthesia in cats was no small feat. The design ended up being a cheap keyword prefilter, then an LLM rubric that accepts or rejects papers. More on that here: https://consciousnesslibrary.org/docs/article-pipeline.html
It's free, no ads, no signup to read anything, and a registered 501(c)(3) I fund myself for now. I'd really like to know what breaks or any other feedback, and would love to answer any questions about it!
So freakin' cool. Overjoyed by your efforts here. I notice when I search for certain different words like "brahmavihara" or just "vihara" sometimes I get different results due to slight differences in the titles of papers. But that might be a feature and not a bug [if there were many papers, for example].
Technical question: Is Gemini not as good at rifling through paper citations? I would think Google scholar would be a moat of sorts for Goog, but maybe every LLM has roughly equal access to it now?
Research question: Did you find any mind-blowing results with your awesome new tool?
Development question: Do you plan on extending this beyond just a search apparatus? I would personally love to have a "factoid list" that actually cites interesting factoids from each article and can link to the source document -- although that is probably a "don't bite off more than you can chew" juncture and what you have is already excellent. But a thought. In case you're interested in expanding on it.
Hey, thanks so much for the kind words and taking a look!
Search observation: What you're noticing with search is the currently intended behavior, but you raise an interesting point. I'm using tsvector for Postgres full-text search over title, abstract, etc. with prefix matching, specifically in English. The stemmer doesn't decompose Sanskrit so "brahmavihara" and "vihara" are separate tokens matching legitimately separate papers. Then it falls back to the embedding geometry to search by meaning instead of strings.
That said, there's enough Sanskrit going on in contemplative papers that a synonym/variant layer for Pali and Sanskrit vocabulary is a pretty good idea. There's a lot of German and Portugese in the corpus as well, so upgrading the search to account for all these languages is a great feature idea.
Tech question: There's actually zero LLM involvement at all regarding citations or references in this project. That all comes from OpenAlex API. LLMs only do relevance gating, study detail abstraction, and evidence synthesis summaries.
Google has a moat in the sense they have no API and don't license anything to anyone, so nobody builds on it and the open ecosystem of sources I'm using is the same as what AI products use. Gemini is trained on the same sources this library uses (and recently, this library itself) and the Google Scholar vault is in my opinion, pretty useless given the open ecosystem available.
Research question: the most mind-blowing results I've discovered are in an unexpected direction - it is wild how thin and low-confidence the evidence is for basically any claim regarding psychedelics, due primarily to functional unblinding and low sample size and diversity. The hype cycle is way beyond science can keep grounded.
Here's an example (disclaimer, it's my own published research!): A recent study reported an effect size for mebufotenin exceeding the theoretical maximum for antidepressants, implying methodological issues account for the results. https://consciousnesslibrary.org/research/52248-blinding-int...
Another "mind-blowing" observation is the sheer volume of consciousness work outside the psychedelic hype cycle. A full half of the corpus is on meditation or philosophy of mind, not related to psychedelics. And we still are so far from understanding the "hard problem.
Development question: I have plans to extend this indefinitely, and your factoid idea is interesting. The "study at a glance" might have most of what you're asking about already on each article, but taking that a step further, it could be cool to have a cross-corpus browsable feed of those. I'll also be integrating data from a project by Josie Kins (maker of https://effectindex.com) on psychedelic substance topics in particular for dose info and potentially trip reports, but that's a little orthogonal to what you're suggesting.
Let me know if you mean something more about factoids than the "study at a glance" and the potential to have those be aggregated in a browsable way and I'll see what I can think of.
I haven't bit off more than I can chew yet, my plan is to work on it until that's the case OR there's just nothing left to do in the scope of the project.
Thanks a lot for your comment!
Well done! It is amazing that society can kill itself and at the same time do not fully understand how the brain (that could triger that) works.
Thank you! I couldn't agree more. Big questions to answer, and your framing is exactly why I went into this.
Interesting that there are masters programs for psychedelics now, but I guess it makes sense given the current hype trends.
Curious why you chose Rails for this? I'd think Python tooling would have been a better fit given the data-heavy pipeline doc you linked.
Also, I saw the https://consciousnesslibrary.org/ai-ethics link in the footer. Lots of ambitious goals there. Do you really think it's possible to run models in-house that can do what you're having deepseek do right now, especially with "renewable energy wherever you can source it?"
Thanks for taking a look! Yeah, these programs are very new and pretty uncommon (for now).
As for why I chose Rails - I just know Rails really well. I've been using it since 2012 or so, and it's basically muscle memory now. The most intensive parts of the app really would lend themselves to Python tooling and I did end up hand-building a lot of the data engineering stuff that there for sure exist plugins for in Python, but the trade off of using what I know well outweighed that by a lot. Plus the rest of it are perfectly CRUD-able server-rendered pages. Pretty Railsy aside from the data eng stuff.
I also started this before AI coding harnesses were available. I shelved it for a long time after the initial few rounds, but the work I did in the beginning was substantial enough to not just start over. If I had Claude Code with Fable when beginning, it might look a lot different. But I do like knowing everything about what I built and understanding every line.
On your AI ethics question... I do think it's possible, eventually. That's why I hedged a bunch on it being a long term goal. If open weight models gain steam and hardware acquisition becomes manageable, it would be a dream to run a self-hosted self-trained model in my house (which is powered by solar, so the renewable thing is real). It is especially realistic if some funding comes through for hardware.
Cool, that makes sense. Did you use AI to help build it when you picked it back up?
Also, any plans to include trip/experience reports? I'm particularly curious about experiences regarding meditation and dreams and that side of what you have in the project. I know there are places like Erowid and whatnot for psychedelic reports but haven't heard of something similar for the rest of what you have in here.
Yeah I’ve used mainly Claude code and recently kimi with pi for dev. Honestly the vast majority of features and improvements happened in the last few months with AI enablement. The Fable fake-outs really pushed me to optimize my time with it.
On experience reports - I decided to keep it purely academic at first but will soon be working on integrating data from a tool by Josie Kins (the maker of https://effectindex.com).
I also don’t know of anywhere that aggregates reports from meditation and dreams etc, but that’s a really interesting idea. Maybe this can be the place!
This looks solid, great to have a site like this focused specifically on psychedelic vertical
Thanks! One minor note, it's got pretty much all accessible research on psychedelics, but it's not exactly specific to psychedelics. About half the corpus is research on meditation/contemplative traditions and philosophy of mind/consciousness theory. They all go pretty well together though!
Psychedelics are the best. I am super excited for research in the area, the "war on drugs" is such a terrible thing.