Setting up Open library locally for development
Bharat Kalluri / 2021-07-23
This note will be in a form of questions and answers. I'll try to answer whatever questions I had when I was starting out. Let's begin!
How do I set up Open library and run it locally?
Open library can be run locally by using docker and docker-compsose. Make sure you have them installed and ready to go.
Clone the repository
git clone git@github.com:internetarchive/openlibrary.git
change directory to openlibrary
cd openlibrary
Run docker-compose build
docker-compose build
This takes around 5 minutes on my MacBook Pro 2015. It downloads all the required images to run the system.
Finally run
docker-compose up
to run all the services needed for docker. You should be seeing a lot of logs in the terminal. Give it 5 minutes and it should also pull down some books as seed data. (even if this fails, thats okay. we can pull in sample data later on)
Now open http://localhost:8080
to visit open library locally. To login, use
openlibrary@example.com
as username and admin123
as password. You should now be able to edit books, add books etc.
This will give you admin privileges.
Note that everything I just described is already neatly structured as a part of Open library documentation
All commands below should be run inside the
openlibrary
folder
I want to only look at the database logs/web logs. How do I do that?
Run docker-compose logs web
for web logs. Similarly, run docker-compose logs db
for db logs.
How do I look into the postgres schema and run queries in postgres?
First up, a lot of openlibrary works of infobase and all writes happen to infobase and all reads happen from infobase. I'll probably have another writeup on that. There will be no table in the schema called books, editions etc. That is by design. Ratings and bookshelves are implemented using postgres. So you can check those tables out.
You can jump into the shell of a container by doing
docker-compose exec db bash
then jump into psql
psql --user postgres -d openlibrary
Now, you can check out the relations by /dt
, run postgres queries etc.
How can I import a specific book into my local instance so that I can debug what is going on?
Let us say we want to "Digital fortress" locally. Here is the link in openlibrary : https://openlibrary.org/books/OL17948330M/Digital_Fortress
Notice the ID after the /books/
. If it ends with an M
, then it is an identifier for book. If it ends with a W
,
then it's an Identifier for a work.
Now let us jump into the shell of the web server
docker-compose exec web bash
run a script
./scripts/copydocs.py /books/OL17948330M --recursive
This should import the work, book and the author. After this, search for digital fortress
in the locally running open
library instance, and it should show up!
Why is it that the reading log is not working on my local OL instance?
That is an issue in the pipeline, here is a fix for now
Cannot edit OL books from local instance, getting recaptcha errors. How can I fix this?
This is also a documented issue. Currently, the solution is to delete all volumes and reset state, restart docker and then log in with the credentials mentioned above.
Happy hacking!