Setting up Open library locally for development

This note will be in a form of questions and answers. I'll try to answer whatever questions I had when I was starting out. Let's begin!

How do I set up Open library and run it locally?

Open library can be run locally by using docker and docker-compsose. Make sure you have them installed and ready to go.

Clone the repository

git clone git@github.com:internetarchive/openlibrary.git

change directory to openlibrary

cd openlibrary

Run docker-compose build

docker-compose build

This takes around 5 minutes on my MacBook Pro 2015. It downloads all the required images to run the system.

Finally run

docker-compose up

to run all the services needed for docker. You should be seeing a lot of logs in the terminal. Give it 5 minutes and it should also pull down some books as seed data. (even if this fails, thats okay. we can pull in sample data later on)

Now open http://localhost:8080 to visit open library locally. To login, use openlibrary@example.com as username and admin123 as password. You should now be able to edit books, add books etc. This will give you admin privileges.

Note that everything I just described is already neatly structured as a part of Open library documentation

All commands below should be run inside the openlibrary folder

I want to only look at the database logs/web logs. How do I do that?

Run docker-compose logs web for web logs. Similarly, run docker-compose logs db for db logs.

How do I look into the postgres schema and run queries in postgres?

First up, a lot of openlibrary works of infobase and all writes happen to infobase and all reads happen from infobase. I'll probably have another writeup on that. There will be no table in the schema called books, editions etc. That is by design. Ratings and bookshelves are implemented using postgres. So you can check those tables out.

You can jump into the shell of a container by doing

docker-compose exec db bash

then jump into psql

psql --user postgres -d openlibrary

Now, you can check out the relations by /dt, run postgres queries etc.

How can I import a specific book into my local instance so that I can debug what is going on?

Let us say we want to "Digital fortress" locally. Here is the link in openlibrary : https://openlibrary.org/books/OL17948330M/Digital_Fortress Notice the ID after the /books/. If it ends with an M, then it is an identifier for book. If it ends with a W, then it's an Identifier for a work.

Now let us jump into the shell of the web server

docker-compose exec web bash

run a script

./scripts/copydocs.py /books/OL17948330M --recursive

This should import the work, book and the author. After this, search for digital fortress in the locally running open library instance, and it should show up!

Why is it that the reading log is not working on my local OL instance?

That is an issue in the pipeline, here is a fix for now

Cannot edit OL books from local instance, getting recaptcha errors. How can I fix this?

This is also a documented issue. Currently, the solution is to delete all volumes and reset state, restart docker and then log in with the credentials mentioned above.

Happy hacking!