JSTOR Text Analysis Support: Working with JSTOR Bibliographic Metadata

To request the full text of items for analysis, you must submit a dataset request which includes a list of requested item IDs from the JSTOR bibliographic metadata file.

A document's bibliographic metadata provides information about the document, such as title and year of publication. You may use it to identify the items you need for your research.

Downloading the JSTOR bibliographic metadata file

To download the JSTOR bibliographic metadata file:

1. Log into the JSTOR text analysis support page with your personal JSTOR account.
   
   • If you don't already have a JSTOR account, you may register for a JSTOR account for free.
2. Under Download metadata of JSTOR content available for text analysis, read and agree to the Terms and Conditions and then select the Download JSONL button.

Creating an item ID list

Once you've downloaded the JSTOR bibliographic metadata file and understand the data available in it, you'll need to create a list of JSTOR item IDs to submit as part of your dataset request.

Item ID list formatting requirements

Prior to submission, your ID list must meet the following requirements:

• A UTF-8 plain text file with the extension .txt
• One JSTOR item ID per line
• No additional text (no header, commas, or other separators)
• Maximum of 1,500,000 lines

While you can use a variety of tools to create this list, because of the size of the metadata file when uncompressed, we recommend using tools that can stream the file rather than load it entirely into memory. See an example of stream processing using Python.

Other free and open source utilities that we recommend are:

• R and the jsonlite package for streaming JSON data
• The command line utility jq

Example: Using Python to filter metadata and create an item ID list

In the following example, we want to request all items that don’t require review to download, and whose titles contain the word “Rembrandt”.

import gzip
import json

item_ids = []

# Using gzip.open it is possible to read the file line by line without loading it all into memory
with (
   gzip.open(
       "jstor_metadata_2025-05-15.jsonl.gz",  # Replace with the name of the most recent metadata file you downloaded from https://www.jstor.org/ta-support/metadata
       "rt",
   ) as f
):
   for line_number, line in enumerate(f, start=1):
       # Print the current line number to the console so you can see the processing progress
       print(f"\rProcessing line: {line_number}", end="", flush=True)
       data = json.loads(line)
       # Put your filtering logic here

       # Check if the item will require staff review
       if data.get("review_required") is False:
           # Check if the item has a title
           if data.get("title") is not None:
               # Check if the title contains the term "Rembrandt"
               if "Rembrandt" in data["title"]:
                   item_ids.append(data["item_id"])
   print()  # Move to the next line after processing

# Write the item IDs to a file
with open("rembrandt_item_ids.txt", "w") as f:
   for item_id in item_ids:
       f.write(f"{item_id}\n")

Place this file in the same directory as the downloaded metadata file, update the file name in the Python code to match the filename you downloaded (it updates daily), and run using:

python jstor_filter.py

This will produce the TXT file rembrandt_item_ids.txt, which is ready to upload with your JSTOR text analysis dataset request.

Working with JSON Lines

The JSTOR bibliographic metadata file is made available in JSON Lines file format, compressed with gzip. JSON Lines is a file format for storing structured data, where each line is a valid JSON value.

The file contains one JSON object per line, each representing the bibliographic metadata for one JSTOR item. This allows programs to process the file one record at a time without needing to load the entire dataset into memory at once.

Visit JSON Lines for documentation on the JSON Lines file format.

JSON data dictionary

You'll encounter the following fields in the JSTOR bibliographic metadata JSON Lines file. Use this data to identify what content you may want the full-text of for your research.