The ENCODE REST API
Project overview | Using the portal | REST API | Data organization | Data submission | Encyclopedia
TERMS OF SERVICE
Users of the ENCODE portal may freely download, analyze and publish results from ENCODE with no restriction, commercial or otherwise. For more details see Data Use Policy
Use of the programmatic API is limited to 10 GET requests/sec from any single user, group, company or lab.
Abuse of this will result in denial of access by IP block>
Quick Links
Swagger Interactive API Documentation (opens external link)
Advanced Swagger Interactive API Documentation (opens external link)
Introduction
Direct interaction with the ENCODE DCC metadata database is typically done through scripts you write and execute on your own Mac, PC or server. Your scripts interact with the database through an industry-standard, Hypertext-Transfer-Protocol-(HTTP)-based, Representational-state-transfer (RESTful) application programming interface (API). Data objects exchanged with the server conform to the standard JavaScript Object Notation (JSON) format. JSON is a defined data interchange format that is used web-wide. If you’re familiar with JavaScript’s object literal notation or Python’s dictionary datatype, then JSON-formatted text will look familiar. You will almost certainly use libraries for your language of choice to handle the network connection and parse the objects returned (requests and json for Python, for example). We have written example scripts you can look at here.
Quickstart examples
While getting objects from the database can often be done in just a few lines of code, here are some quick ways of testing your connection and exploring the ENCODE data objects without writing any code.
Prerequisites
- A JSON pretty-printer plugin for your web browser, such as JSONView (for Chrome or Firefox) or json-lite (for Safari)
- The
curl
command, which ships with Mac OSX and all major LINUX/UNIX implementations. If you don't havecurl
, an executable can be downloaded from here. Make sure you get the SSL and SSH aware version. You will findcurl
useful later for debugging your scripts (it always works), so if you don't already have it, you'll want to get it.
ENCODE JSON in your browser
Look at biosample ENCBS000AAA at https://www.encodeproject.org/biosamples/ENCBS000AAA/.
By clicking the JSON button, which looks like {;}
, you should see the page viewed as JSON. Alternatively, you can manually add the following parameter to the URL: /?format=json
By adding the /?format=json
parameter, you've instructed the ENCODE webserver to send you the raw contents of the database record for ENCBS000AAA, in JSON format. Rather than being rendered in your browser, you see the native JSON object as it is stored in the ENCODE database. This is a biosample object, and you will see all the properties and their values for this object. You need a JSON pretty printer (see prerequisites above) in order to make sense of the JSON. Notice that many of the object’s properties contain other (nested) objects.
Notice also that the object has its own URL, and that the URL corresponds to the object’s accession number. This always works to access ENCODE objects. In fact, because accession numbers are unique, you can construct a URL from the accession number alone and the server will return your object. For example, the resource https://www.encodeproject.org/ENCBS000AAA maps to the same object identified by the longer URL https://www.encodeproject.org/biosamples/ENCBS000AAA/
As you learn more about the ENCODE REST API, you’ll see that when you retrieve objects (using the HTTP “GET” method) from the server, they always come back as JSON.
A simple command-line GET
Try this in a terminal window:
$ curl -H "Accept: application/json" https://www.encodeproject.org/biosamples/ENCBS000AAA/
You should see lots of text that starts with something like "{"system_slims": [], "accession": "ENCBS000AAA", ...".
Your command instructs curl
to download the database record for the ENCODE biosample ENCBS000AAA in JSON format (it's the same object rendered above and at https://www.encodeproject.org/biosamples/ENCBS000AAA/). In fact, the ENCODE server sees that the request is coming from something other than a web browser running the ENCODE web app, so it returns the object in JSON format. All curl
is really doing is downloading the "web page" that corresponds to that ENCODE record. Transport is by HTTP, just like a web browser.
This sort of JSON-formatted text is sometimes referred to as a "JSON document". You can redirect the output to a file:
$ curl -L -H "Accept: application/json" https://www.encodeproject.org/biosamples/ENCBS000AAA/ > ENCBS000AAA.json $ less ENCBS000AAA.json
In this example, the -L
option tells curl
to follow redirects. While not necessary in this example, -L
is necessary for identifiers, like aliases, that get redirected to another resource. In cases like that, without -L
you may see something like:
{"status": "error", "code": 301, "description": "Moved Permanently" ... }
If in doubt, use curl -L
.
If you supply an invalid identifier (like an accession number that doesn't refer to anything), you'll see something like:
{"status": "error", "code": 404, "description": "The resource could not be found." ... }
You can see that even error responses come back in JSON format. This makes parsing the results of your queries much easier.
To pretty-print JSON on the command line, you can use the jq
utility, available here. To use it with the output of curl
, you can simply pipe its output to jq
:
curl -H "Accept: application/json" https://www.encodeproject.org/biosamples/ENCBS000AAA/ | jq
REST, JSON and curl in perspective
As you develop your scripts, curl can be a valuable testing tool. If you can get an object with a curl command, you should be able to get it with your script.
The important thing to remember is that each ENCODE object is accessible via a simple URL containing an identifier (like an accession number) for that object. That's all curl does, that's all your browser does, and that's all your scripts will do. The resource you GET, using that URL, will be in JSON format. You will then use a library to parse the JSON and construct a native data structure (like a Python dictionary).
The ENCODE REST API
The ENCODE REST API uses GET to transport JSON-formatted information between the server and your scripts.
Please limit your GET scripts to no more than 10 requests/second.
Prerequisites
- A library or module for your language of choice that supports HTTP. The
requests
library here for Python is good, and will be used for examples in this documentation. - A library or module for parsing JSON-structured text and building native data structures. The Python
json
library here is good.
GET
The HTTP GET request is used to retrieve objects from the ENCODE server. GET will work without a username and password to fetch publicly-released ENCODE objects. Sending a GET request is accomplished here by using the requests.get()
method.
import requests, json # Force return from the server in JSON format headers = {'accept': 'application/json'} # This URL locates the ENCODE biosample with accession number ENCBS000AAA url = 'https://www.encodeproject.org/biosample/ENCBS000AAA/?frame=object' # GET the object response = requests.get(url, headers=headers) # Extract the JSON response as a Python dictionary biosample = response.json() # Print the Python object print(json.dumps(biosample, indent=4))
This script GET's the JSON representation of ENCODE biosample ENCBS000AAA. The ENCODE server's JSON response is extracted as a Python dictionary using the response.json()
method. The dictionary is then dumped back to a string and pretty-printed with useful indentation.
The output should look something like:
{ "biosample_ontology": "/biosample-types/cell_line_EFO_0001203/", "description": "mammary gland, adenocarcinoma", "donor": "/human-donors/ENCDO000AAE/", "passage_number": 5, "lab": "/labs/richard-myers/", "accession": "ENCBS000AAA", "parent_of": [], "@type": [ "Biosample", "Item" ], "url": "http://www.atcc.org/Products/All/HTB-22.aspx", "status": "released", "uuid": "56e94f2b-25ac-4c58-9828-f63b66220999" "notes": "(PMID: 4357757)", "award": "/awards/U54HG004576/", "age_display": "69 year", "product_id": "HTB-22", "culture_harvest_date": "2012-04-10", "applied_modifications": [], "dbxrefs": [ "UCSC-ENCODE-cv:MCF-7" ], "@id": "/biosamples/ENCBS000AAA/", "culture_start_date": "2012-03-16", "life_stage": "adult", "organism": "/organisms/human/", "aliases": [ "richard-myers:MCF7-003" ], "health_status": "breast cancer (adenocarcinoma)", "internal_tags": [], "references": [], "age_units": "year", "characterizations": [], "submitted_by": "/users/df9f3c8e-b819-4885-8f16-08f6ef0001e8/", "date_created": "2013-12-12T05:50:02.101495+00:00", "alternate_accessions": [], "source": "/sources/atcc/", "age": "69", "summary": "Homo sapiens MCF-7 cell line", "genetic_modifications": [], "schema_version": "24", "sex": "female", "treatments": [], "documents": [ "/documents/984071d4-9149-476a-b353-93592c6f48f3/" ] }
The important point is that the response object has been cast into a native Python dict datastructure. For example, documents is an array of document identifiers and so you can loop over that array.
Adding the following code to the example script above loops through the documents array, GET's each document object, and prints its description:
for doc_uri in biosample['documents']: doc_response = requests.get('https://www.encodeproject.org/' + doc_uri, headers=headers) document = doc_response.json() print document['description']
Which prints:
MCF-7 Cell Culture and 4-hydroxytamoxifen treatment
Programmatic search
In the GET example above, we use the ENCODE REST API to retrieve an individual object using its ENCODE accession number. You can also search for objects programmatically and get the search result back in JSON format. The ENCODE web app is a good place to start to see an example. Enter the string "bone chip" in the search box at the top right of the page at https://www.encodeproject.org/. The result are those ENCODE objects that match that string. Note that these results are by default not restricted to experiments, but include other object types as well. The object types of the results, among other properties, can be filtered using the facets to the left of the results.
Search URL format
The same URL that returns the search results to the web app can be used in a script. This script does the same search and returns the results in JSON format:
import requests, json # Force return from the server in JSON format headers = {'accept': 'application/json'} # This searches the ENCODE database for the phrase "bone chip" url = 'https://www.encodeproject.org/search/?searchTerm=bone+chip&frame=object' # GET the search result response = requests.get(url, headers=headers) # Extract the JSON response as a python dictionary search_results = response.json() # Print the object print(json.dumps(search_results, indent=4))
The structure of the response will be as follows. Keys with array values are denoted with [...]
and keys with object values are denoted with {...}
:
{ "notification": "Success", "title": "Search", "@id": "/search/?searchTerm=bone+chip", "clear_filters": "/search/?searchTerm=bone+chip", "columns": {...}, "all": "/?searchTerm=bone+chip&limit=all&format=json", "facets": [...], "@context": "/terms/", "@type": [...], "total": 180, "filters": [...], "@graph": [...] }
The @graph
property in the response script contains the array of search results, it should look something like this:
[ { "authors": "Johnson ME, Deliard S, Zhu F, Xia Q, Wells AD, Hankenson KD, Grant SF", "identifiers": [ "PMID:24337390" ], "title": "A ChIP-seq-defined genome-wide map of MEF2C binding reveals inflammatory pathways associated with its role in bone density determination.", "@id": "/publications/03e13868-6090-4f61-b4f1-b4a22a6e833a/", "issue": "4", "data_used": "ENCODE TF ChIP used throughout, Figures 1-2", "journal": "Calcified tissue international", "volume": "94", "date_created": "2014-08-13T01:26:28.846038+00:00", "@type": [ "Publication", "Item" ], "date_published": "2014 Apr", ... }, { "biosample_ontology": { "term_name": "bone marrow", "classification": "tissue" }, "replicates": [ { "antibody": { "accession": "ENCAB000ADX" }, "biological_replicate_number": 1, "technical_replicate_number": 1, "@id": "/replicates/8ca96774-c0d1-4f62-b7d3-f2988d406de2/", "library": { "biosample": { "age_units": "week", "life_stage": "adult", "organism": { "scientific_name": "Mus musculus" }, "accession": "ENCBS130ENC", "age": "8" } } }, ... ] }, ... ]
Whereas before, when we retrieved a single object, the @graph property was a list with only one element. Now @graph is a multi-element list, with each object that satisfies the search condition represented as one entry. Note that there are multiple types of objects returned by the query because we did not specify a type
.
The objects returned contain only a subset of the properties. These are the properties that the server returns to the web app to render the search results page. To get the full objects back, add &frame=object
to the query. The query URL looks like:
url = 'https://www.encodeproject.org/search/?searchTerm=bone+chip&frame=object&format=json'
This returns the full objects, with all their properties and values. Here is the @graph
array in the response:
[ { "data_used": "ENCODE TF ChIP used throughout, Figures 1-2", "categories": [ "basic biology" ], "status": "released", "volume": "94", "@type": [ "Publication", "Item" ], "uuid": "03e13868-6090-4f61-b4f1-b4a22a6e833a", "submitted_by": "/users/ce2bde01-07ec-4b8a-b179-554ef95b71dd/", "@id": "/publications/03e13868-6090-4f61-b4f1-b4a22a6e833a/", "page": "396-402", "issue": "4", "authors": "Johnson ME, Deliard S, Zhu F, Xia Q, Wells AD, Hankenson KD, Grant SF", "published_by": [ "community" ], "date_created": "2014-08-13T01:26:28.846038+00:00", "supplementary_data": [], "identifiers": [ "PMID:24337390" ], ... }, { "biosample_ontology": "/biosample-types/tissue_UBERON_0002371/", "aliases": [], "status": "released", "files": [ "/files/ENCFF001JXO/", "/files/ENCFF001JXP/", "/files/ENCFF001JXQ/", ..., ], "uuid": "2ed470f3-a57a-4ca4-83fe-d0bfa7022a36", "documents": [], "@id": "/experiments/ENCSR000CAG/", "description": "H3K4me1 ChIP-seq on 8-week mouse bone marrow", "replicates": [ "/replicates/8ca96774-c0d1-4f62-b7d3-f2988d406de2/", "/replicates/0cb223fe-4893-4068-aaad-43b0398bc5da/" ], ... }, ... ]
Embedded objects can be fully expanded in the search results with &frame=embedded
so that the query would look like:
url = 'https://www.encodeproject.org/search/?searchTerm=bone+chip&frame=embedded&format=json'
The @graph
of the response should start with something like:
[ { "supplementary_data": [], "authors": "Johnson ME, Deliard S, Zhu F, Xia Q, Wells AD, Hankenson KD, Grant SF", "lab": "/labs/encode-consortium/", "volume": "94", "published_by": [ "community" ], "data_used": "ENCODE TF ChIP used throughout, Figures 1-2", "identifiers": [ "PMID:24337390" ], "categories": [ "basic biology" ], "journal": "Calcified tissue international", "page": "396-402", "date_published": "2014 Apr", "award": "/awards/ENCODE/", "submitted_by": "/users/ce2bde01-07ec-4b8a-b179-554ef95b71dd/", "date_created": "2014-08-13T01:26:28.846038+00:00", "@type": [ "Publication", "Item" ], ... }, { "date_submitted": "2011-06-24", "superseded_by": [], "assay_term_name": "ChIP-seq", "lab": { "name": "bing-ren", "status": "current", "title": "Bing Ren, UCSD", "address2": "9500 Gilman Drive, Room 231", "@type": [ "Lab", "Item" ], "state": "CA", "pi": "/users/ea5e2bd6-be7a-4b18-a2e8-dee86005422f/", ..., }, ... }, ... ]
Note, for example, the lab
properties of results in the second query are objects, whereas in the previous search with frame=object
they were simply paths to lab
objects on the portal.
Note: the last four examples here return a lot of data, so the format=json
parameter has been omitted so as to avoid attempting to render it using an in-browser JSON formatter.
- Every object that matches the string “CTCF”:
https://www.encodeproject.org/search/?searchTerm=CTCF&format=json&frame=object - A file with a particular MD5 7b9f8ccd15fea0bda867e042db2b6f5a:
https://www.encodeproject.org/search/?type=File&md5sum=7b9f8ccd15fea0bda867e042db2b6f5a&format=json - All the file objects from a particular experiment ENCSR000AKS (with links to embedded objects):
https://www.encodeproject.org/search/?type=File&dataset=/experiments/ENCSR000AKS/&format=json&frame=object - All the fastq files from a particular experiment ENCSR000AKS:
https://www.encodeproject.org/search/?type=File&dataset=/experiments/ENCSR000AKS/&file_format=fastq&format=json&frame=object - All the replicates for a particular experiment ENCSR000AKS:
https://www.encodeproject.org/search/?type=Replicate&experiment.accession=ENCSR000AKS&format=json - The same query as above but with embedded objects expanded:
https://www.encodeproject.org/search/?type=Replicate&experiment.accession=ENCSR000AKS&format=json&frame=embedded - All biosamples (abbreviated metadata):
https://www.encodeproject.org/search/?type=Biosample - All biosamples (full metadata with object references):
https://www.encodeproject.org/search/?type=Biosample&frame=object - All experiments that are not ChIP-seq (note the use of
!=
in the query):
https://www.encodeproject.org/search/?type=Experiment&assay_title!=ChIP-seq&frame=object - All treated biosamples (note the use of * in the query):
https://www.encodeproject.org/search/?type=Biosample&treatments=*&frame=object
Summary of search features
- If you do not specify
frame=something
, you will get a subset of object properties as defined by the webapp. If what you need is in that subset (like accession, maybe), it’s a relatively fast way of getting back a long list of objects. - If you specify
frame=embedded
you will get all the object properties, with selected embedded objects expanded. frame=object
will always give you all of the properties, with embedded objects referred to by an identifier. So it’s consistent, and it’s faster. Which embedded objects are expanded byframe=embedded
may change in the future as our requirements for search change. Also, the depth of expansion is chosen to support search and might change as well. So, for the most robust code you might choose to useframe=object
and then GET only the embedded objects you need.- You do not need to specify
frame=embedded
in order to search within embedded objects. For example, the query paramaterlab.state=CA
would work when searching withframe=object
- All of the results from search can be visualized within the web app by opening the URL in your browser, omitting
format=json
. Not all objects have fancy rendered collections or pages, but you will always get back a clickable list. limit=all
is not strictly necessary but without it your result will contain only the first 25 hits (or all if < 25 hits).limit=all
should always be used in conjunction withformat=json
.- We plan to implement a mechanism to return very large searches in batches, but for now
limit=all
can generate large result sets. - Giving
searchTerm=somestring
will often return multiple object types. You can look into the@type
property of each returned object for its data type. - To make a search specifying that a property is NOT equal to a value (negation), use the
!=
operator (see example 9 above) - The
*
character can be used as a wildcard, matching any value. This is useful when you would like to check the existence or non-existence of a property. Specifingproperty=*
will return objects for which the property is both present and non-empty. It can be used with!=
as well to search for objects where a property is missing or empty.
Summary of syntax features
Parameter or syntax feature | Example | Description |
---|---|---|
frame=object | https://www.encodeproject.org/biosample/ENCBS000AAA/?frame=object | Returns all properties of the object, with any embedded objects referred to by an identifier. Note that if there is no "frame" parameter in the query, a subset of object properties as defined by the webapp, which is subject to change. |
frame=embedded | https://www.encodeproject.org/biosample/ENCBS000AAA/?frame=embedded | Returns all properties of the object, with selected embedded objects expanded, subject to change. |
limit |
https://www.encodeproject.org/search/?type=Biosample&limit=all&format=json https://www.encodeproject.org/search/?type=Biosample&limit=20 |
Returns up to the specified number of results, or returns all results in the case of limit=all . |
searchTerm | https://www.encodeproject.org/search/?type=Experiment&searchTerm=CTCF | Returns results matching the specified search string. |
lt | https://www.encodeproject.org/search/?type=File&quality_metrics.frip=lt:0.65 | Range syntax for "less than" for use with numeric properties. |
lte | https://www.encodeproject.org/search/?type=File&quality_metrics.frip=lte:0.65 | Range syntax for "less than or equal" for use with numeric properties. |
gt | https://www.encodeproject.org/search/?type=File&file_size=gt:1000000 | Range syntax for "greater than" for use with numeric properties. |
gte | https://www.encodeproject.org/search/?type=File&file_size=gte:1000000 | Range syntax for "greater than or equal" for use with numeric properties. |
* | https://www.encodeproject.org/search/?type=Biosample&treatments=* | Wildcard character, which can be used to check the existence/non-existence of a property. |
!= | https://www.encodeproject.org/search/?type=Biosample&treatments!=* | Negation. Query results will exclude results matching the specified value. |