How to Set and List Environmental Variable in Python and Jupyter Notebook

John D K -

In this short guide, you’ll see how to set environmental variables in Jupyter Notebook and JupyterLab.

Below you can find most popular options:

(1) Jupyter magic commands

%env AWS_KEY=XXX

(2) Setup environment variables

import os
os.environ['AWS_KEY'] = "XXX"

You will see multiple ways to set variables. Which one you are going to use will depend on your case.

Step 1: Set Environmental Variable with Jupyter Magic

Simplest option to set environment variables in Jupyter Notebook and JupyterLab is to use magic functions.

There two types of magic commands:

  • cell magic commands (start with %%)
  • line magic commands (start with %)

To find more about the magic commands check this out: Built-in magic commands

List all Environment Variable

To list all environmental variables in Jupyter Notebook and JupyterLab with magic you can use:

%env

result of this command is a dictionary in the form of:

 'JPY_PARENT_PID': '4217',
 'CLICOLOR': '1',
 'PAGER': 'cat',
 'GIT_PAGER': 'cat',
 'MPLBACKEND': 'module://ipykernel.pylab.backend_inline',
 'AWS_KEY': 'XXX'}

Set new Environmental Variable in Jupyter

Setting new variable or new value for existing environmental variable can be done by command:

%env AWS_KEY=XXX

Note: If you use quotes like:

%env AWS_KEY="XXX"

they will be included in the value:

'MY_VAR': '"MY_VALUE"'}

Step 2: Set Environmental Variable with Jupyter Magic

To set environment variable in Jupyter we can use: os.environ['AWS_KEY']. The difference between this one and the Jupyter magic is the quotes.

Set new Environmental Variable in Jupyter with os.environ

So if you need to use quotes you can use this one. You need to import the os module and use command like:

import os
os.environ['AWS_KEY'] = "XXX"

List Environmental Variables in Jupyter with os.environ

To list all or some environmental variables with os.environ. You can use this syntax:

import os
os.environ

Jupyter set environment variable by python-dotenv

To read .env file with Jupyter Notebook we can use Python library called: python-dotenv.

This library can be instaled by command:

pip install python-dotenv

Now it can be use to read .env files in Python or JupyterLab.

Suppose we have sample .env file:

# .env
MY_VAR=MY_VAL
S3_BUCKET=XXX
S3_SECRET_KEY=XXX

Jupyter environment variables

Reading environment variables from .env file and python-dotenv can be done by:

from dotenv import load_dotenv

# env file is in the same folder
load_dotenv()

Now we can read all variables from this env file by:

import os
print(os.environ.get('MY_VAR'))

returns:

MY_VAL

Load env file another folder

To load .env file stored in another folder we can do:

from dotenv import load_dotenv
from dotenv import load_dotenv

# env file is in the same folder
load_dotenv()

# explicitly provide path to '.env' file
from pathlib import Path 
env_path = Path('.conf/') / '.env'
load_dotenv(dotenv_path=env_path)

Example: Python read .env variables

We can create .env file in JupyterLab in folder /conf/:

API_KEY="xxx"
API_KEY_SECRET="xxx"
BEARER_TOKEN="xxx"
ACCESS_TOKEN="xxx"
ACCESS_TOKEN_SECRET="xxx"

The file won't be visible in JupyterLab or Notebook and you need to edit it from another app or terminal:

nano .env

Now we can read the variables from python by:

from dotenv import load_dotenv, find_dotenv
from pathlib import Path

path='~/conf/.env'
load_dotenv(dotenv_path=path,verbose=True)


consumer_key = os.environ["API_KEY"]
consumer_secret = os.environ["API_KEY_SECRET"]
access_token = os.environ["ACCESS_TOKEN"]
access_token_secret = os.environ["ACCESS_TOKEN_SECRET"]

This would make your Juputer Notebooks more secure and organized. Another tip is to exlude .env files from the git versioning.

Conclusion

We saw several option to set and read environment variable in Jupyter notebook. We covered in details how to work with Jupyter notebook and environmental variables.

If you need to learn more you can refer to: