You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
545 lines
13 KiB
545 lines
13 KiB
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Importing Jupyter Notebooks as Modules"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"It is a common problem that people want to import code from Jupyter Notebooks.\n",
|
|
"This is made difficult by the fact that Notebooks are not plain Python files,\n",
|
|
"and thus cannot be imported by the regular Python machinery.\n",
|
|
"\n",
|
|
"Fortunately, Python provides some fairly sophisticated [hooks](https://www.python.org/dev/peps/pep-0302/) into the import machinery,\n",
|
|
"so we can actually make Jupyter notebooks importable without much difficulty,\n",
|
|
"and only using public APIs."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import io, os, sys, types"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from IPython import get_ipython\n",
|
|
"from nbformat import read\n",
|
|
"from IPython.core.interactiveshell import InteractiveShell"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Import hooks typically take the form of two objects:\n",
|
|
"\n",
|
|
"1. a Module **Loader**, which takes a module name (e.g. `'IPython.display'`), and returns a Module\n",
|
|
"2. a Module **Finder**, which figures out whether a module might exist, and tells Python what **Loader** to use"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def find_notebook(fullname, path=None):\n",
|
|
" \"\"\"find a notebook, given its fully qualified name and an optional path\n",
|
|
"\n",
|
|
" This turns \"foo.bar\" into \"foo/bar.ipynb\"\n",
|
|
" and tries turning \"Foo_Bar\" into \"Foo Bar\" if Foo_Bar\n",
|
|
" does not exist.\n",
|
|
" \"\"\"\n",
|
|
" name = fullname.rsplit('.', 1)[-1]\n",
|
|
" if not path:\n",
|
|
" path = ['']\n",
|
|
" for d in path:\n",
|
|
" nb_path = os.path.join(d, name + \".ipynb\")\n",
|
|
" if os.path.isfile(nb_path):\n",
|
|
" return nb_path\n",
|
|
" # let import Notebook_Name find \"Notebook Name.ipynb\"\n",
|
|
" nb_path = nb_path.replace(\"_\", \" \")\n",
|
|
" if os.path.isfile(nb_path):\n",
|
|
" return nb_path"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Notebook Loader"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Here we have our Notebook Loader.\n",
|
|
"It's actually quite simple - once we figure out the filename of the module,\n",
|
|
"all it does is:\n",
|
|
"\n",
|
|
"1. load the notebook document into memory\n",
|
|
"2. create an empty Module\n",
|
|
"3. execute every cell in the Module namespace\n",
|
|
"\n",
|
|
"Since IPython cells can have extended syntax,\n",
|
|
"the IPython transform is applied to turn each of these cells into their pure-Python counterparts before executing them.\n",
|
|
"If all of your notebook cells are pure-Python,\n",
|
|
"this step is unnecessary."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"class NotebookLoader(object):\n",
|
|
" \"\"\"Module Loader for Jupyter Notebooks\"\"\"\n",
|
|
"\n",
|
|
" def __init__(self, path=None):\n",
|
|
" self.shell = InteractiveShell.instance()\n",
|
|
" self.path = path\n",
|
|
"\n",
|
|
" def load_module(self, fullname):\n",
|
|
" \"\"\"import a notebook as a module\"\"\"\n",
|
|
" path = find_notebook(fullname, self.path)\n",
|
|
"\n",
|
|
" print(\"importing Jupyter notebook from %s\" % path)\n",
|
|
"\n",
|
|
" # load the notebook object\n",
|
|
" with io.open(path, 'r', encoding='utf-8') as f:\n",
|
|
" nb = read(f, 4)\n",
|
|
"\n",
|
|
" # create the module and add it to sys.modules\n",
|
|
" # if name in sys.modules:\n",
|
|
" # return sys.modules[name]\n",
|
|
" mod = types.ModuleType(fullname)\n",
|
|
" mod.__file__ = path\n",
|
|
" mod.__loader__ = self\n",
|
|
" mod.__dict__['get_ipython'] = get_ipython\n",
|
|
" sys.modules[fullname] = mod\n",
|
|
"\n",
|
|
" # extra work to ensure that magics that would affect the user_ns\n",
|
|
" # actually affect the notebook module's ns\n",
|
|
" save_user_ns = self.shell.user_ns\n",
|
|
" self.shell.user_ns = mod.__dict__\n",
|
|
"\n",
|
|
" try:\n",
|
|
" for cell in nb.cells:\n",
|
|
" if cell.cell_type == 'code':\n",
|
|
" # transform the input to executable Python\n",
|
|
" code = self.shell.input_transformer_manager.transform_cell(cell.source)\n",
|
|
" # run the code in themodule\n",
|
|
" exec(code, mod.__dict__)\n",
|
|
" finally:\n",
|
|
" self.shell.user_ns = save_user_ns\n",
|
|
" return mod"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## The Module Finder"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"The finder is a simple object that tells you whether a name can be imported,\n",
|
|
"and returns the appropriate loader.\n",
|
|
"All this one does is check, when you do:\n",
|
|
"\n",
|
|
"```python\n",
|
|
"import mynotebook\n",
|
|
"```\n",
|
|
"\n",
|
|
"it checks whether `mynotebook.ipynb` exists.\n",
|
|
"If a notebook is found, then it returns a NotebookLoader.\n",
|
|
"\n",
|
|
"Any extra logic is just for resolving paths within packages."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"class NotebookFinder(object):\n",
|
|
" \"\"\"Module finder that locates Jupyter Notebooks\"\"\"\n",
|
|
"\n",
|
|
" def __init__(self):\n",
|
|
" self.loaders = {}\n",
|
|
"\n",
|
|
" def find_module(self, fullname, path=None):\n",
|
|
" nb_path = find_notebook(fullname, path)\n",
|
|
" if not nb_path:\n",
|
|
" return\n",
|
|
"\n",
|
|
" key = path\n",
|
|
" if path:\n",
|
|
" # lists aren't hashable\n",
|
|
" key = os.path.sep.join(path)\n",
|
|
"\n",
|
|
" if key not in self.loaders:\n",
|
|
" self.loaders[key] = NotebookLoader(path)\n",
|
|
" return self.loaders[key]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Register the hook"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Now we register the `NotebookFinder` with `sys.meta_path`"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"sys.meta_path.append(NotebookFinder())"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"After this point, my notebooks should be importable.\n",
|
|
"\n",
|
|
"Let's look at what we have in the CWD:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"ls nbpackage"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"So I should be able to `import nbpackage.mynotebook`."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import nbpackage.mynotebook"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Aside: displaying notebooks"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Here is some simple code to display the contents of a notebook\n",
|
|
"with syntax highlighting, etc."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from pygments import highlight\n",
|
|
"from pygments.lexers import PythonLexer\n",
|
|
"from pygments.formatters import HtmlFormatter\n",
|
|
"\n",
|
|
"from IPython.display import display, HTML\n",
|
|
"\n",
|
|
"formatter = HtmlFormatter()\n",
|
|
"lexer = PythonLexer()\n",
|
|
"\n",
|
|
"# publish the CSS for pygments highlighting\n",
|
|
"display(\n",
|
|
" HTML(\n",
|
|
" \"\"\"\n",
|
|
"<style type='text/css'>\n",
|
|
"%s\n",
|
|
"</style>\n",
|
|
"\"\"\"\n",
|
|
" % formatter.get_style_defs()\n",
|
|
" )\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def show_notebook(fname):\n",
|
|
" \"\"\"display a short summary of the cells of a notebook\"\"\"\n",
|
|
" with io.open(fname, 'r', encoding='utf-8') as f:\n",
|
|
" nb = read(f, 4)\n",
|
|
" html = []\n",
|
|
" for cell in nb.cells:\n",
|
|
" html.append(\"<h4>%s cell</h4>\" % cell.cell_type)\n",
|
|
" if cell.cell_type == 'code':\n",
|
|
" html.append(highlight(cell.source, lexer, formatter))\n",
|
|
" else:\n",
|
|
" html.append(\"<pre>%s</pre>\" % cell.source)\n",
|
|
" display(HTML('\\n'.join(html)))\n",
|
|
"\n",
|
|
"\n",
|
|
"show_notebook(os.path.join(\"nbpackage\", \"mynotebook.ipynb\"))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"So my notebook has some code cells,\n",
|
|
"one of which contains some IPython syntax.\n",
|
|
"\n",
|
|
"Let's see what happens when we import it"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from nbpackage import mynotebook"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Hooray, it imported! Does it work?"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"mynotebook.foo()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Hooray again!\n",
|
|
"\n",
|
|
"Even the function that contains IPython syntax works:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"mynotebook.has_ip_syntax()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Notebooks in packages"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"We also have a notebook inside the `nb` package,\n",
|
|
"so let's make sure that works as well."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"ls nbpackage/nbs"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Note that the `__init__.py` is necessary for `nb` to be considered a package,\n",
|
|
"just like usual."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"show_notebook(os.path.join(\"nbpackage\", \"nbs\", \"other.ipynb\"))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from nbpackage.nbs import other\n",
|
|
"\n",
|
|
"other.bar(5)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"So now we have importable notebooks, from both the local directory and inside packages.\n",
|
|
"\n",
|
|
"I can even put a notebook inside IPython, to further demonstrate that this is working properly:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import shutil\n",
|
|
"from IPython.paths import get_ipython_package_dir\n",
|
|
"\n",
|
|
"utils = os.path.join(get_ipython_package_dir(), 'utils')\n",
|
|
"shutil.copy(\n",
|
|
" os.path.join(\"nbpackage\", \"mynotebook.ipynb\"), os.path.join(utils, \"inside_ipython.ipynb\")\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"and import the notebook from `IPython.utils`"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"collapsed": false
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from IPython.utils import inside_ipython\n",
|
|
"\n",
|
|
"inside_ipython.whatsmyname()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"This approach can even import functions and classes that are defined in a notebook using the `%%cython` magic."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"gist_id": "6011986",
|
|
"nbsphinx": {
|
|
"execute": "never"
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.5.1+"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
}
|