Commit 79ddbef7 authored by Anchal Gupta's avatar Anchal Gupta

Copied ipynb clean test from cit_ctnlab/ctn_labdata

parent 0b774b93
Pipeline #142646 passed with stage
in 31 seconds
- test
image: python:3.5
stage: test
- echo "Running python notebook checker"
- pip install -r .testing/requirements.txt
- python .testing/ "./"
- echo "All ci check tasks complete"
This folder contains elements essential for gitlab CI testing and deployment. It is a only 'hidden' for default view to keep it out of the way and creating clutter.
This is mostly a place to put bash and python scripts used to run tests on things commited to the noise budget code.
There are also scripts for checking commited notebooks to make sure users aren't commiting junk in the output
Usage: python notebook.ipynb
System will exit with code 1 in the event that a notebook in the search path
contains uncleared output. This is so that gitlab CI can catch commits with
bloated content and prompt the user to fix their oversite.
At this stage its not clear how to pass a speific message with the offending
notebook to gitlab CI, but users can find the terminal dialog in the web
interface pipline dialog.
Author: Andrew Wade (
Date: Oct 24, 2017
Modified: Anchal Gupta (Superficial changes for PEP8 conformity)
import sys
import io
import os
import fnmatch
from nbformat import read, write
def checkNbClean():
# flag state one if all notebooks found pass the test
passflag = 1
# get arg of input dir to search
topdir = sys.argv[1]
# loop over all files in dir that match pattern
for filename in recursive_glob(topdir,"*.ipynb"):
# exclude checkpoints, these are junk
if not "-checkpoint.ipynb" in filename:
with, 'r') as f:
nb = read(f, as_version=4) # open contents of notebook
# run check_nb_clean sub routine
if not check_nb_clean(nb):
print("Uncleared cells in notebook "
"located at {loc}".format(loc=filename))
# flip the pass flag to the failed state
passflag = passflag * 0
print("Notebook clean at {loc}".format(loc=filename))
# take action to exit in pass/fail state,
# fail means that just one notebook has been commited with output
if passflag != 1:
sys.exit(1) # this throws a system exit with code 1 to trigger CI fail
sys.exit(0) # clean exit for system
def check_nb_clean(nb):
"""Checks if codes cells have output that hasn't been cleared"""
for cell in nb.cells:
if cell.cell_type == 'code':
if cell.outputs != []:
return 0
return 1
def recursive_glob(rootdir='.', pattern='*'):
"""Search recursively for files in specified directory. """
matchingList = []
for dirpath, dirnames, filenames in os.walk(rootdir):
for filename in fnmatch.filter(filenames, pattern):
matchingList.append(os.path.join(dirpath, filename))
return matchingList
# custom error messages for this script
class nbNotCleanError(Exception):
Exception raised when nb submitted to test fails the test of clean
def __init__(self, message):
self.message = message
def __str__(self):
return repr(self.message)
if __name__ == '__main__':
# for running notebook in cmdline
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment