Roadmap and Notes¶

Triage¶

[x] Read runbook plan ... support reading params from file
[ ] If no argument included for RUNBOOK TITLE in edit/plan/run then prompt with options
[x] Include field ‘embeds’ in the metadata to help with referencing them
[ ] — Add helper for referencing the embeds?
[ ] Use execute and upload output to S3 for non-interactive: https://github.com/nteract/papermill/tree/main?tab=readme-ov-file#execute-via-cli

[ ] Add linter for runbooks (using runbook lint) that checks for presence of title, desc, rollback, cleanup, etc
[ ] Add runbook init suggestion or automation to add export RUNBOOK_WORKING_DIR=... to shell initializations
[ ] Install pre-commit.yml or git integration during init for secure linting and talisman
[x] Setup versioning and bumper (using versioner from npm ecosystem @release-it and @release-it/bumper)
[x] Fix tag key getting stripped out of planned runbooks b/c it breaks papermill
[x] Setup tagging in the notebooks to auto-set those values
[ ] Security: Setup git to autoclear cell outputs for a given folder’s notebooks (ie templates) https://stackoverflow.com/a/58004619
[x] Assess if we need shell completions
- [x] Yes, very nice and use https://click.palletsprojects.com/en/8.1.x/shell-completion/#custom-type-completion to define for custom types, ie only find ipynb files for edit, create, etc
[ ] Integrate mechanisms to read Grafana or other data sources and package “pre/post check” helpers
- [ ] ie poll for values over N period and if exceeding Y or Z then throw error and suggest rollbacks

[ ] Include safe way to retry when APIs fail but with confirmation
- for handling the case when infrastructure API calls fail when made in large async batch
[ ] Immutably store everything with bookstore https://github.com/nteract/bookstore
[ ] Setup a watcher for auto-exporting html or other formats
[x] Use slack notify for posting execution steps: https://github.com/keitakurita/jupyter-slack-notify

[x] Setup different folder from runbooks for execution files?
[x] Only allow for executing runbooks that are pre-processed
[x] Should I follow the tf convention of plan | apply
[x] Setup decorator to embed dry_run into shell command
[ ] (won’t do yet) Allow for executing cell by cell from commandline in a repl?
- [ ] Running cell by cell: https://github.com/odewahn/ipynb-examples/blob/master/Importing%20Notebooks.ipynb
[ ] Build auditability through a custom runner interface – Or a custom kernel wrapper? – https://ipython.readthedocs.io/en/stable/config/options/kernel.html
[x] Jupytext runner that watches fs changes and exports to scripts (can be done via jupytext config)
[ ] After execution in web browser jupyter notebook or in vscode dynamic environment, execute as HTML for archival. It looks great.
- Use custom format for nbconvert exports: https://nbconvert.readthedocs.io/en/latest/customizing.html
jupyter nbconvert –TemplateExporter.extra_template_basedirs=nbconvert/templates –to markdown –template=mdaudit exec uted-Untitled4.ipynb
[x] (NA it’s in modern jupyter) Bundle in the timing nbextension jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/execute_time/readme.html
[x] (won’t do) Freeze dangerous cells as a safegurad? https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/freeze/readme.html
[x] (won’t do) Cell filtering https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/cell_filter/README.html

How to connect to existing running notebook from different interface:

https://stackoverflow.com/questions/31382405/ipython-notebook-how-to-connect-to-existing-kernel

Runners
- Run from local
- Run against remote deployed server behind auth? can be done via Spawners
- Store files on S3 (ie repo published to S3 for centralized repo?) https://github.com/danielfrg/s3contents
Get extension for skip working: https://raw.githubusercontent.com/RobbeSneyders/Jupyter-skip-extension/master/skip_kernel_extension.py