Refactoring - Step 1

[pya]

This refactoring that does not change any functionality of the library. The tests allow to make the plotting optional for runs without user interaction. Furthermore, a new runner can run all tests.

These are the main changes:

1. The Python files form a library, i.e. are importable via from scrins.operators.avg import avg.
   The main directory scrins and its subdirectories now have a file __init__.py.
   If the library is not in the Python path, setting the environmental variable PYTHONPATH will
   make it importable.
   Windows: set PYTHONPATH=C:\path\to\dir\with_scrins
   Linux/MacOS (bash): export PYTHONPATH=/path/to/dir/with_scrins
   The names of the subdirectories are all lower case as suggested by PEP8 .
   Unneeded import-all files are deleted. All imports are explicit in the corresponding module.

2. The style of some files now conform to PEP8 .
   The tool pylint gives a score of 10 out of 10 and the tool pycodestyle does not produce
   any output. This means the style is compliant. TODO: Do this style change for all files.

3. The test have been moved into a subdirectory of scrins. This makes them importable.
   A test runner test_all.py runs all tests. The test to run have to imported in
   scrins/test/__init__.py. All tests are in functions. The name of these functions is always
   main. All of them are defined as def main(show_plot=True):, meaning that
   how_plot is True if the function is called without arguments. All test files end with:

               if __name__ == '__main__':
                   main()

3. (contd) Therefore, the test will be run when the file is run as a standalone script from the command line with python test_name.py. These tests do not run when imported and need to be explicitly started with test_name.main(). The test runner test_all.py does this and starts them via test_name.main(show_plot=False).

This is the first iteration of changes. More style changes to several files are needed to complete this first refactoring phase.

[Niceno]

Dear Mike,

Daniel and me analyzed the changes you did, and with this reply we would like to comment on them:

1 - Importing only the functionality needed in each file. We do understand that this is the best programming practice in modern object oriented (OO) languages like Python, but as I pointed out in the message I sent to you on March 29, 2017, at 13:21, it only confuses my students and distracts them from their main goal which is to simulate certain fluid flow. For convenience, I repeat what I wrote then: Keep in mind that my students are either mechanical, nuclear or chemical engineering students. Hence, education for them is to understand physical models rather than semantics of a programming language. For example, for them is completely irrelevant if "sqrt" and "solve" are in "math" or "numpy", they just want to use these functions to simulate or write computational models for certain physical phenomena. Writing "np.solve", followed by "math.sqrt" is a distraction for them, it should be abstract. Then I went on to add: we should still end up in an environment in which all mathematical functionality is available to student developers. If my student has to google "sqrt doesn't work in Python 3.6", he or she doesn't have a proper environment to work in. (That is why I include all Python modules that I use as a bunch, although I am aware that it might be blasphemy for a puritan Python programmer) . Then finally, when referring to individual work packages, I stressed it again with: Make the code more pythonic, i.e. refactor the code and the package structure, but keeping all necessary mathematical functionality available to developers in all parts of the code, imported from a centralized location.
If the first contact of my student with numerical analysis of fluid flows is:

from scrins.physical_models.properties_for_air import properties_for_air
from scrins.physical_models.properties_for_water import properties_for_water
from scrins.constants.boundary_conditions import DIRICHLET, NEUMANN, OUTLET
from scrins.constants.coordinates import X, Y, Z
from scrins.constants.gravitational_constant import G
from scrins.constants.compass import W, E, S, N, B, T, C
from scrins.display.plot_isolines import plot_isolines
from scrins.discretization.adj_n_bnds import adj_n_bnds
from scrins.discretization.cartesian_grid import cartesian_grid
from scrins.discretization.nodes import nodes
from scrins.discretization.create_unknown import create_unknown
from scrins.discretization.cfl_max import cfl_max
from scrins.discretization.calc_p import calc_p
from scrins.discretization.calc_t import calc_t
from scrins.discretization.calc_uvw import calc_uvw
from scrins.discretization.corr_uvw import corr_uvw
from scrins.discretization.vol_balance import vol_balance
from scrins.display.print_time_step import print_time_step
from scrins.operators.avg import avg
from scrins.operators.par import par

he or she might get discouraged to keep on working with us, or programming in Python :(

2 - Style changes. OK, this is super important and I am really glad you pointed this out. We will do our best to continue spreading these changes throughout the code. Thanks 👍

3 - Directory re-organization. Well, another a pretty good one. I wasn't happy with previous directory structure, and never quite understood what those:

if __name__ == '__main__':
    main()

stand for. Putting all tests in a loop is a great addition too 👍

Speaking of tests, you mentioned in one of your e-mails that they take several hours to conduct. I can believe that. We should reduce the number of tests, and the length of each particular test to make them more manageable. Daniel and me already made a short list yesterday.

In one of the e-mails you mentioned that this work took you more time than you anticipated. I am not quite sure if it means that you are out of time to work on this project, but we would certainly appreciate if you could suggest us the following:

1 - Could you reduce the number of imports in each of the files in the "tests" directory? Those are the files all students will be looking at. The other files, in sub-directories "discretization", "display", ... only advanced students will have a look, and they won't mind properly pythonized coding.

2 - How to conduct profiling of a Python code? Can you suggest a tool? With your hint, we can take care of it.

3 - I believe that the way constants (like N, W, E, S ...) are defined in the code is a downright disgrace. I know that constants are by their very definition non-pythonic, but could you nevertheless suggest a better way to define them?

[pya]

I. Imports

This is a first iteration. The main purpose is to make it work at all as a regular Python package. There is lots of room for improvement.

We need to import the needed functions from the modules in either way.

1. Implicit import - As done before with all the from module import *

The functions and constants just appear out of nowhere. The user needs to know that they exist. There is no way of knowing what comes from where by looking at the source code. The user has to know the names of all functions and constants by heart or get them from somewhere. This is far from ideal for a good learning effect. How does the user know what function he/she needs? What if the user wants to know what a function does? How does he find its source code? It took me a long time to find the origin of some of them. I had to scan through many files

2. Explicit imports of all functions and constants

This leads to a really long series of import statement at the beginning of some files. But the user can just copy the imports. The advantage is that the user can see where each function comes from. The name and the origin of each function is right there. In the interactive prompt or a Jupyter Notebook the user gets call tips and interactive help, i.e. the docstring shows up. Furthermore, in the Jupyter Notebook a simple ?? at the beginning or end of the function name will show its source code.

3. Solution "Flattening the name space"

A more usable approach would be to make all important functions and constants the user needs for writing his client code part of the name space scrins. For example, a student would only write:

import scrins

at the top the program.

Later in the code all functions exposed by scrins can be used like:

scrins.properties_for_air(rc)

or

scrins.calc_p(p, (uf, vf, wf), rho, dt, (dx, dy, dz), obst)

This has the advantage that all decent tools (for example Jypyter Notebook, IPython, IDLE, Spyder, PyCharm etc.) would display all possible options after writing scrins. and pressing a <TAB> key (or a different key depending on the tool/editor).

This is how NumPy works. The package numpy has about 600 attributes. But (nearly) all of them are defined in files in sub(sub-sub)-packages.

II. Profiling

I would use cProfile (part of the standard library, https://docs.python.org/3.6/library/profile.html) for function-level profiling and line_profiler (https://github.com/rkern/line_profiler) for line-by-line profiling. The latter adds a lot of overhead and can therefore only be used on a few functions (if at all) at a time.

III. Constants

I would put all constants in a simple ini file and read it with configparser (https://docs.python.org/3.6/library/configparser.html). This separates data and code. Also it would make it possible to add user-define or per-project constants if needed. For example, just putting a scrins.ini file into the user home directory or the current project directory would allow to override some of the constants. Just in case somebody wants to use the software for processes on Mars and needs to change g. ;)