More Python Traps

I recently posted on some Python Traps, focussing on code where changes in the values of sub-routine arguments were not reflected in the calling routines.  The reverse can also be a problem: if an array, a, is passed to a sub-routine in VBA, then a new array, b, with the same values created in the sub-routine using b = a, any changes to b will not be reflected in a.

In Python things work differently; the statement b = a means that the array a is now also called b, so changes to b will also change a, both in the sub-routine and in the calling routine.  This was discussed in some detail in The meaning of = in Python; this post looks at some practical examples, and how to create a new copy of an array, rather than just giving the existing array a new name.

The screenshots below illustrate further variations of the routines discussed in the Python Traps post:

  • An array, x, is read from the spreadsheet and passed to a subroutine, either as a list, a list of lists, or a 1D or 2D Numpy array.
  • In the subroutine an array, y, is created, either with the statement y = x, or with the Python copy or deepcopy commands.
  • y is multiplied by a constant.
  • y is retuned to the calling routine, as the function return value.
  • y and x are returned to the spreadsheet.

In the first three examples below the array x is passed as a Python list or a 1D Numpy array.  When the array y is created with y = x, the changes to y are also seen in x, but when x is created with  y = np.copy(x), x is not affected by the changes to y.  The Python copy function (y = copy.copy(x)) works the same way for a 1D list.

With a 2D list (list of lists) however the y array created with the Python copy function behaves the same as using y = x; the changes in y are also seen in x.  This is because copy only creates a new object for the top level list.  The lower level lists still refer to the same objects as in the x list.  To create a new list of lists at all levels it is necessary to use the Python deepcopy function: y = copy-deepcopy(x).

The numpy copy command however works the same on all levels of Numpy arrays. In the final example below the 2D array x is not affected  by the changes to y, which was created with y = np.copy(x).

Posted in Arrays, Excel, Link to Python, Newton, NumPy and SciPy, PyXLL, UDFs | Tagged , , , , , | Leave a comment

LinEstGap with non-linear functions

The LinEstGap user defined function (UDF) allows the built-in Linest function to be used on data with empty cells or hidden rows, returning results that agree with the chart trend-line function, which automatically ignores blank cells and hidden data.  It also optionally allows any lines with Error values to be ignored, rather than returning an error.

Following a comment here I have modified the code and added additional functions to work with all the non-linear curve  types provided by the chart trend line (Exponential, Logarithmic, Polynomial, and Power).

The revised spreadsheet includes examples and full open-source code, and can be downloaded from:

Linest-PolyDA.zip

The download zip file contains two versions of the spreadsheet.  Linest-PolyDA uses the new dynamic array feature.  Linest-PolyA has fixed size array functions that will work with older versions of Excel (see Using Array Formulas for more details).

I have extracted the code for extracting blanks and errors into a separate function (RemoveGaps), so the LinEstGap function is now reduced to:

Function LinEstGap(Ycells As Range, XA As Variant, Optional Const0 As Boolean = True, Optional Stats As Boolean = False, _
    Optional IgnoreErrors = False)
    Dim iErr As Variant
    Dim YA2() As Double, XA2() As Double

    iErr = RemoveGaps(Ycells, XA, XA2, YA2, Const0, Stats, IgnoreErrors)

    If iErr = 0 Then
        LinEstGap = WorksheetFunction.LinEst(YA2, XA2, Const0, Stats)
    Else
        LinEstGap = iErr
    End If
End Function

I have also added separate functions that call the RemoveGaps function, then Linest with modified X and Y data to return parameters for non-linear curves (see Using LINEST for non-linear curve fitting for more information on using Linest, and the download file for the VBA code).  Examples of the new functions are shown below (click on any image for a full size view):

The PolyEstGap function will return parameters for any order of polynomial function (but see Fitting high order polynomials for warnings on using orders greater than 3 or 4).

The functions ExpEstGap, PowerEstGap, and LogEstGap all return a two element array with values a and b for the functions:

  • ExpEstGap:   y = b. e^a
  • PowerEstGap; y = b.x^a
  • LogEstGap:     y = a.ln(x) + b

The screen shots below show the PolyEstGap function used on data with gaps and errors and filtered data, but the other Gap functions all work the same way.

Using the built in LinEst on data with gaps returns an error, but the Gap functions return the same results as the chart trend line; i.e. any line with a gap in the data is ignored:

Filtered data is also ignored:

The chart trend line treats error values as zero, so returns invalid results.  The Gap functions return an error by default (line 19 below), but if the final optional argument ( IgnoreErrors) is set to True they will return the correct results for the data without the error lines.  Note that the results with errors in rows 12 and 13 below are exactly the same as those with rows 12 and 13 hidden above.

The Gap functions may also be used on data arranged in rows as shown below:

For the polynomial examples above the LinEstGap function may also be used with modified input, or with multiple columns of x data, to generate the same results. For the other non-linear curve types however modifying the LinEstGap input data will generate an error, but the new Gap functions return the correct results.

Posted in Arrays, Curve fitting, Excel, Maths, Newton, UDFs, VBA | Tagged , , , , , , , , , | 4 Comments

DigitGraph Spreadsheet and WebPlot Digitiser

I have posted here previously a spreadsheet that allows XY data to be extracted from images of graphs, maps or other images of objects in a single plane: How to digitise a scanned image.

I have updated the instructions for the procedure to cover Excel’s new dynamic array feature, which can return an array of data from a function entered in a single cell.  The new version can be downloaded from:

DigitGraph2.zip

I also recently discovered the Webplot Digitizer program, that can either be used on-line, or as a download.  The download is free (with no advertising), with a button for voluntary donations.  The screen-shot below shows the downloaded version.  The program is considerably more sophisticated than my spreadsheet, providing not only a magnified image of the graph at the cursor location, but also allowing the option of either manual selection of data points, or a fully automated process to detect the graph line(s) and generate the data points.

Output from the WebPlot program (both manual and automated) is shown below plotted together with output from my Digitgraph function, showing close agreement for all three lines.

Posted in Charts, Charts, Computing - general, Curve fitting, Drawing, Excel, UDFs, VBA | Tagged , , , , , | 5 Comments

Katherine Johnson, NASA Legend

Cartoon by Steve Breen from The San-Diego Union Tribune.

More on Katherine Johnson.

Posted in Maths, Newton | Tagged , , | Leave a comment

Microsoft Python Resources

In addition to the documentation of their software products, Microsoft now have a huge amount of on-line resources of more general content, including:

YouTube: Python for Beginners

Probably the largest hurdle when learning any new programming language is simply knowing where to get started. This is why we, Chris and Susan, decided to create this series about Python for Beginners! (44 Videos)

For those who prefer written tutorials, with interactive examples:

Introduction to Python

Learn how to write basic Python code, declare variables, and work with console input and output

In this module, you will:

Explore the choices available to run Python apps
Use the Python interpreter to execute statements and scripts
Learn how to declare variables
Build a simple Python app that takes input and produces output

Also see Microsoft Learn for other Python based tutorials, or search for a huge range of other topics.

For documentation of the use of Python in Visual Studio see Python at Visual Studio

Welcome Python Developers

See how Microsoft is making our platform better for Python developers
Python Blog
Posted in Computing - general, Excel, Link to Python | Tagged , , , | Leave a comment