Fergus Laing (and a rolling stone)

A song by Richard Thompson.

Any resemblance to any living person is, I’m sure, entirely coincidental.

Fergus Laing is a beast of a man
He stitches up and fleeces
He wants to manicure the world
And sell it off in pieces
He likes to build his towers high
He blocks the sun out from the sky
In the penthouse the champagne’s dry
And slightly gassy

Fergus Laing, he works so hard
As busy as a bee is
Fergus Laing has 17 friends
All as dull as he is
His 17 friends have 17 wives
All the perfect shape and size
They wag their tails and bat their eyes
Just like Lassie

Fergus he builds and builds
Yet small is his erection
Fergus has a fine head of hair
When the wind’s in the right direction

Fergus Laing and his 17 friends

They live inside a bubble
There they withdraw and shut the door
At any sign of trouble
Should the peasants wail and vent
And ask him where the money went
He’ll simply say, it’s all been spent
On being classy

Fergus’ buildings reach the sky
Until you cannot see ‘um
He thinks the old stuff he pulls down
Belongs in a museum
His fits are famous on the scene
The shortest fuse, so cruel, so mean
But don’t call him a drama queen
Like Shirley Bassey

Fergus Laing he flaunts the law
But one day he’ll be wired
And as they drag him off to jail
We’ll all shout, “You’re fired!”

And for something completely different, Richard Thompson is sometimes called England’s Bob Dylan, so here is Scotland’s Bob Dylan, Robin Williamson plays “Like a Rolling Stone”:

Posted in Bach | Tagged , , | Leave a comment

xlwSciPy 1.09 – update for xlwings 0.10 and Scipy 0.18.1

The xlwSciPy spreadsheet (last presented here) has been updated for the latest version of xlwings and Scipy.

The new spreadsheet can be downloaded from:

xlScipy-xlw.zip

including full open source code.

The spreadsheet requires Python, including xlwings, Numpy, Scipy and Pandas (all of which are free, and included in the Anaconda package).

The new spreadsheet includes a CubicSpline function, which is new in Scipy 0.18.  Some options for the new function are shown in the screen shots below:

The function has an optional “BC-type” argument, that controls the spline end conditions.  The argument may be entered as a single text string (one of: “not-a-knot”, “periodic”, “clamped”, or “natural”), or a 2×2 array.  The default value is “not-a-knot”, which returns the same results as the xl_UniSpline and xl_Splev functions:scipy4-1

For the “periodic” option the first and last Y value in the spline data must be equal. The function then returns a curve with equal slope and curvature at each end:scipy4-2

“Clamped” end conditions result in zero slope at the ends:scipy4-3

“Natural” end conditions have zero curvature at the ends:scipy4-4

Using the array argument the slope or curvature may be set separately at each end. The input shown below specifies a slope (1 in column 1) of -1 at both ends:scipy4-5

Similarly the curvature may be set to any desired value with a 2 in column 1 of the BC_type array:scipy4-6

See more detailed documentation at the Scipy Docs.

Xlwings 0.10 introduces a new feature that expands array return values in user defined functions (UDFs) to show all the results, without entering as an array function:scipy4-7

This feature is currently only used in the xl_evala function, on the Eval sheet. Xl_evala returns an array with the same number of rows as the rows with numeric data in the input data.  When entered with the data from row 106 to 110 in the screen shot above, results are  automatically returned to the same rows when the functioned is entered (just press enter, not ctrl-shift enter).

If the input range is extended down to row 136, the output is adjusted to suit: scipy4-8

The Python code required is quite short:

@xw.func
@xw.arg("x", ndim=2)
@xw.ret(expand='table')
def rtnarray2(x):
    return x

This can then be called from VBA …:

Function rtnarray2(x)
        If TypeOf Application.Caller Is Range Then On Error GoTo failed
        rtnarray2 = Py.CallUDF("xlwScipy", "rtnarray2", Array(x), ThisWorkbook, Application.Caller)
        Exit Function
failed:
        rtnarray2 = Err.Description
End Function

… and tacked on the end of any other VBA function:

Function xl_EvalA(func As String, xRange As Variant, Optional SymRange As Variant, Optional ValRange As Variant, Optional ReturnType As Long = 1) As Variant
    ...
    Set result = Py.Call(Methods, "xl_Evalx", Py.Tuple(func, xRange, VarName, SymRange, ValRange))
    Set Result_List = Py.Call(result, "tolist")
    Rtn = Py.Var(Result_List)

    Rtn = TransposeA(Rtn)
    xl_EvalA = rtnarray2(Rtn)
    Exit Function

 

Posted in Arrays, Curve fitting, Excel, Link to Python, Maths, NumPy and SciPy, Python Pandas, UDFs, VBA | Tagged , , , , , , , , | 2 Comments

VBA routines for splitting and joining text

As mentioned in the previous post, I have written two short VBA routines to aid the process of splitting a column of text strings into separate columns, using either a space or any other chosen character as the delimiter.  These routines have been added to the Text-in2 spreadsheet, along with a new JoinText function to reverse the process.  The new file can be downloaded (including full open source code) from:

Text-in2.ZIP

For an example of the use of the new routines see the Txt2Col sheet:

txt2cols2-1

Text (including text from pdf files) can be copied and pasted anywhere.  Select all the rows and as many columns as you want to split, then press Alt-F8, select Text2TextCols, and click Run:

txt2cols2-2

The text in the first column is split into the selected columns in text format, so that the original number formats are retained:

txt2cols2-3

The ResetTxt2Cols macro is for use when the Excel Text to Columns wizard has been used, and you want to paste text copied from external files into a single column.  To run press Alt-F8, select ResetTxt2Cols, and click run.

Split text (or any other text in a continuous column or row) can be combined with the JoinText user defined function (UDF) as shown below:

txt2cols2-4

JoinText has two optional arguments:

  • Separate defines the separator to add between cell contents (default a single space).
  • IgnoreBlank ignores blank cells if set to true.

Excel 2016 now has two new built in functions providing similar functionality, Concat() and TextJoin().  The JoinText UDF still has a couple of advantages however:

  • It will work in any version of Excel that supports VBA.
  •  The Separate and IgnoreBlank arguments are optional, simplifying use when the default values are to be used.
Posted in Excel, UDFs, VBA | Tagged , , , , , , , | Leave a comment

The error made in 20% of papers on genes …

… and how to avoid it.

According to a recent scientific paper “Gene name errors are widespread in the scientific literature” (authors: Mark Ziemann, Yotam Eren1, and Assam El-Osta).  The paper says that “approximately one-fifth of papers with supplementary Excel gene lists contain erroneous gene name conversions”, and that the errors are caused by Excel converting certain gene names into dates, and others into numbers in scientific notation.  A search finds this is nothing new.  A 2004 paper states “Mistaken Identifiers: Gene name errors can be introduced inadvertently when using Excel in bioinformatics”.  The more recent paper has an interesting graph of the frequency of these errors over time:

It seems that the errors were at a low level after the 2004 paper, but have since risen substantially.

Errors of this type are of course not limited to gene names.  Problems with dates being interpreted differently in different regions are widespread, and in the engineering context fractions may be converted to dates, or left as a fraction in text format.

Although  widespread, these problems are reasonably easy to avoid.  This post will look at the built in Excel methods, and the next will present some VBA solutions.  As an example, we will import a table from a pdf file.  The table is copied to the clipboard:txt2cols1-0

When pasted in Excel all the text goes in one column:

txt2cols1-1

The text can be split into columns with the Text to Columns Wizard, under the Data tab, which has three steps.  First select the “delimited” option:

txt2cols1-2

For delimiters select “space” and “treat consecutive delimiters as one”:

txt2cols1-3

Finally select “Text” as the data format for all columns.  To do this in one operation scroll to the right of the data preview, hold down the Ctrl key, and click on the right hand column:

txt2cols1-4

The text is split into columns with text format, so the fractions display as formatted in the original document:

txt2cols1-5

If the same clipboard data is now pasted into another range, Excel remembers the text to columns with space delimiters settings, but not the text format setting, so the integers are pasted as numbers, but the fractions are pasted as either dates or text, depending on whether the fraction can be interpreted as a valid date or not:

txt2cols1-6

To paste the data as text all the cells in the paste range must be formatted as text before pasting.  All the fraction cells will then be pasted in their original format:

txt2cols1-7

The settings selected in the Text to Columns Wizard will remain in place so long as the spreadsheet is open, even if another workbook is opened.  Options for resetting, so that pasted text will go into a single column again are:

  1. Save and re-open the spreadsheet
  2. Or select a single cell containing text, and go through the Text to Columns process, selecting “delimited” but deselecting all delimiters.
Posted in Excel | Tagged , , , , | Leave a comment

A long history of civilisation …

… and climate change:

http://xkcd.com/1732/

earth_temperature_timeline

Posted in Climate, Newton | Tagged , , | 2 Comments