Extending the PersonalMoney Application

 

Part 6

 

 

Objectives and Overview

 

This exercise revisits binary filing to put in versioning that should have gone in before, addressing multiple versions of the application.  A simple import and export facility is built to transfer data using the Quicken Interchange Format (qif).  The editable combo box is expanded to search for partial user-typed entries.

 

 

Binary filing revisited

 

In Part 4, as part of implementing bi-directional sorting, we looked at versioning instances of ListViewColumn.  We changed the class definition, adding an instance variable, then overrode two class-side methods, #stbVersion and #stbConvertFrom:, so that instances previously filed out could be reloaded with the binary filer (STBFiler).  While that's well and good, we did not do the same for PersonalMoney's various model classes which have gone through internal structural changes since the original Object Arts Tutorial.

 

In Part 1 we added 'accountTypes' to PersonalMoney and 'accountType' to PersonalAccount.  In Part 5 we added 'categories' to PersonalAccount and 'category' to PersonalAccountTransactions.  This means that if you had created some data in the original PersonalMoney application and then filed it out, you could not read it into the current version of PersonalMoney.  Likewise, if you had created some data in the version following the Part 1 changes, you could not read it in now.  In the case of PersonalMoney data this might not be a big thing, but if this had been a deployed application, you might be getting calls from different people saying "What gives ... and where's the fix?" and in fact you'd need to be fixing each and every versioning problem.

 

There's a third versioning problem and that's for users who created data files with Dolphin version 2.1.  Although this tutorial is geared for version 3 and above, it's possible that someone with version 2.1 created data, saved it, and now wants to read that into the latest PersonalMoney application.  Again, for a deployed application, this is a reasonable scenario and something you'd have to deal with.  Here are the instance variables for class PersonalMoney for the various stages.

 

 

This shows that in the original tutorial, PersonalMoney had two instance variables -- owner and accounts.  Dolphin version 3 added 'events' in class Model, so that is inherited by PersonalMoney -- in slot 1.  Then in Part 1 of this tutorial we added 'accountTypes'.  These different versions of the model are indicated by the column 'class #stbVersion', though we haven't implemented version 2 yet.  #stbVersion 0 is the default and that is what's found in Dolphin 2.1.  Dolphin 3 sets stbVersion to 1 and this is what is inherited by PersonalMoney so far.

 

Let's say we have a data file from Dolphin version 2.1.  If we just read it back into Dolphin 2.1, the version is zero and no conversion takes place.  The binary filer reads in the instance, finds two values, creates a PersonalMoney instance and assigns the two values to the two instance variables.  As for reading the same file into Dolphin verion 3, take a quick look at the conversion method in class Model, Dolphin 3:

 

Model, class side:

stbConvertFrom: anSTBClassFormat

        "Convert from earlier version models.

        1: Added 'events' instance variable to Model."

 

        ^[:data | | newInst |

                 newInst := self basicNew.

                 data keysAndValuesDo: [:i :v | newInst instVarAt: i+1 put: v].

                 newInst]

 

In this case, the filer reads in two values, but the new instance will have three slots.  The method above essentially rightshifts the values.  'data' is an array that contains the values read in by the binary filer.  The first value goes into slot 2 (owner) and the second value goes into slot 3 (accounts).  'events' is left at nil, which is a valid value (for events), so no initialization is needed.

 

The key things to remember when overriding #stbConvertFrom: are that a block is returned.  The block will be passed an argument, 'data', an array containing the values in the binary file, which may or may not represent an old class version.  The code in the block needs to create a new blank instance of the receiver (the model), copy the data values from 'data' into the new instance, initialize or change values in the new instance as needed, and then return the new instance.

 

So what changes do we want to make for the next version, version 2.  We want to use the existing code so that version 0 data will be rearranged and copied (from data to the newInst) and so that version 1 data will be copied from data to the newInst.  Then, since the 'data' array will not have any values for 'accountTypes', but newInst does have a slot for it, we need to initialize the value in slot 4.  This looks like so:

 

In PersonalMoney, class side:

stbVersion

        "Answer the current binary filer version number for instances of the receiver."

 

        ^2

 

stbConvertFrom: anSTBClassFormat

        "Convert from earlier version models.

        1: Added 'events' instance variable to Model.

        2: Added 'accountTypes' instance variable."

 

        ^[:data | | newInst ver offset |

                 newInst := self basicNew.

                 ver := anSTBClassFormat version.

                 offset := ver < 1 ifTrue: [1] ifFalse: [0].

                 1 to: data size do: [:i | newInst instVarAt: i+offset put: (data at: i)].

                 "Perhaps use lazy initialization instead of the next statement."

                 newInst accountTypes isNil ifTrue: [

                         newInst instVarAt: 4 put:

                                 (ListModel with: OrderedCollection new searchPolicy: SearchPolicy equality)].

                 newInst]

 

A few notes:  We add a line to the comment, indicating the change(s) for this version.  We extract the version ('ver') from anSTBClassFormat, which is passed in as a parameter.  We set up an 'offset' local variable which we use to shift the data values into the new instance variable, if needed.  If version is zero, data will be shifted, otherwise it was already shifted before writing out to file.  Then we copy the values from the data array into the appropriate slots in newInst.  At this point we test to see if newInst accountTypes isNil, which it will be if we're reading in data filed from versions 0 or 1, so we initialize it here.  Note that we need to use the method #instVarAt:put: since there's no public method in PersonalMoney to assign accountTypes.

 

Note also the comment on Lazy Initialization.  My first thoughts were to leave accountTypes uninitialized, simplifying this method.  Then move the initialization code for accountTypes from PersonalMoney>>#initialize to the accountTypes accessor method.  However, following the suggestion and reasoning in the Pattern for Lazy Initialization, I put the code in here, duplicating what is in #initialize.  I do still wonder about this -- I know there are those who swear by Lazy Initialization.

 

Now we look at PersonalAccountTransaction, which is very similar to PersonalMoney.

 

 

Version 0 had 4 instance variables. Version 1 (Dolphin verion 3) added an 'events' instance variable and verion 2 (tutorial Part 5) added a 'category' instance variable.  So we essentially do the same thing here that we did in PersonalMoney class>>#stbConvertFrom: except that since category can be nil, we don't even bother initializing it.  Here are the new methods:

 

In PersonalAccountTransaction, class side:

stbVersion

        "Answer the current binary filer version number for instances of the receiver."

 

        ^2

 

stbConvertFrom: anSTBClassFormat

        "Convert from earlier version models.

        1: Added 'events' instance variable to Model.

        2: Added 'category' instance variable."

 

        ^[:data | | newInst ver offset |

                 newInst := self basicNew.

                 ver := anSTBClassFormat version.

                 offset := ver < 1 ifTrue: [1] ifFalse: [0].

                 1 to: data size do: [:i | newInst instVarAt: i+offset put: (data at: i)].

                 ver < 2 ifTrue: ["leave 'category' nilled."].

                 newInst]

 

Finally, we look at PersonalAccount, which is similar to the previous two model classes, though a bit more complex.

 

 

PersonalAccount started out with 5 instance variables in version 0 (Dolphin 2.1).  Version 1 saw the addition of 'events', and then 'accountType' was added in Tutorial Part 1 and 'categories' was added in Tutorial Part 5.  Note that 'accountType' was added in the middle of the list of instance variables (slot 3), rather than at the beginning or the end -- this complicates things a bit but it's not too bad.

 

In PersonalAccount, class side:

stbVersion

        "Answer the current binary filer version number for instances of the receiver."

 

        ^3

 

stbConvertFrom: anSTBClassFormat

        "Convert from earlier version models.

        1: Added 'events' instance variable to Model.

        2. Added 'accountType' instance variable.

        3: Added 'categories' instance variable."

 

        ^[:data | | newInst ver offset |

                 newInst := self basicNew.

                 ver := anSTBClassFormat version.

                 "Copy values from data to newInst.  Shift by 1, for events, if ver = 0."

                 offset := ver < 1 ifTrue: [1] ifFalse: [0].

                 1 to: data size do: [:i | newInst instVarAt: i+offset put: (data at: i)].

 

                 (ver < 2 and: [(newInst instVarAt: 5) isLiteral not]) ifTrue: [

                         "shift data and nil out 'accountType' in 4th slot."

                         self instSize-1 to: 4 by: -1 do: [:i | newInst instVarAt: i+1 put: (newInst instVarAt: i)].

                         newInst instVarAt: 4 put: nil].

                 ver < 3 ifTrue: ["initialize categories."

                         newInst instVarAt: 8 put:

                                 (ListModel with: SortedCollection new searchPolicy: SearchPolicy equality)].

                         newInst]

 

Most of this is pretty much the same as in the previous two model classes.  The tricky part is the section starting with "(ver < 2 and: [(newInst instVarAt: 5) isLiteral not])".  If someone had saved data in Dolphin 2.1 or Dolphin 3 (before the changes in Tutorial Part 1) then we want to shift initialBalance, transactions, and currentBalance one slot to the right to make room for accountType.  However, if they had saved data after making the changes in Tutorial Part 1, the saved file would have a version of 1, but it would have accountType data in the proper slot.  #isLiteral returns false if transactions data (a ListModel) is in the fifth slot, in which case we do need to shift the columns to the right, then nil out the old data in the fourth slot (accountType).  Note that we go backwards, using #to:by:do:, decrementing by 1 (by: -1) to perform the right-shift.

 

At this point, you can read in almost any file that was created in a prior version of PersonalMoney.  The exception is those files with sorted data that were created prior to the fix to #stbSaveOn: (in Part 3).  By this I mean the case where a listview column is clicked (which sorts the data) and then the data is saved to file.  These files, as far as I can see, are not readable.  I won't even go into what I've tried -- if you look (using a hex editor or even notepad) at a binary file with the data sorted, you'll see a whole shipload of stuff, I assume brought in by a block closure or two.  The fix in Part 3 prevents the problem but does not offer a solution.  It seems like somewhere it would be necessary to bypass all the gunk, picking out the data and making it an OrderedCollection, but I couldn't figure out the where or how.

 

The bottom line AND the big lesson on all of this is that if you're using the binary filer, make sure you update the #stbVersion and #stbConvertFrom: class-side methods whenever changes are made to the class definition.  Also, make sure you use #stbSaveOn: to convert any sorted collections back to ordered collections before filing them out.

 

 

Streams, FileStreams and QIF Data

 

In brief, a stream (usually a ReadStream, WriteStream or FileStream) provides a simple and consistent protocol that allows you to access an underlying sequenceable collection (string, file, or byte array).  An excellent primer on Smalltalk streams can be found in the IBM Smalltalk Tutorial, Chapter 8.  The Dolphin hierarchy of stream and filestream classes differs from that of IBM, but there is effectively little difference -- most of the methods and usage are the same.

 

The minimum requirement for qif file transfer (in and out of PersonalMoney) is to be able to write an account's transactions to file so that it can be read into Quicken and also to be able to read an account's transactions that were written to file from within Quicken. 

 

Below is a file (22 lines) that was generated by Quicken.  The account has an opening balance of $500 and two transactions.  The first line is a Quicken 'export header' and identifies the account type, in this case a Bank account.  This is followed by three transactions, including the opening balance.  Each transaction consists of a number of lines ending with a circumflex (^).  Each line starts with a single character, identifying the type of information, and then the information itself.

 

!Type:Bank

D8/11/99

U500.00

T500.00

CX

POpening Balance

L[Wells Fargo]

^

D3/ 3' 0

U-80.00

T-80.00

N101

PBell Market

LGroceries

^

D9/15' 0

U100.00

T100.00

NATM

PDeposit

LReimbursement

^

 

The single character codes have the following meanings:  'D' stands for date, 'U' and 'T' stand for amount (I'm not sure what the difference is, the Quicken documentation is unclear), 'C' stands for transaction cleared, 'P' stands for payee or description, 'L' stands for transaction category, and 'N' stands for the check number or reference.

 

For now we can ignore some of these lines and just process the ones we're interested in.  Specifically, we look for date, amount, description, category, and end of item.  These correspond to codes D, T, P, L, and ^.  These are the items we'll read in to the model as well as write out.  Export headers, check numbers, 'U' amounts, and other coded information we'll make use of at a later date.

 

Starting with the export side of the business, look at #fileSave and #fileSaveAs in DocumentShell for some ideas on how to prompt for a filename, then save data to file.  These methods use FileSaveDialog to prompt for the filename -- if it's not nil then a stream is opened on the file, another method is called to write the data out, and the file is closed.  An exception is raised if there is a problem with the file.  Combining these for our purposes, we get:

 

In PersonalAccountShell:

accountExport

        "Prompts for a filename, then exports the receiver to file."

 

        | filename stream |

        filename := (FileSaveDialog on: self model name)

                 fileTypes: #( ('Quicken Files (*.qif)' '*.qif') ); defaultExtension: 'qif';

                 caption: 'Export Document As'; showModal.

        [filename notNil ifTrue: [

                 stream := FileStream write: filename.

                 [self streamOutQIF: stream] ensure: [stream close]]

        ] on: FileException do: [:e |

                 MessageBox errorMsg: 'Unable to export file ',

                         e file name printString caption: 'Error - ', e messageText]

 

This method creates a FileStream on the user-supplied filename, then tells itself to stream out qif data to that stream, ensuring that the stream is closed (upon success or failure).  The first cut at implementing #streamOutQIF: is simple -- just have each transaction stream itself to the filestream:

 

In PersonalAccountShell:

streamOutQIF: aStream

        "Write receiver's transactions out to aStream."

 

        self transactions do: [:trans | trans streamOutQIF: aStream]

 

In PersonalAccountTransaction:

streamOutQIF: aStream

        "Write receiver's data out to aStream in Quicken (QIF) format."

 

        aStream

                 nextPut: $D; nextPutAll: self date displayString; cr;