Retrospective

So, here I am on the feather-edge of retirement (I'll be 70 in a few months) and I'm still learning things.  I had an insight last night that kept me awake mulling it.  My last contract was with Bank of America in Texas and, while it was fun, it was also more than just a little frustrating.

When I first started looking at the code I would be working with at BofA, I was confused.  Everybody these days writes 'strictly structured', right?  No, wrong, and that was what was so confusing.  Last night's insight cleared away all the cobwebs...  just in time for Hallowe'en.

 

There are two ways to approach a programming problem.  In the first, you start out by assuming that this is an easy problem;  you have to do A, B, C, and finally D.  Voila!  You sit down and write code and it comes out as a single long module.  It may be fairly complex.  (This was typical for most of the BofA code I saw.)

If, instead, you assume that the problem will be complex, that you have to do A, B, C, and finally D, you will sit down and write a top-level module with stubs for the called subroutines.  Then you will write the innards of the subroutines, probably as routers with stubs for their called subroutines.  This process will continue through n levels until each subroutine is so simple it just doesn't make sense to break it down further.  The resulting program will be longish, but (all things considered) pretty simple regardless of the initial estimate of complexity.

I (almost) always presume a programming task will be complex.  If that turns out to be wrong, no big loss.  If I were to assume some programming task were simple and it turns out not to be quite as simple as I originally thought — that would hurt.  It would hurt because halfway through writing that 'one long module', I would discover the need for the same code I used in lines 47 through 101.  Stop.  Grab a copy of that code.  Create a subroutine at the end of the code.  Insert CALLs at both places.  Continue writing that 'one long module' where you left off.

If that scene happens more than once or twice, what we wind up with is a long main module with several calls to randomly-placed subroutines.  The coefficient of complexity has just been bumped up, and the bump could be quite a lot.  If it's one of the newly-created subroutines whose function needs to be partitioned, the code soon takes on a distinct air of 'disorganization'.

Do I have to point out that there's way too much overly-complex and disorganized code out there and running in production?  No, I probably don't;  we've all experienced Windows.

So, there's a built-in penalty for assuming simplicity, and it turns out this penalty applies (in REXX, at any rate) no matter how complex the eventual program actually is.

If a (REXX) program is written as 'one long module', possibly with a few random subroutines for function required in more than one place, diagnosis becomes a problem.  Unless the programmer has anticipated bypassing iterative loops, a trace will have to endure every iteration in every loop before getting to the next stage.  To avoid this most painful experience, what happens most often with such code is a quick one-time fix to turn TRACE on here and shut it off there.  But then, the program being diagnosed is no longer the program that failed;  it's a modified version of the failing program.

If a (REXX) program is highly-structured, function will be very encapsulated to the point that any error will be isolated to one or a very small number of suspect segments.  Running such a heavily-encapsulated program in trace-mode means that entire trees of logic can be bypassed:  if TRACE is on for a higher-level module, it can be turned off in a submodule (and all its children) but will still be on when control returns to the higher-level module.  The more structured the code, the easier it is to debug.  With one proviso...

You can have a highly-structured program that is nevertheless disorganized.  If, for example, you place your subroutines in alphabetical order, the flow of control will appear chaotic.  Ideally, submodules that are merely segments of an upper-level router should appear in roughly their order-of-execution.  Although they're broken out into separate segments, they still retain the flavor of that 'one long module' insofar as they appear one after the other like the cars of a train.  Reading such code becomes easier because a CALL to a submodule is a call to code which is (probably) physically close by.  (This is not always strictly true.)

COBOL programmers long ago adopted a more-or-less universal convention: they prefix the name of each code segment with a glyph ('D100', perhaps) that indicates its logical position in the complete program.  A COBOL programmer seeing a reference to 'D100-something' in module 'C850-GET-USER-ID' knows to look later in the code for that segment.  The same technique works equally well in all languages, and REXX is not an exception.  (I tend to use alpha-only such that the mainline calls A_INIT, B_something, etc.  Module C_whatever calls CA_blah, CB_blah, etc.  Whatever works...)

Exactly the same sorts of things can be said about modifying an existing program.  The 'one long module' requires careful planning and skillful execution when inserting new function or changes to existing function.  Testing the new function is a chore to the same extent diagnosing an error is a chore, and for the same reasons.  Highly-structured code is designed to be modified;  it was written that way.

Summarizing:  a highly-structured REXX program may be a little longer than it (strictly) has to be, but it will be easier to understand and easier to diagnose in case of an error.  This understanding can be enhanced by strategic naming of segments and by arranging the segments to more closely align with the actual order of execution.

Recommendation:  Structure is your friend.  It may be your best friend.

What's Wrong With This picture?

How many times have you seen this and thought nothing of it?

"EXECIO * DISKR FIRST (STEM FIRST. FINIS"
"EXECIO * DISKR NEXT  (STEM NEXT.  FINIS"
do nn = 1 to next.0
   --parse token from next.nn --
   --35 lines of binary search thru first. --
   if notfound then do
      --diagnostic message--
      end 
   else do
      --process the match--
      end
end 

Not only have I seen this kind of code, I have written this kind of code.  A startling revelation, an epiphany, has rocked my world.

The 'revelation' is this:  in MVS, the first time you say 'READ', you don't just read one record; you read five buffers.  If you're talking about modern DASD and 80-byte records, you've just 'read' something like 1400 or 1500 records. They're all in main storage.  Whatever operation you do to them, you do at main storage speeds.  And "EXECIO * DISKR" doesn't stop at five buffers;  it reads all of it.  All the heavy lifting, the SIOs, has already been done.  You've just spent $11 to read from DASD, and now you propose to save four cents by doing a binary search.  Are we all nuts?

In a situation like this, we should leverage the power of REXX's data-associated arrays by spinning through one of those stems and parsing everything we can find, then use the data-association to the second file to know (intuitively) whether there is or is not a match.  It's all in main storage, right?  You would need highly-sophisticated and very expensive equipment to discern how much time you saved by doing a binary search over using a sequential process.  The cost of having a programmer write those thirty-five lines of binary search will never be paid back by the time saved.

Edsger Dijkstra once proposed that a programmer worrying about how fast (or slow) hir code would execute was worrying about the wrong thing.  "Get a faster computer", he advised.  Easier said than done in many cases, but always the optimal solution.

That's not, however, what we're seeing here.  This is truly "penny-wise and pound-foolish" to do all that I/O and then waste the advantage of having it all immediately accessible (let's face it) for free.

I think I may have written my last binary search.  What do you think?

Embedding ISPF (and other) assets

When I write a tool for myself — for my own use — I will typically include ISPF assets at the bottom of the code and have software extract them as part of the initialization phase.  I rarely load panel text to ISPPLIB or skeletons to ISPSLIB.  There are several advantages to keeping your ISPF assets 'local':

• I/O is reduced, sometimes very substantially reduced
• changes made to these assets are reflected immediately without having to be in TEST-mode and certainly without having to leave ISPF and restart it
• there is no doubt about the identity of subsidiary elements
• there is no danger of duplicate member names
• When distributing or installing, there is only one element to distribute or install: the enclosing REXX code

When ISPF is invoked by a non-developer (call it 'standard mode') its habit is to cache any panels, skeletons, or messages that it uses.  It keeps them in storage so that if the same element is re-used, ISPF can get it from the cache rather than doing I/O to get a fresh copy.  Obviously, if you're modifying that element, saving it to its library won't do a thing for your current session.  To get that new panel, you have to exit ISPF to READY-mode and restart ISPF.  That takes a lot of I/O because on start-up, ISPF opens and reads ISPPLIB, ISPSLIB, ISPTLIB, ISPMLIB, and ISPLLIB so that it knows all the available membernames and where they're located — in case you ask for one of them.

Developers who work with ISPF services generally invoke ISPF in TEST-mode when developing because in TEST-mode, ISPF caches nothing and always does I/O to handle service requests.  If you've just saved a change to a panel and you're in TEST-mode, your next DISPLAY request will retrieve the new version.  The penalty you pay for this is that every service request is handled via I/O.

Embedding your ISPF assets gives you the best of both worlds:  because ISPF caches elements based on DSN+membername, re-extracting ISPF assets at execution-time creates a new dataset (in VIO) and ISPF recognizes that this member XYZ is not the same XYZ as that in the cache, so it reloads a fresh copy.  All other service requests are handled via the cache — because you don't use TEST-mode.

It gets better:  When you invoke the enclosing REXX, it all gets read into storage immediately, and that includes your panels and skeletons.  Extracting them thus happens at 'core speed' and if they're written to VIO datasets, that happens at 'core speed' as well.  No need to read the panel(s) from ISPPLIB or the skeleton(s) from ISPSLIB separately.  It's already here.

You no longer have to search through the libraries to ensure you're not using a duplicate membername — because your data is not going to live in any of those libraries.  The elements embedded in your application are extracted and loaded to a library which is then LIBDEF'd into a preferential position to all others.  If you use a name replicated elsewhere, you will only use your element;  the other will be 'masked' by virtue of being too far down the concatenation.  Of course, when the application ends, LIBDEF tears down those purpose-built libraries and the environment is restored to its pre-invocation state.  Neat.

When you install the application, you install one element.  All the other panels and skeletons used by the main REXX routine do not get separately installed — there's no need to formally install them because they will be regenerated dynamically as and when needed.

If anyone out there can find a 'down-side' to any of this, I'd be very interested to hear it.  To me, it all looks like 'up-side'.

Code for extracting ISPF assets can be found on my REXX Tools page and a short example of how it's implemented at ALIST.

Scrolling a Panel Left and Right

Back on November 3rd, I wrote about "cylindrical arrays" and said "ISPF facilities do not allow the shifting of panels right and left except for some very special IBM-originated panels", but Chris Lewis introduced me to a way to simulate it.  Herewith.

The first thing is to rig the panel so that it can have any column headings you desire and fit it with a variable )MODEL.  The "desc" following is the last line of the )BODY and is where the column headings would be stored.  It is followed by a )MODEL section where the actual model is a variable:

@desc
)MODEL ROWS(SCAN)
&modl

Before the panel can be TBDISPL'd, values for those must be set up...

   /* Set up values for the variable-format panel(s)                 */
   desc1   = "V -Member- Type  A/R Base  -Userid-  ---Date--- Time   Li",
             || "btype   Cleared"
   desc2   = "V -Member- Type  A/R Base  -Userid-  -Username----      Cl",
             || "eared  As"
   modl1   = "_z!mlroot  +!z   !mlbase   !mluid    !mldate    !mltime!ml",
             || "loc    !mlclrdt "
   modl2   = "_z!mlroot  +!z   !mlbase   !mluid    !mlsname           !m",
             || "lclrdt !mlclrnm "

...and loaded to the variables named on the screen:

      modl = Value('modl'style)
      desc = Value('desc'style)

Now, whenever the response from the panel is a PF10 or PF11, call the code that inches the "cylindrical array" left or right:

   select
 
      when pfkey = "PF11" then,
         style = style//stylect + 1    /* next style                 */
 
      when pfkey = "PF10" then,
         style = (style+stylect-2)//stylect + 1       /* prev style  */
 
      otherwise nop
   end

That's all there is to it.  Once STYLE is set based on the PF10 or PF11, that style-number is used to select DESC1/MODL1 or DESC2/MODL2 just before the panel is redisplayed.  It's the same panel (really), but the content displayed there will be different enough that it will appear the panel has actually been shifted.

File Tailoring for Fun and Profit

Creating JCL for running a mainframe task is the most common use for ISPF's File Tailoring — because it does it so well.  Too often, though, such tasks are approached with a 'Bigger Hammer' mentality because the process to be tailored is seen as uncomfortably complex.  For instance:  your installation uses COBOL, PL/I, and ForTran;  some programs use DB2;  some run as part of CICS.  When it comes time to compile something, you may have to run the DB2 preprocessor;  you may have to run the CICS preprocessor;  the compiler must be appropriate to the source language;  if the compile is successful, the object module is to be link-edited;  much of the compiler listing for DB2/CICS modules is crap that does little beyond making the printed version harder to lift.  Is the compiler listing to be saved as a dataset or just printed?  If 'saved', where?  Are you going to trim away all the frou-frou before you print or save?

Faced with such a collection of decisions, most installations opt to have individual PROCs for each potential combination and to name them under a convention that allows the programmer to correctly specify the name of the PROC that does a DB2/non-CICS/PLI compile-and-link.  When a new compiler is released, somebody gets to fix and test a dozen PROCs to insert the updated library information, and it's usually several weeks before anyone realizes that an error sneaked through.  Ouch.

The reason file tailoring is not considered in these instances is because of all those mays above.  The DB2 precompile hands its output to the CICS precompile, but there may not be a DB2 precompile step at all.  The compiler gets its input from the CICS precompile step...  unless there isn't a CICS precompile step.  Where does the input come from then?  There is a solution.

You always start off with an IEFBR14, the safest program in the visible universe:

//ORIGIN   EXEC PGM=IEFBR14
//SOURCE    DD DISP=SHR,DSN=&SRCDS(&SRC)
)SET LSTEP = ORIGIN
)SET LDDN  = SOURCE

Note that this piece of skeleton identifies itself by setting LSTEP and LDDN.  Later steps can use these to find the input they need to focus upon.  Let's posit that if we have to ZORK and PARFLE and FLENCH a program, they always happen in that order, although some (or none) of them may happen at all.

)SEL    &ZORK = Y
//* --------------- ZORK processor --------------     */
//ZORK     EXEC PGM=ZORK
//SYSIN     DD DSN=*.&LSTEP..&LDDN
//SYSPRINT  DD SYSOUT=*
//SHIPOUT   DD UNIT=VIO,SPACE=(CYL,(5,1)),DISP=(,PASS),
//             DCB=(RECFM=FB,LRECL=80,BLKSIZE=0)
)SET LSTEP = ZORK
)SET LDDN  = SHIPOUT
//            IF (ZORK.RC << 8) THEN
)ENDSEL &ZORK = Y
)SEL    &PARFLE = Y
//* --------------- PARFLE processor ------------     */
//PARFLE   EXEC PGM=PARFLV2
//PARFIN    DD DSN=*.&LSTEP..&LDDN
//SYSPRINT  DD SYSOUT=*
//PARFOUT   DD UNIT=VIO,SPACE=(CYL,(5,1)),DISP=(,PASS),
//             DCB=(RECFM=FB,LRECL=80,BLKSIZE=0)
)SET LSTEP = PARFLE
)SET LDDN  = PARFOUT
//            IF (PARFLE.RC << 4) THEN
)ENDSEL &PARFLE = Y
)SEL    &FLENCH = Y
//* --------------- FLENCH processor ------------     */
//FLENCH     EXEC PGM=FLENCH00
//SYSIN     DD DSN=*.&LSTEP..&LDDN
//SYSPRINT  DD SYSOUT=*
//OUTFLEN   DD UNIT=VIO,SPACE=(CYL,(5,1)),DISP=(,PASS),
//             DCB=(RECFM=FB,LRECL=80,BLKSIZE=0)
)SET LSTEP = FLENCH
)SET LDDN  = OUTFLEN
//            IF (FLENCH.RC << 8) THEN
)ENDSEL &FLENCH = Y

If the ZORK processor is activated, it gets its input from '*.ORIGIN.SOURCE' because those are the current settings for LSTEP and LDDN.  When it's complete, it indicates that any later steps should use ZORK.SHIPOUT by setting LSTEP to 'ZORK' and LDDN to 'SHIPOUT'.  If the ZORK processor is not activated, LSTEP remains set at 'ORIGIN' and LDDN at 'SOURCE'.

If only the FLENCH processor is cleared to run, neither of the other two steps have altered LSTEP or LDDN and it picks up the dataset mentioned by the IEFBR14 step as its input.  Whichever combination of steps ran or didn't, the next step downstream will always find its input via '*.&LSTEP..&LDDN'.

Of course, things could be more complicated necessitating a little more finesse, but File Tailoring is, I assure you, up to the task.

What about you?

I'm still reviewing old code looking for techniques that might be either useful or entertaining.  It occurs to me that readers may have tips of their own they'd like to pass along.  Anybody?  Bueller?

INTERPRET?? WDNNS INTERPRET!

(For those who don't know what 'wdnns' means, it is a reference to a misquote of a line from Treasure of the Sierra Madre and for the rest, you can look it up on the web.  The expansion is "We don' need no steenking ..."  'Nuf said.)

It is my firmly held conviction that INTERPRET is almost never required for ordinary REXX processing.  Les Koehler has come up with one or two scenaria where it is useful but they are so bizarre that 'esoteric' is a risible understatement.  Most uses of INTERPRET run something like this:

/*  'parm' looks like  ' name=Smith '   */
parse var parm  tag "=" tagval
interpret tag "=" tagval

I hope I got that right.  Since I never use INTERPRET and I don't have a mainframe to test this out, I have to guess as to what the code would actually look like.  If I were writing this code, I would use VALUE instead.

parse var parm tag '=' tagval
rc = Value(tag,tagval)

I have also seen entire commands constructed piece-by-piece and then INTERPRETed to cause the command to be executed.  Unnecessary.

interpret "ISPEXEC TBCREATE" tblnm blah blah blah

I have never seen a case where the INTERPRET couldn't simply be converted to a straight execution:

"ISPEXEC TBCREATE" tblnm blah blah blah

Anyone with a true instance of a necessary INTERPRET is welcome to present the same here.  I'd love to be proven wrong.