Jesse, If one end of the pipe fails for any reason, then both ends would have to be rerun. The redbook "Batch Modernization on z/OS" chapter 16 explains the use of Batch pipes. Martin Packer is one of the co-author.
http://www.redbooks.ibm.com/abstracts/sg247779.html Thanks, Kolusu From: Jesse 1 Robinson <jesse1.robin...@sce.com> To: IBM-MAIN@LISTSERV.UA.EDU Date: 11/21/2017 03:13 PM Subject: Re: [TSO-REXX] Fwd: Pipelines in the z/OS base. Sent by: IBM Mainframe Discussion List <IBM-MAIN@LISTSERV.UA.EDU> I have never worked with pipes in a production batch environment. In a classic batch job, if a problem occurs somewhere in the middle, it's common to restart the job at some recoverable point after fixing the downstream problem. I cannot visualize how to restart a pipes job in such a case. Without intermediate files to resume with, how does all that in-memory data get recreated without starting back at the top? In a short or medium length job, starting over may not be such a big problem, but for a job that runs hours on a good day, I could envision some major production schedule glitches with pipes. . . J.O.Skip Robinson Southern California Edison Company Electric Dragon Team Paddler SHARE MVS Program Co-Manager 323-715-0595 Mobile 626-543-6132 Office ⇐=== NEW robin...@sce.com -----Original Message----- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of Sri h Kolusu Sent: Tuesday, November 21, 2017 1:54 PM To: IBM-MAIN@LISTSERV.UA.EDU Subject: (External):Re: [TSO-REXX] Fwd: Pipelines in the z/OS base. Hi, Here is a sample DFSORT job that reads a pipe and does a sort while modifying the data using inrec(before sorting) Outrec(after sorting) and Outfil //PIPEDEMA JOB (DA26,001,098,J69),'KOLUSU', // CLASS=A, // MSGCLASS=H, // MSGLEVEL=(1,1), // NOTIFY=&SYSUID //* //************************************************************** //* DELETE THE INPUT PIPE DATASET IF IT EXISTED ** //************************************************************** //STEP0001 EXEC PGM=IEFBR14 //SYSPRINT DD SYSOUT=* //DD1 DD DSN=&SYSUID..PIPE.INPUT, // UNIT=SYSDA,SPACE=(TRK,0),DISP=(MOD,DELETE) //* //************************************************************** //* CREATE THE INPUT TO SORT AND WRITE IT TO A PIPE ** //* WILL HAVE 60 RECORDS (20 RECORDS FOR EACH NAME) ** //************************************************************** //PIPERCRE EXEC PGM=SORT //SYSOUT DD SYSOUT=* //SORTIN DD * PAUL MARTIN HOBART //SORTOUT DD DSN=&SYSUID..PIPE.INPUT, // SUBSYS=(BP01,'WAITOPEN=180'), // DCB=(LRECL=80,RECFM=FB,BLKSIZE=27920) //SYSIN DD * OPTION COPY OUTFIL REPEAT=20,OVERLAY=(20:SEQNUM,3,ZD,RESTART=(1,10)) /* //PIPEDEMB JOB (DA26,001,098,J69),'KOLUSU', // CLASS=A, // MSGCLASS=H, // MSGLEVEL=(1,1), // NOTIFY=&SYSUID //* //************************************************************** //* READ THE PIPE AND USING INREC MODIFY MARTIN RECORDS ** //* SORT THE DATA ON NAME AND SEQUENCE DESCENDING ** //* MODIFY PAUL RECORDS USING OUTREC ** //* MODIFY HOBART RECORDS USING OUTFIL ** //************************************************************** //PIPEREAD EXEC PGM=SORT //SYSOUT DD SYSOUT=* //SORTDIAG DD DUMMY //SORTIN DD DISP=SHR,DSN=&SYSUID..PIPE.INPUT, // SUBSYS=(BP01,'WAITOPEN=180'), // DCB=(LRECL=80,RECFM=FB,BLKSIZE=27920) //SORTOUT DD SYSOUT=* //SYSIN DD * INREC IFTHEN=(WHEN=(01,10,CH,EQ,C'MARTIN',AND, 20,03,SS,EQ,C'005,010,015,020'), OVERLAY=(31:C'EVERY 5TH RECORD TAGGED USING INREC')) SORT FIELDS=(01,10,CH,A, $ SORT ON NAME ASC 21,03,ZD,D) $ SORT ON SEQ DESC OUTREC IFTHEN=(WHEN=(01,10,CH,EQ,C'PAUL',AND, 20,03,SS,EQ,C'006,012,018'), OVERLAY=(31:C'TAGGED EVERY 6TH RECORD USING OUTREC')) OUTFIL IFTHEN=(WHEN=(01,10,CH,EQ,C'HOBART',AND, 20,03,CH,EQ,C'020'), OVERLAY=(31:C'RECORD 20 IS TAGGED USING OUTFIL')) /* Further if you have any questions please let me know Thanks, Kolusu DFSORT Development IBM Corporation IBM Mainframe Discussion List <IBM-MAIN@LISTSERV.UA.EDU> wrote on 11/21/2017 01:05:40 PM: > From: Hobart Spitz <orexx...@gmail.com> > To: IBM-MAIN@LISTSERV.UA.EDU > Date: 11/21/2017 01:06 PM > Subject: Re: [TSO-REXX] Fwd: Pipelines in the z/OS base. > Sent by: IBM Mainframe Discussion List <IBM-MAIN@LISTSERV.UA.EDU> > > Martin wrote: > > A DFSORT stage would/could/should allow all the myriad* DFSORT > > control > statements. And be for heavy lifting. Those are the two reasons I suggested > it. > > I was going to do some details research (my DFSORT knowledge is > probably out of date, for which I take responsibility), but time is > short, and I don't want the issue to get stale. I believe you are > referring to INCLUDE/OMIT, INREC/OUTREC, etc. > > I hope this answers your point. I don't know if your statement is > true, or > will ever be implemented. I will say that Pipelines offers so many > more options in record selection, formatting, conversion and field > specification, that what you suggest probably will never need to be > implemented. > > Here's why: > > - Of the 300+ Pipelines stages, many are selection stages. In this case, > "many" is much more than myriad, AFAIK, by an order of magniture. > There are simple selectors, but also complex ones that support > conversions, logic, arithmetic, and reformatting. > > - Fields can be identified by character, word, or delimited field. > Each of > these can used as substring to an enclosing specification, to multiple > levels. E.g., take the second slash delimited subfield in the seventh tab > delimited field to get a numeric month value. And you are not limited to > blanks as white space or tabs as delimiters. > > - The DFSORT program product control statements can only be used in > connection with the sort process. Pipelines stages can be used even when > you are not sorting. In other words, Pipelines skills, while being > easy to > learn (unlike my recollection of DFSORT control statement), they have > a vastly broader range of applicable uses. > > - The DFSORT program product E15 and E35 internal sort exits can only > be used once per sort. Pipelines stages bring this efficiency to each > and every stage in a Pipeline. To press the point, unlike the need > for record > and field definitions in, say, a COBOL internal sort exit, the intermediate > records don't have to be defined, and fields only have to be > referenced if > they are actually involved in the a particular stage. > > - Finally, you can process unselected records without having to reprocess > the entire input file, as shown below. > > Business problem: You have an externally produced name and address > file that has to be sorted by ZIP code to get a USPS discount rate. > There are > two types of records, business and residential. The fields are free format > and tab delimited. The business records differ from residential > records in > that there are additional fields: Company after the name fields and > an > ATTN: field after the ZIP code. This would be trivial in pipes if the ZIP > code were the last field, but we can't use that because of the ATTN: field. > > A pipes approach is as follows: Separate the records into streams and > prefix each record with its ZIP code. Then sort by the ZIP in the > first field, discard the prefix ZIP. It would look something like > this in commented REXX. ( Visualizing someone else's multi-stream pipe > is easier with a flow chart.) > > ... > /*--------------------------------------------------------*/ > /* f200: f800: */ > /* +-+ +-+ +----+ +-+ +------+ +----+ +-+ */ > /* |<|-->|p|-->|spec|-->|f|-->|dfsort|-->|chop|-->|>| */ > /* +-+ |i| +----+ |a| +------+ +----+ +-+ */ > /* |c| |n| */ > /* |k| +----+ |i| */ > /* | |-->|spec|-->|n| */ > /* +-+ +----+ +-+ */ > /*--------------------------------------------------------*/ > "pipe (listerr name ZIPSort.exec end ? ", > "| < in.file ", > "| f200: pick 1.1 == /R/", /* Residential records */ > "| specs field 8 1 1-* nextfield", /* ZIP pfx */ > "| f800: fanin ", /* Concat residential and business */ > "| dfsort option nolist sort fields=(5,10,cha,a)", > , /* NOT compound stage causes CHOP to keep the orig. rcd.*/ > "| not chop after 05", /* Trim prefix key and tab.*/ > "| > out.file */ > > "? f200: ", /* Business record come here. */ > "| specs field 9 1 1-* nextfield", /* ZIP pfx */ > "| f800: ", /* Back to first stream. */ > ... > > > I hope this helps. Even if there are features new (to me) of DFSORT would > make some of my statements incorrect, I'm quite sure that the > remaining ones would still drive the point home easily. > > On Sat, Nov 18, 2017 at 3:01 AM, Martin Packer <martin_pac...@uk.ibm.com> > wrote: > > > A DFSORT stage would/could/should allow all the myriad* DFSORT > > control statements. And be for heavy lifting. Those are the two > > reasons I suggested > > it. > > > > And note a DFSORT stage could be tested outside of a pipeline. > > > > Also note Pipelines could orchestrate multiple DFSORT invocations in > > a chain. > > > > Cheers, Martin > > > > * If you don’t think the word “myriad” is appropriate RTFM. :-) The point > > is there’s lots of processing capability there. > > > > Sent from my iPad > > > > > On 17 Nov 2017, at 19:40, Hobart Spitz <orexx...@gmail.com> wrote: > > > > > > Let me clarify a couple of things. > > > > > > The PIPEs SORT stage only uses virtual storage. The DFSORT stage > > > is used > > > when your data is larger than the virtual storage available. > > > DFSORT uses > > > disk also, and runs a bit slower. > > > > > > TSO Pipelines and BatchPipesWorks are two versions/names of > > > CMS/TSO Pipelines that runs under z/OS. I have used both at a > > > number of sites. > > > AFAIK, they are still available. They are both chargeable. > > > Someone will > > > correct me if I'm wrong, but NetView pipes is a different animal > > > and I > > > doubt it has any significant bearing on this discussion. > > > > > > > > > On Fri, Nov 17, 2017 at 2:29 PM, Jesse 1 Robinson < > > jesse1.robin...@sce.com> > > > wrote: > > > > > >> Ah Netview Access. Some years ago we had a SHARE (JES2) Singalong ditty > > >> entitled the Netview Bunch. Sung of course to tune of the Brady Bunch. > > All > > >> about a family (of software products) bound together without a common > > >> parent but designated siblings through a marriage of convenience. > > >> > > >> But alas, I also doubt that Netview Pipes could be implemented outside > > of > > >> the Netview environment, of which SA is one. And if you don't already > > have > > >> Netview for other business purposes, acquiring it just for Pipes might > > be a > > >> tough sell. > > >> > > >> . > > >> . > > >> J.O.Skip Robinson > > >> Southern California Edison Company Electric Dragon Team Paddler > > >> SHARE MVS Program Co-Manager > > >> 323-715-0595 Mobile > > >> 626-543-6132 Office ⇐=== NEW > > >> robin...@sce.com > > >> > > >> > > >> -----Original Message----- > > >> From: IBM Mainframe Discussion List [ mailto:IBM-MAIN@LISTSERV.UA.EDU] > > On > > >> Behalf Of Farley, Peter x23353 > > >> Sent: Friday, November 17, 2017 11:07 AM > > >> To: IBM-MAIN@LISTSERV.UA.EDU > > >> Subject: (External):Re: [TSO-REXX] Fwd: Pipelines in the z/OS base. > > >> > > >> Dumb question of the day: Assuming you have NetView installed, > > >> can the > > >> NetView pipes implementation be used outside of NetView? E.G., > > >> in a > > batch > > >> step that executes IRXJCL or IKJEFTxx to invokes a Rexx program? > > >> > > >> Based on the discussion so far I am guessing probably not, but thought > > it > > >> was worth throwing out the question. > > >> > > >> I am totally ignorant of NetView functionality other than using NetView > > >> Access for certain 3270 terminal logons, so please be gentle. > > >> > > >> Peter > > >> > > >> -----Original Message----- > > >> From: IBM Mainframe Discussion List [ mailto:IBM-MAIN@LISTSERV.UA.EDU] > > On > > >> Behalf Of Jesse 1 Robinson > > >> Sent: Friday, November 17, 2017 1:26 PM > > >> To: IBM-MAIN@LISTSERV.UA.EDU > > >> Subject: Re: [TSO-REXX] Fwd: Pipelines in the z/OS base. > > >> > > >> I suspected that this function belonged to Netview, but as you noted, it > > >> was implemented here only during SA installation. I brought it up > > because > > >> there is obviously native z/OS code involved. Porting that to TSO might > > be > > >> a tad easier than lifting code across from z/VM. > > >> . > > >> . > > >> J.O.Skip Robinson > > >> Southern California Edison Company Electric Dragon Team Paddler > > >> SHARE MVS Program Co-Manager > > >> 323-715-0595 Mobile > > >> 626-543-6132 Office ⇐=== NEW > > >> robin...@sce.com > > >> > > >> > > >> -----Original Message----- > > >> From: IBM Mainframe Discussion List [ mailto:IBM-MAIN@LISTSERV.UA.EDU] > > On > > >> Behalf Of Nims,Alva John (Al) > > >> Sent: Friday, November 17, 2017 7:09 AM > > >> To: IBM-MAIN@LISTSERV.UA.EDU > > >> Subject: (External):Re: [TSO-REXX] Fwd: Pipelines in the z/OS base. > > >> > > >> Thank you for "PIPing" up on this! :) > > >> > > >> One thing to note, the PIPE command is part of the Base NetView product, > > >> you do not have to add SA to get it. > > >> -- ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
