SV: write to file with quick /using quiz instead
Joe Boyle
joeboyle_adt@hotmail.com
Tue, 21 Sep 2004 12:16:09 +0100
alternatively, and assuming that quiz is around 50 times faster than quick
when writing the records( or at least sufficiently faster for you to prefer
to use quiz), you could allow quiz to write the subfile, and then access the
subfile in quick and run it through a while loop processing each record as
you are now. I am confident that updates are always significantly faster
than writes, and over 9 million records the time saving should be
significant.
I think it's the case that the data part of the subfile should be in direct
format already, so it's possible that you needn't use the convert function
at all.
Regards, Joe.
This e-mail and all information contained in it is confidential and may be
legally privileged. If you are not the intended recipient, your access to
this e-mail is unauthorized. Any use, dissemination, distribution,
publication or copying by you of this e-mail or any of the information
contained within it is prohibited and may be unlawful. Do not open any
attachments, delete it immediately from your system and notify the sender
promptly by e-mail that you have done so. The content of this
e-mail and any attachments sent with it may have been altered without the
consent or knowledge of the author.
-----Original Message-----
From: powerh-l-admin@lists.sowder.com
[mailto:powerh-l-admin@lists.sowder.com] On Behalf Of
Jari.Kaljunen@teliasonera.com
Sent: 21 September 2004 11:31
To: powerh-l@lists.sowder.com
Subject: SV: SV: write to file with quick /using quiz instead
Hi
the output record format is not critical, the result file will be used by
other system that don't care.
But what do you recommend to convert the characters &,<,>,' and " to
xml-entities if quiz or qtp is used produce the file?
We don't have controls for these characters in DB now and we get files from
many sources.
We already had a solution with qtp that produced the file but the problem
with these characters was not solved. A collegue succested to use quick for
that purpose (first separate the data from xml-tags with binary characters
and then only check the data part) but the quick performance problem still
remains. DCL was far too slow. Management don't want to install Perl (I
believe that Perl using regular expressions would be fast enough), collegues
don't like the idea of using external C-program...
We appreciate any suggestions.
/Jari
-----Ursprungligt meddelande-----
Från: Joe Boyle [mailto:joeboyle_adt@hotmail.com]
Skickat: den 21 september 2004 12:03
Till: Kaljunen, Jari S. /TSS - Sergel Kredittjänster /+46-8-4526435;
powerh-l@lists.sowder.com
Ämne: RE: SV: write to file with quick /using quiz instead
personally, I would prefer to code the requirements that you list below and
then use 'convert', rather than wait for quick to write 9 million 600 byte
records ( 5 gig ); especially as you say that performance still isn't great
even when using the direct format.
Regards, Joe.
This e-mail and all information contained in it is confidential and may be
legally privileged. If you are not the intended recipient, your access to
this e-mail is unauthorized. Any use, dissemination, distribution,
publication or copying by you of this e-mail or any of the information
contained within it is prohibited and may be unlawful. Do not open any
attachments, delete it immediately from your system and notify the sender
promptly by e-mail that you have done so. The content of this
e-mail and any attachments sent with it may have been altered without the
consent or knowledge of the author.
-----Original Message-----
From: Jari.Kaljunen@teliasonera.com [mailto:Jari.Kaljunen@teliasonera.com]
Sent: 20 September 2004 14:37
To: joeboyle_adt@hotmail.com; powerh-l@lists.sowder.com
Subject: SV: SV: write to file with quick /using quiz instead
Hi,
there will be "report" from 4 different tables. Of course we could have
several quiz-runs producing files with additional sorting fields and then
merge the files omitting sorting fields.
Then there would be 2 problems left: disc space for the intermediate files
and conversion of characters &,<,>,'," to xml-entities which is not very
practical i quiz.
/Jari
-----Ursprungligt meddelande-----
Från: Joe Boyle [mailto:joeboyle_adt@hotmail.com]
Skickat: den 17 september 2004 23:28
Till: Kaljunen, Jari S. /TSS - Sergel Kredittjänster /+46-8-4526435;
powerh-l@lists.sowder.com
Ämne: RE: SV: write to file with quick
why not allow the performance of quiz to generate the subfile and then use
the DCL command 'CONVERT/FDL' to convert the file to the required sequential
format. It may even be the case, that if the 'nodict' qualifier is used when
creating the subfile, then the subfile need only be renamed and copied to
the required directory.
I am guessing that a 'nodict' subfile will have a sequential rather than
direct record format; you could always use the analyse command to examine
the subfile's record format if you aren't sure.
Presumably, once the file is built, subsequent updates will be writing fewer
records into the file, and the slow performance of Quick may not be as great
an issue - or you could even go back to Qtp.
Regards, Joe.
This e-mail and all information contained in it is confidential and may be
legally privileged. If you are not the intended recipient, your access to
this e-mail is unauthorized. Any use, dissemination, distribution,
publication or copying by you of this e-mail or any of the information
contained within it is prohibited and may be unlawful. Do not open any
attachments, delete it immediately from your system and notify the sender
promptly by e-mail that you have done so. The content of this e-mail and any
attachments sent with it may have been altered without the consent or
knowledge of the author.
From: powerh-l-admin@lists.sowder.com
[mailto:powerh-l-admin@lists.sowder.com] On Behalf Of
Jari.Kaljunen@teliasonera.com
Sent: 16 September 2004 20:42
To: powerh-l@lists.sowder.com
Subject: SV: SV: write to file with quick
You mean file asc600 designer open append ?
Well, it didn't help..
/Jari
-----Ursprungligt meddelande-----
Från: Peter Bateman [mailto:ppfb@hotmail.com]
Skickat: den 16 september 2004 21:02
Till: powerh-l@lists.sowder.com
Ämne: RE: SV: write to file with quick
I would try 'file asc600 designer open append' It may be a machine specific
option.
But I didn't see it in the manual though.
Regards,
Peter
>From: <Jari.Kaljunen@teliasonera.com>
>To: <powerh-l@lists.sowder.com>
>Subject: SV: write to file with quick
>Date: Thu, 16 Sep 2004 12:22:20 +0200
>
>Hello again,
>
>a little test without reading database still very slow (actually no
difference...). It takes 49 s to run this with 100 records on subfile. The
result file is 20100 lines.
>
>I even wrote a test quiz reading the same subfile and having 200 defines,
each including systime and some more, each defined item written on an own
line. It takes quiz 3 s to write these 20000 lines.
>
>
>/Jari
>
>
>screen test_writing_file menu stopscreen
>
>file *ik006002 designer
>file asc600 designer
>
>temp t_count
>
>procedure internal write
> begin
> for 200
> begin
> let t_count = t_count + 1
> let rec600 = asc(t_count) &
> + " " + asc(systime) + " procedure internal write "
> put asc600 reset
> end
> end
>
>procedure initialize
> begin
> let t_count = 0
> while retr ik006002 seq
> begin
> let rec600 = asc(t_count) &
> + " " + asc(systime) + " procedure initialize "
> put asc600 reset
> do internal write
> end
> return
> end
>
>build
>
>-----Ursprungligt meddelande-----
>Från: Johnson, Harold A EDUC:EX
>[mailto:Harold.A.Johnson@gems1.gov.bc.ca]
>Skickat: den 15 september 2004 17:50
>Till: Kaljunen, Jari S. /TSS - Sergel Kredittjänster /+46-8-4526435;
>powerh-l@lists.sowder.com
>Ämne: RE: write to file with quick
>
>
>Are you sure the performance problem is with the writing and not with the
>reading of the original data? I would suspect a problem there instead.
>
>
>-----Original Message-----
>From: powerh-l-admin@lists.sowder.com
>[mailto:powerh-l-admin@lists.sowder.com]On Behalf Of
>Jari.Kaljunen@teliasonera.com
>Sent: 2004 September 15 8:41 AM
>To: powerh-l@lists.sowder.com
>Subject: write to file with quick
>
>
>Hello,
>
>I am writing to
>
>File: ASC600
>Organization: SEQUENTIAL
>Type: RMS
>Open: ASC600
>Record Format: Variable
>Supersede: No
>Records & Sizes: ASC600 600 bytes
>Item Type Size Occ Offset
>REC600 CHARACTER 600
>
>in quick using let and put as follows
>
>let rec600 = "<HeaderRef2>" + asc(arnr of arenden,8) + asc(parmblnr,2) &
>+ atgkod of atgkoder + "</HeaderRef2>"
>
>put asc600 reset
>
>
>The performance is very poor. The administrator believs that quick closes
>and opens the file all the time.
>Can that be true?
>What should I do?
>
>We are creating an xml-file (it will be about 5 GB) with data from several
>tables and our solution with qtp needed too many subfiles and sortings
using
>lots of space.
>
>/Jari
>
>= = = = = = = = = = = = = = = = = = = = = = = = = = = =
>Mailing list: powerh-l@lists.sowder.com
>Subscribe: "subscribe" in message body to powerh-l-request@lists.sowder.com
>Unsubscribe: "unsubscribe <password>" in message body to
>powerh-l-request@lists.sowder.com
>http://lists.sowder.com/mailman/listinfo/powerh-l
>This list is closed, thus to post to the list you must be a subscriber.
>
>= = = = = = = = = = = = = = = = = = = = = = = = = = = =
>Mailing list: powerh-l@lists.sowder.com
>Subscribe: "subscribe" in message body to powerh-l-request@lists.sowder.com
>Unsubscribe: "unsubscribe <password>" in message body to
powerh-l-request@lists.sowder.com
>http://lists.sowder.com/mailman/listinfo/powerh-l
>This list is closed, thus to post to the list you must be a subscriber.
Don't just Search. Find! The new MSN Search: Fast. Clear. Easy. = = = = = =
= = = = = = = = = = = = = = = = = = = = = = Mailing list:
powerh-l@lists.sowder.com Subscribe: "subscribe" in message body to
powerh-l-request@lists.sowder.com Unsubscribe: "unsubscribe " in message
body to powerh-l-request@lists.sowder.com
http://lists.sowder.com/mailman/listinfo/powerh-l This list is closed, thus
to post to the list you must be a subscriber.
= = = = = = = = = = = = = = = = = = = = = = = = = = = =
Mailing list: powerh-l@lists.sowder.com
Subscribe: "subscribe" in message body to powerh-l-request@lists.sowder.com
Unsubscribe: "unsubscribe <password>" in message body to
powerh-l-request@lists.sowder.com
http://lists.sowder.com/mailman/listinfo/powerh-l
This list is closed, thus to post to the list you must be a subscriber.