Wild Card/Generic retrieval
Deskin, Bob
Bob.Deskin@Cognos.COM
Thu, 11 Sep 2003 09:12:03 -0400
This message is in MIME format. Since your mail reader does not understand
this format, some or all of this message may not be legible.
------_=_NextPart_001_01C37866.4ADD72D0
Content-Type: text/plain
Thanks Darren. This has been quite educational IMHO.
Bob
-----Original Message-----
From: Darren Reely [mailto:darren.reely@latticesemi.com]
Sent: Wednesday, September 10, 2003 4:57 PM
To: Deskin, Bob
Cc: 'Edis, Robert'; Powerh-L (E-mail)
Subject: Re: Wild Card/Generic retrieval
That cursor example is what I did not include, but mentioned. The time
was 2 minutes 47 seconds. I just reran the test and got 2 minutes 36
seconds. I did find this a surprising result. After all, you'd expect
the query to filter out rows that the other tests have to go through.
Understanding that Quiz is doing a little more work I tried to sway
Oracle to do the same via my subselect example. Somehow Oracle 'saw'
through that, or worked much more efficiently at its low levels. Quiz on
the other hand still fetches real data from Oracle before applying the
SELECT, taking up crucial time. Oh, and Quiz is placing that fetched
data somewhere else also.
Here is a new faster version of the cursor sample. It takes 10 seconds.
set report nolimit
sql in mfg declare lss cursor for select lot_id from lot_step_summ
where lot_id like '%A'
access lss
define mycount num = 1
final footing mycount subtotal
This suggests that when fetching data, you should only fetch what you
need, not everything, as the SELECT * does. Temporary Quiz storage
concerns are not being taxed as much in this new example.
My testing was a simple curiosity experiment. Real reporting could alter
the results somewhat. Still, I know to avoid MATCHPATTERN, and SELECT *
when not required.
Darren
Deskin, Bob wrote:
> Well, QUIZ is retrieving each individual row from the database and
> then applying the selection. The SQL is letting the database do all of
> the selection before returning anything.
>
> How about trying this:
>
> declare x cursor for select select * from lot_step_summ where lot_id
> like '%A'
> access x
> define mycount num = 1
> final footing mycount subtotal
> go
>
> or something similar.
>
> Bob
>
> -----Original Message-----
> From: Edis, Robert [mailto:REdis@Blistex.com]
> Sent: Wednesday, September 10, 2003 3:37 PM
> To: Powerh-L (E-mail)
> Subject: RE: Wild Card/Generic retrieval
>
> What IS an issue is 4 sec. for SQL vs. 53 sec for Quiz. Why is
> Quiz so slow?
>
> -----Original Message-----
> From: Deskin, Bob [mailto:Bob.Deskin@cognos.com]
> Sent: Wednesday, September 10, 2003 2:25 PM
> To: 'Darren Reely'
> Cc: powerh-l@lists.swau.edu
> Subject: RE: Wild Card/Generic retrieval
>
> Thanks Darren for confirming that pattern matching is still
> quite resource intensive relative to the other PowerHouse
> functions. While we can't say exactly how much more intensive
> without testing various volumes, I'm not surprised at these
> results. What is interesting is how close the other function
> tests are.
>
> Bob
>
> -----Original Message-----
> From: Darren Reely [mailto:darren.reely@latticesemi.com]
> Sent: Wednesday, September 10, 2003 2:57 PM
> Cc: powerh-l@lists.swau.edu
> Subject: Re: Wild Card/Generic retrieval
>
>
> I found this interesting enough to 'play' a little with it. I
> choose the
> main table in our Oracle test system containing 1,482,455
> rows. I
> altered the queries for my known data and limited them to
> counting.
>
> In this scenario it is obvious to avoid MATCHPATTERN. I've
> never been
> impressed with its performance. Most of the options proposed
> came in at
> around 53 seconds, MATHCPATTERN added 2 minutes to those times!
>
> Not listed here is a combination of the Oracle SQL select as a
> cursor
> for a new version of the quiz file. It took 2 minutes 47 seconds.
>
> At the end of each select statement are the timings.
>
> Oracle SQL - approximately 4 seconds
>
> select count(*)
> from lot_step_summ
> where lot_id like '%A';
>
> OR
>
> select count(*)
> from (select * from lot_step_summ where lot_id like '%A');
>
>
> COUNT(*)
> ----------
> 145878
>
>
> Powerhouse times
>
> set report nolimit
> access lot_step_summ in mfg
> ;select lot_step_summ if MATCHPATTERN(TRUNCATE(lot_id),"@A")
> ;; 2
> minutes 53 seconds
> ;select lot_step_summ If 0 = index(lot_id,"A") -
> size(truncate(lot_id))
> ;; 53 seconds fetches too many records, no surprise
> ;select lot_step_summ if 'A' = REVERSE(TRUNCATE( lot_id
> ))[1:1] ;; 53
> seconds
> select if "A" = lot_id[ SIZE(TRUNCATE(lot_id)) :
> 1] ;; 56 seconds
> define mycount num = 1
> final footing mycount subtotal
> go
>
> 09/10/2003
> mfg
> PAGE 1
>
> MYCOUNT
> (SUBTOTAL)
>
> 145878
>
>
> Darren
>
>
> = = = = = = = = = = = = = = = = = = = = = = = = = = = =
> Mailing list: powerh-l@lists.swau.edu
> Subscribe: "subscribe" in message body to
> powerh-l-request@lists.swau.edu
> Unsubscribe: "unsubscribe <password>" in message body to
> powerh-l-request@lists.swau.edu
> http://lists.swau.edu/mailman/listinfo/powerh-l
> This list is closed, thus to post to the list you must be a
> subscriber.
>
> Join us at Cognos' biggest event of the year Enterprise 2003,
> The Cognos Business Forum. Taking place in over 25 cities
> around the world, it's an opportunity for Business and IT
> leaders to learn about strategies for driving performance.
> Visit http://www.cognos.com/enterprise03 for more details.
>
> This message may contain privileged and/or confidential
> information. If you have received this e-mail in error or are
> not the intended recipient, you may not use, copy, disseminate
> or distribute it; do not open any attachments, delete it
> immediately from your system and notify the sender promptly by
> e-mail that you have done so. Thank you.
>
>
> Join us at Cognos' biggest event of the year Enterprise 2003, The
> Cognos Business Forum. Taking place in over 25 cities around the
> world, it's an opportunity for Business and IT leaders to learn about
> strategies for driving performance. Visit
> http://www.cognos.com/enterprise03 for more details.
>
>
> This message may contain privileged and/or confidential information.
> If you have received this e-mail in error or are not the intended
> recipient, you may not use, copy, disseminate or distribute it; do not
> open any attachments, delete it immediately from your system and
> notify the sender promptly by e-mail that you have done so. Thank you.
>
Join us at Cognos' biggest event of the year Enterprise 2003, The Cognos
Business Forum. Taking place in over 25 cities around the world, it's an
opportunity for Business and IT leaders to learn about strategies for
driving performance. Visit http://www.cognos.com/enterprise03 for more
details.
This message may contain privileged and/or confidential information. If you
have received this e-mail in error or are not the intended recipient, you
may not use, copy, disseminate or distribute it; do not open any
attachments, delete it immediately from your system and notify the sender
promptly by e-mail that you have done so. Thank you.
------_=_NextPart_001_01C37866.4ADD72D0
Content-Type: text/html
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<META NAME="Generator" CONTENT="MS Exchange Server version 5.5.2654.89">
<TITLE>RE: Wild Card/Generic retrieval</TITLE>
</HEAD>
<BODY>
<P><FONT SIZE=2>Thanks Darren. This has been quite educational IMHO.</FONT>
</P>
<P><FONT SIZE=2>Bob</FONT>
</P>
<P><FONT SIZE=2>-----Original Message-----</FONT>
<BR><FONT SIZE=2>From: Darren Reely [<A HREF="mailto:darren.reely@latticesemi.com">mailto:darren.reely@latticesemi.com</A>]</FONT>
<BR><FONT SIZE=2>Sent: Wednesday, September 10, 2003 4:57 PM</FONT>
<BR><FONT SIZE=2>To: Deskin, Bob</FONT>
<BR><FONT SIZE=2>Cc: 'Edis, Robert'; Powerh-L (E-mail)</FONT>
<BR><FONT SIZE=2>Subject: Re: Wild Card/Generic retrieval</FONT>
</P>
<BR>
<P><FONT SIZE=2>That cursor example is what I did not include, but mentioned. The time </FONT>
<BR><FONT SIZE=2>was 2 minutes 47 seconds. I just reran the test and got 2 minutes 36 </FONT>
<BR><FONT SIZE=2>seconds. I did find this a surprising result. After all, you'd expect </FONT>
<BR><FONT SIZE=2>the query to filter out rows that the other tests have to go through.</FONT>
</P>
<P><FONT SIZE=2>Understanding that Quiz is doing a little more work I tried to sway </FONT>
<BR><FONT SIZE=2>Oracle to do the same via my subselect example. Somehow Oracle 'saw' </FONT>
<BR><FONT SIZE=2>through that, or worked much more efficiently at its low levels. Quiz on </FONT>
<BR><FONT SIZE=2>the other hand still fetches real data from Oracle before applying the </FONT>
<BR><FONT SIZE=2>SELECT, taking up crucial time. Oh, and Quiz is placing that fetched </FONT>
<BR><FONT SIZE=2>data somewhere else also.</FONT>
</P>
<P><FONT SIZE=2>Here is a new faster version of the cursor sample. It takes 10 seconds.</FONT>
</P>
<P><FONT SIZE=2> set report nolimit</FONT>
<BR><FONT SIZE=2> sql in mfg declare lss cursor for select lot_id from lot_step_summ </FONT>
<BR><FONT SIZE=2>where lot_id like '%A'</FONT>
<BR><FONT SIZE=2> access lss</FONT>
<BR><FONT SIZE=2> define mycount num = 1</FONT>
<BR><FONT SIZE=2> final footing mycount subtotal</FONT>
</P>
<P><FONT SIZE=2>This suggests that when fetching data, you should only fetch what you </FONT>
<BR><FONT SIZE=2>need, not everything, as the SELECT * does. Temporary Quiz storage </FONT>
<BR><FONT SIZE=2>concerns are not being taxed as much in this new example.</FONT>
</P>
<P><FONT SIZE=2>My testing was a simple curiosity experiment. Real reporting could alter </FONT>
<BR><FONT SIZE=2>the results somewhat. Still, I know to avoid MATCHPATTERN, and SELECT * </FONT>
<BR><FONT SIZE=2>when not required.</FONT>
</P>
<P><FONT SIZE=2>Darren</FONT>
</P>
<BR>
<P><FONT SIZE=2>Deskin, Bob wrote:</FONT>
</P>
<P><FONT SIZE=2>> Well, QUIZ is retrieving each individual row from the database and </FONT>
<BR><FONT SIZE=2>> then applying the selection. The SQL is letting the database do all of </FONT>
<BR><FONT SIZE=2>> the selection before returning anything.</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> How about trying this:</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> declare x cursor for select select * from lot_step_summ where lot_id </FONT>
<BR><FONT SIZE=2>> like '%A'</FONT>
<BR><FONT SIZE=2>> access x</FONT>
<BR><FONT SIZE=2>> define mycount num = 1</FONT>
<BR><FONT SIZE=2>> final footing mycount subtotal</FONT>
<BR><FONT SIZE=2>> go</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> or something similar.</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> Bob</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> -----Original Message-----</FONT>
<BR><FONT SIZE=2>> From: Edis, Robert [<A HREF="mailto:REdis@Blistex.com">mailto:REdis@Blistex.com</A>]</FONT>
<BR><FONT SIZE=2>> Sent: Wednesday, September 10, 2003 3:37 PM</FONT>
<BR><FONT SIZE=2>> To: Powerh-L (E-mail)</FONT>
<BR><FONT SIZE=2>> Subject: RE: Wild Card/Generic retrieval</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> What IS an issue is 4 sec. for SQL vs. 53 sec for Quiz. Why is</FONT>
<BR><FONT SIZE=2>> Quiz so slow?</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> -----Original Message-----</FONT>
<BR><FONT SIZE=2>> From: Deskin, Bob [<A HREF="mailto:Bob.Deskin@cognos.com">mailto:Bob.Deskin@cognos.com</A>]</FONT>
<BR><FONT SIZE=2>> Sent: Wednesday, September 10, 2003 2:25 PM</FONT>
<BR><FONT SIZE=2>> To: 'Darren Reely'</FONT>
<BR><FONT SIZE=2>> Cc: powerh-l@lists.swau.edu</FONT>
<BR><FONT SIZE=2>> Subject: RE: Wild Card/Generic retrieval</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> Thanks Darren for confirming that pattern matching is still</FONT>
<BR><FONT SIZE=2>> quite resource intensive relative to the other PowerHouse</FONT>
<BR><FONT SIZE=2>> functions. While we can't say exactly how much more intensive</FONT>
<BR><FONT SIZE=2>> without testing various volumes, I'm not surprised at these</FONT>
<BR><FONT SIZE=2>> results. What is interesting is how close the other function</FONT>
<BR><FONT SIZE=2>> tests are.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> Bob</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> -----Original Message-----</FONT>
<BR><FONT SIZE=2>> From: Darren Reely [<A HREF="mailto:darren.reely@latticesemi.com">mailto:darren.reely@latticesemi.com</A>]</FONT>
<BR><FONT SIZE=2>> Sent: Wednesday, September 10, 2003 2:57 PM</FONT>
<BR><FONT SIZE=2>> Cc: powerh-l@lists.swau.edu</FONT>
<BR><FONT SIZE=2>> Subject: Re: Wild Card/Generic retrieval</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> I found this interesting enough to 'play' a little with it. I</FONT>
<BR><FONT SIZE=2>> choose the</FONT>
<BR><FONT SIZE=2>> main table in our Oracle test system containing 1,482,455</FONT>
<BR><FONT SIZE=2>> rows. I</FONT>
<BR><FONT SIZE=2>> altered the queries for my known data and limited them to</FONT>
<BR><FONT SIZE=2>> counting.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> In this scenario it is obvious to avoid MATCHPATTERN. I've</FONT>
<BR><FONT SIZE=2>> never been</FONT>
<BR><FONT SIZE=2>> impressed with its performance. Most of the options proposed</FONT>
<BR><FONT SIZE=2>> came in at</FONT>
<BR><FONT SIZE=2>> around 53 seconds, MATHCPATTERN added 2 minutes to those times!</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> Not listed here is a combination of the Oracle SQL select as a</FONT>
<BR><FONT SIZE=2>> cursor</FONT>
<BR><FONT SIZE=2>> for a new version of the quiz file. It took 2 minutes 47 seconds.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> At the end of each select statement are the timings.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> Oracle SQL - approximately 4 seconds</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> select count(*)</FONT>
<BR><FONT SIZE=2>> from lot_step_summ</FONT>
<BR><FONT SIZE=2>> where lot_id like '%A';</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> OR</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> select count(*)</FONT>
<BR><FONT SIZE=2>> from (select * from lot_step_summ where lot_id like '%A');</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> COUNT(*)</FONT>
<BR><FONT SIZE=2>> ----------</FONT>
<BR><FONT SIZE=2>> 145878</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> Powerhouse times</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> set report nolimit</FONT>
<BR><FONT SIZE=2>> access lot_step_summ in mfg</FONT>
<BR><FONT SIZE=2>> ;select lot_step_summ if MATCHPATTERN(TRUNCATE(lot_id),"@A") </FONT>
<BR><FONT SIZE=2>> ;; 2</FONT>
<BR><FONT SIZE=2>> minutes 53 seconds</FONT>
<BR><FONT SIZE=2>> ;select lot_step_summ If 0 = index(lot_id,"A") -</FONT>
<BR><FONT SIZE=2>> size(truncate(lot_id))</FONT>
<BR><FONT SIZE=2>> ;; 53 seconds fetches too many records, no surprise</FONT>
<BR><FONT SIZE=2>> ;select lot_step_summ if 'A' = REVERSE(TRUNCATE( lot_id</FONT>
<BR><FONT SIZE=2>> ))[1:1] ;; 53</FONT>
<BR><FONT SIZE=2>> seconds</FONT>
<BR><FONT SIZE=2>> select if "A" = lot_id[ SIZE(TRUNCATE(lot_id)) :</FONT>
<BR><FONT SIZE=2>> 1] ;; 56 seconds</FONT>
<BR><FONT SIZE=2>> define mycount num = 1</FONT>
<BR><FONT SIZE=2>> final footing mycount subtotal</FONT>
<BR><FONT SIZE=2>> go</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> 09/10/2003 </FONT>
<BR><FONT SIZE=2>> mfg </FONT>
<BR><FONT SIZE=2>> PAGE 1</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> MYCOUNT</FONT>
<BR><FONT SIZE=2>> (SUBTOTAL)</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> 145878</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> Darren</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> = = = = = = = = = = = = = = = = = = = = = = = = = = = =</FONT>
<BR><FONT SIZE=2>> Mailing list: powerh-l@lists.swau.edu</FONT>
<BR><FONT SIZE=2>> Subscribe: "subscribe" in message body to</FONT>
<BR><FONT SIZE=2>> powerh-l-request@lists.swau.edu</FONT>
<BR><FONT SIZE=2>> Unsubscribe: "unsubscribe <password>" in message body to</FONT>
<BR><FONT SIZE=2>> powerh-l-request@lists.swau.edu</FONT>
<BR><FONT SIZE=2>> <A HREF="http://lists.swau.edu/mailman/listinfo/powerh-l" TARGET="_blank">http://lists.swau.edu/mailman/listinfo/powerh-l</A></FONT>
<BR><FONT SIZE=2>> This list is closed, thus to post to the list you must be a</FONT>
<BR><FONT SIZE=2>> subscriber.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> Join us at Cognos' biggest event of the year Enterprise 2003,</FONT>
<BR><FONT SIZE=2>> The Cognos Business Forum. Taking place in over 25 cities</FONT>
<BR><FONT SIZE=2>> around the world, it's an opportunity for Business and IT</FONT>
<BR><FONT SIZE=2>> leaders to learn about strategies for driving performance.</FONT>
<BR><FONT SIZE=2>> Visit <A HREF="http://www.cognos.com/enterprise03" TARGET="_blank">http://www.cognos.com/enterprise03</A> for more details.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> This message may contain privileged and/or confidential</FONT>
<BR><FONT SIZE=2>> information. If you have received this e-mail in error or are</FONT>
<BR><FONT SIZE=2>> not the intended recipient, you may not use, copy, disseminate</FONT>
<BR><FONT SIZE=2>> or distribute it; do not open any attachments, delete it</FONT>
<BR><FONT SIZE=2>> immediately from your system and notify the sender promptly by</FONT>
<BR><FONT SIZE=2>> e-mail that you have done so. Thank you.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> Join us at Cognos' biggest event of the year Enterprise 2003, The </FONT>
<BR><FONT SIZE=2>> Cognos Business Forum. Taking place in over 25 cities around the </FONT>
<BR><FONT SIZE=2>> world, it's an opportunity for Business and IT leaders to learn about </FONT>
<BR><FONT SIZE=2>> strategies for driving performance. Visit </FONT>
<BR><FONT SIZE=2>> <A HREF="http://www.cognos.com/enterprise03" TARGET="_blank">http://www.cognos.com/enterprise03</A> for more details.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> This message may contain privileged and/or confidential information. </FONT>
<BR><FONT SIZE=2>> If you have received this e-mail in error or are not the intended </FONT>
<BR><FONT SIZE=2>> recipient, you may not use, copy, disseminate or distribute it; do not </FONT>
<BR><FONT SIZE=2>> open any attachments, delete it immediately from your system and </FONT>
<BR><FONT SIZE=2>> notify the sender promptly by e-mail that you have done so. Thank you.</FONT>
<BR><FONT SIZE=2>></FONT>
</P>
<BR>
<P><FONT SIZE=2>Join us at Cognos' biggest event of the year<I> Enterprise 2003, The Cognos Business Forum</I>. Taking place in over 25 cities around the world, it's an opportunity for Business and IT leaders to learn about strategies for driving performance. Visit<U> <A HREF="http://www.cognos.com/enterprise03" TARGET="_blank">http://www.cognos.com/enterprise03</A></U> for more details. </FONT></P>
<P><FONT SIZE=2>This message may contain privileged and/or confidential information. If you have received this e-mail in error or are not the intended recipient, you may not use, copy, disseminate or distribute it; do not open any attachments, delete it immediately from your system and notify the sender promptly by e-mail that you have done so. Thank you.</FONT></P>
</BODY>
</HTML>
------_=_NextPart_001_01C37866.4ADD72D0--