Hi, I'm getting back to work on the index prefetching patch [1], but one annoying aspect of that patch is that it's limited to the context of a single executor node. It can be very effective when there's an index scan with many matches for a key, but it's entirely useless for plans with many tiny index scans.
For example, consider a plan like:
Nested Loop
-> ... some scan of a "fact" table ...
-> Index Scan on PK of a "dimension" table
For this the index prefetching is entirely useless - there'll be just
one match for each outer row.
But there still is opportunity for prefetching - we could look ahead in
the outer relation (which can be arbitrary node feeding the nestloop),
and request the index scan to prefetch the matching tuples.
Of course, this is not something the nestloop can do on it's own, it
would require support from the executor. For some nodes prefetching is
pretty straightforward (e.g. index scan), for other nodes it can be
impossible or at least very hard / too expensive.
I was a bit bored over the weekend, so I decided to experiment a bit and
see how difficult would this be, and how much could it gain. Attached is
an early PoC version of that patch - it's very limited (essentially just
NL + inner index scan), but it seems to be working. It's only about 250
insertions, to tiny.
The patch does this:
--------------------
1) ExecPrefetch executor callback
This call is meant to do the actual prefetching - the parent node sets
everything up almost as if for ExecProcNode(), but does not expect the
actual result. The child either does some prefetching or nothing.
2) ExecSupportsPrefetch to identify what nodes accept ExecPrefetch()
This simply says if a given node supports prefetching. The only place
calling this is the nested loop, to enable prefetching only for nested
loops with (parameterized) index scans.
3) ExecPrefetchIndexScan doing prefetching in index scans
This is just trivial IndexNext() variant, getting TIDs and calling
PrefetchBuffer() on them. Right now it just prefetches everything, but
that's seems wrong - this is where the original index prefetching patch
should kick in.
4) ExecNestLoop changes
This is where the actual magic happens - if the inner child knows how to
prefetch stuff (per ExecSupportsPrefetch), this switches to reading
batches of outer slots, and calls ExecPrefetch() on them. Right now the
batch size is hardcoded to 32, but it might use effective_io_concurrency
or something like that. It's a bit naive in other aspects too - it
always reads and prefetches the whole batch at once, instead of ramping
up and then consuming and prefetching slots one by one. Good enough for
PoC, but probably needs improvements.
5) adds enable_nestloop_prefetch to enable/disable this easily
benchmark
---------
Of course, the main promise of this is faster queries, so I did a simple
benchmark, with a query like this:
SELECT * FROM fact_table f JOIN dimension d ON (f.id = d.id)
WHERE f.r < 0.0001;
The "r" is simply a random value, allowing to select arbitrary fraction
of the large fact table "f". Here it selects 0.01%, so ~10k rows from
100M table. Dimensions have 10M rows. See the .sql file attached.
For a variable number of dimensions (1 to 4) the results look like this:
prefetch 1 2 3 4
----------------------------------------
off 3260 6193 8993 11980
on 2145 3897 5531 7352
----------------------------------------
66% 63% 62% 61%
This is on "cold" data, with a restart + drop caches between runs. The
results suggest the prefetching makes it about twice as fast. I was
hoping for more, but not bad for a Poc, chances are it can be improved.
I just noticed there's a couple failures in the regression tests, if I
change the GUC to "true" by default. I haven't looked into that yet, but
I guess there's some mistake in resetting the child node, or something
like that. Will investigate.
What I find a bit annoying is the amount of processing required to
happen twice - once for the prefetching, once for the actual execution.
In particular, this needs to set up all the PARAM_EXEC slots as if the
inner plan was to do regular execution.
The other thing is that while ExecPrefetchIndexScan() only prefetches
the heap page, it still needs to navigate the index to the leaf page. If
the index is huge, that may require I/O. But we'd have to pay that cost
shortly after anyway. It just isn't asynchronous.
One minor detail is that I ran into some issues with tuple slots. I need
a bunch of them to stash the slots received from the outer plan, so I
created a couple slots with TTSOpsVirtual. And that mostly works, except
that later ExecInterpExpr() happens to call slot_getsomeattrs() and that
fails because tts_virtual_getsomeattrs() says:
elog(ERROR, "getsomeattrs is not required to be called on a virtual
tuple table slot");
OK, that call is not necessary for virtual slots, it's noop. But it's
not me calling that, the interpreter does that for some reason. I did
comment that error out in the patch, but I wonder what's the proper way
to make this work ...
regards
[1]
https://www.postgresql.org/message-id/cf85f46f-b02f-05b2-5248-5000b894ebab%40enterprisedb.com
--
Tomas VondraFrom a30e56a76e48c9ca69e7552a821e49bd0b6605a6 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <to...@2ndquadrant.com>
Date: Mon, 26 Aug 2024 01:13:56 +0200
Subject: [PATCH] nestloop prefetch - initial PoC
---
src/backend/executor/execAmi.c | 41 +++++++++
src/backend/executor/execTuples.c | 3 +-
src/backend/executor/nodeIndexscan.c | 67 ++++++++++++++
src/backend/executor/nodeNestloop.c | 131 ++++++++++++++++++++++++++-
src/backend/utils/misc/guc_tables.c | 10 ++
src/include/executor/executor.h | 2 +
src/include/executor/nodeIndexscan.h | 1 +
src/include/nodes/execnodes.h | 7 ++
src/include/optimizer/cost.h | 1 +
9 files changed, 257 insertions(+), 6 deletions(-)
diff --git a/src/backend/executor/execAmi.c b/src/backend/executor/execAmi.c
index 3289e3e0219..8d336c713a2 100644
--- a/src/backend/executor/execAmi.c
+++ b/src/backend/executor/execAmi.c
@@ -594,6 +594,28 @@ ExecSupportsBackwardScan(Plan *node)
}
}
+/*
+ * ExecSupportsPrefetch - does a plan type support prefetching?
+ *
+ * For now only plain index scans do, we could extend that to IOS. Not sure
+ * about other plans.
+ */
+bool
+ExecSupportsPrefetch(Plan *node)
+{
+ if (node == NULL)
+ return false;
+
+ switch (nodeTag(node))
+ {
+ case T_IndexScan:
+ return true;
+
+ default:
+ return false;
+ }
+}
+
/*
* An IndexScan or IndexOnlyScan node supports backward scan only if the
* index's AM does.
@@ -651,3 +673,22 @@ ExecMaterializesOutput(NodeTag plantype)
return false;
}
+
+
+/*
+ * ExecPrefetch
+ */
+void
+ExecPrefetch(PlanState *node)
+{
+ switch (nodeTag(node))
+ {
+ case T_IndexScanState:
+ ExecPrefetchIndexScan((IndexScanState *) node);
+ break;
+
+ default:
+ elog(ERROR, "unrecognized node type: %d", (int) nodeTag(node));
+ break;
+ }
+}
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 00dc3396156..9337913d899 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -129,7 +129,8 @@ tts_virtual_clear(TupleTableSlot *slot)
static void
tts_virtual_getsomeattrs(TupleTableSlot *slot, int natts)
{
- elog(ERROR, "getsomeattrs is not required to be called on a virtual tuple table slot");
+ // we don't know if the slot is virtual or not
+ // elog(ERROR, "getsomeattrs is not required to be called on a virtual tuple table slot");
}
/*
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index 8000feff4c9..325230ae1d8 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -1729,3 +1729,70 @@ ExecIndexScanInitializeWorker(IndexScanState *node,
node->iss_ScanKeys, node->iss_NumScanKeys,
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+
+void
+ExecPrefetchIndexScan(IndexScanState *node)
+{
+ EState *estate;
+ ScanDirection direction;
+ IndexScanDesc scandesc;
+ ItemPointer tid;
+
+ /*
+ * extract necessary information from index scan node
+ */
+ estate = node->ss.ps.state;
+
+ /*
+ * Determine which direction to scan the index in based on the plan's scan
+ * direction and the current direction of execution.
+ */
+ direction = ScanDirectionCombine(estate->es_direction,
+ ((IndexScan *) node->ss.ps.plan)->indexorderdir);
+ scandesc = node->iss_ScanDesc;
+
+ if (scandesc == NULL)
+ {
+ /*
+ * We reach here if the index scan is not parallel, or if we're
+ * serially executing an index scan that was planned to be parallel.
+ */
+ scandesc = index_beginscan(node->ss.ss_currentRelation,
+ node->iss_RelationDesc,
+ estate->es_snapshot,
+ node->iss_NumScanKeys,
+ node->iss_NumOrderByKeys);
+
+ node->iss_ScanDesc = scandesc;
+
+ /*
+ * If no run-time keys to calculate or they are ready, go ahead and
+ * pass the scankeys to the index AM.
+ */
+ if (node->iss_NumRuntimeKeys == 0 || node->iss_RuntimeKeysReady)
+ index_rescan(scandesc,
+ node->iss_ScanKeys, node->iss_NumScanKeys,
+ node->iss_OrderByKeys, node->iss_NumOrderByKeys);
+ }
+
+ /*
+ * XXX This should probably prefetch only a limited number of tuples for
+ * each key, not all of them - the index can easily have millions of them
+ * for some keys. And prefetching more of those items would be subject
+ * to the "regular" index prefetching, if ever implemented.
+ *
+ * XXX The one question is how to communicate back how many items would
+ * be prefetched. If we find a key with 1M TIDs, it probably dones not
+ * make much sense to prefetch further in nestloop, because the 1M will
+ * likely trash the cache anyway.
+ *
+ * XXX This should consider how many items we actually need. For semi
+ * join we only need the first one, for example.
+ */
+ while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ PrefetchBuffer(scandesc->heapRelation, MAIN_FORKNUM, ItemPointerGetBlockNumber(tid));
+ }
+}
diff --git a/src/backend/executor/nodeNestloop.c b/src/backend/executor/nodeNestloop.c
index 7f4bf6c4dbb..c40acc50b83 100644
--- a/src/backend/executor/nodeNestloop.c
+++ b/src/backend/executor/nodeNestloop.c
@@ -24,7 +24,9 @@
#include "executor/execdebug.h"
#include "executor/nodeNestloop.h"
#include "miscadmin.h"
+#include "optimizer/cost.h"
+bool enable_nestloop_prefetch = false;
/* ----------------------------------------------------------------
* ExecNestLoop(node)
@@ -105,15 +107,110 @@ ExecNestLoop(PlanState *pstate)
if (node->nl_NeedNewOuter)
{
ENL1_printf("getting new outer tuple");
- outerTupleSlot = ExecProcNode(outerPlan);
/*
- * if there are no more outer tuples, then the join is complete..
+ * Without prefetching, get the next slot from the outer plan
+ * directly. With prefetching get the next slot from the batch,
+ * or fill the batch if needed.
*/
- if (TupIsNull(outerTupleSlot))
+ if (node->nl_PrefetchCount == 0) /* no prefetching */
{
- ENL1_printf("no outer tuple, ending join");
- return NULL;
+ outerTupleSlot = ExecProcNode(outerPlan);
+
+ /*
+ * if there are no more outer tuples, then the join is complete.
+ */
+ if (TupIsNull(outerTupleSlot))
+ {
+ ENL1_printf("no outer tuple, ending join");
+ return NULL;
+ }
+ }
+ else /* prefetching */
+ {
+
+ /*
+ * No more slots availabe in the queue - try to load from
+ * the outer plan, unless we've already reached the end.
+ */
+ if ((node->nl_PrefetchNext == node->nl_PrefetchUsed) &&
+ (!node->nl_PrefetchDone))
+ {
+ /* reset */
+ node->nl_PrefetchNext = 0;
+ node->nl_PrefetchUsed = 0;
+
+ while (node->nl_PrefetchUsed < node->nl_PrefetchCount)
+ {
+ outerTupleSlot = ExecProcNode(outerPlan);
+
+ /*
+ * if there are no more outer tuples, then the join is complete.
+ */
+ if (TupIsNull(outerTupleSlot))
+ {
+ node->nl_PrefetchDone = true;
+ break;
+ }
+
+ ExecClearTuple(node->nl_PrefetchSlots[node->nl_PrefetchUsed]);
+
+ ExecCopySlot(node->nl_PrefetchSlots[node->nl_PrefetchUsed],
+ outerTupleSlot);
+
+ ENL1_printf("prefetching inner node");
+ econtext->ecxt_outertuple = node->nl_PrefetchSlots[node->nl_PrefetchUsed];
+
+ /*
+ * fetch the values of any outer Vars that must be passed to the
+ * inner scan, and store them in the appropriate PARAM_EXEC slots.
+ */
+ foreach(lc, nl->nestParams)
+ {
+ NestLoopParam *nlp = (NestLoopParam *) lfirst(lc);
+ int paramno = nlp->paramno;
+ ParamExecData *prm;
+
+ prm = &(econtext->ecxt_param_exec_vals[paramno]);
+ /* Param value should be an OUTER_VAR var */
+ Assert(IsA(nlp->paramval, Var));
+ Assert(nlp->paramval->varno == OUTER_VAR);
+ Assert(nlp->paramval->varattno > 0);
+ prm->value = slot_getattr(outerTupleSlot,
+ nlp->paramval->varattno,
+ &(prm->isnull));
+ /* Flag parameter value as changed */
+ innerPlan->chgParam = bms_add_member(innerPlan->chgParam,
+ paramno);
+ }
+
+ Assert(node->nl_PrefetchSlots[node->nl_PrefetchUsed]->tts_nvalid > 0);
+
+ /*
+ * now prefetch the inner plan
+ */
+ ENL1_printf("prefetching inner plan");
+ ExecReScan(innerPlan);
+ ExecPrefetch(innerPlan);
+
+ node->nl_PrefetchUsed++;
+ }
+ }
+
+ /*
+ * Now we should either have a slot in the queue, or know
+ * that we've exhausted the outer side.
+ */
+ if (node->nl_PrefetchNext == node->nl_PrefetchUsed)
+ {
+ Assert(node->nl_PrefetchDone);
+ ENL1_printf("no outer tuple, ending join");
+ return NULL;
+ }
+
+ /* get the next slot from the queue */
+ outerTupleSlot = node->nl_PrefetchSlots[node->nl_PrefetchNext++];
+ Assert(outerTupleSlot->tts_nvalid > 0);
}
ENL1_printf("saving new outer tuple information");
@@ -345,6 +442,24 @@ ExecInitNestLoop(NestLoop *node, EState *estate, int eflags)
nlstate->nl_NeedNewOuter = true;
nlstate->nl_MatchedOuter = false;
+ /* with inner plan supporting prefetching, initialize the batch */
+ nlstate->nl_PrefetchCount = 0;
+ if (enable_nestloop_prefetch &&
+ ExecSupportsPrefetch(innerPlan(node)))
+ {
+ /* batch of 32 slots seem about right for now */
+#define NL_PREFETCH_BATCH_SIZE 32
+ nlstate->nl_PrefetchCount = NL_PREFETCH_BATCH_SIZE;
+ nlstate->nl_PrefetchSlots = palloc0(sizeof(TupleTableSlot *) * nlstate->nl_PrefetchCount);
+
+ for (int i = 0; i < nlstate->nl_PrefetchCount; i++)
+ {
+ nlstate->nl_PrefetchSlots[i]
+ = MakeSingleTupleTableSlot(ExecGetResultType(outerPlanState(nlstate)),
+ &TTSOpsVirtual);
+ }
+ }
+
NL1_printf("ExecInitNestLoop: %s\n",
"node initialized");
@@ -369,6 +484,12 @@ ExecEndNestLoop(NestLoopState *node)
ExecEndNode(outerPlanState(node));
ExecEndNode(innerPlanState(node));
+ /* cleanup batch of prefetch slots */
+ for (int i = 0; i < node->nl_PrefetchCount; i++)
+ {
+ ExecDropSingleTupleTableSlot(node->nl_PrefetchSlots[i]);
+ }
+
NL1_printf("ExecEndNestLoop: %s\n",
"node processing ended");
}
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index af227b1f248..0d5d1c2a600 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -879,6 +879,16 @@ struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_nestloop_prefetch", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables prefetching in nested-loop join plans."),
+ NULL,
+ GUC_EXPLAIN
+ },
+ &enable_nestloop_prefetch,
+ false,
+ NULL, NULL, NULL
+ },
{
{"enable_mergejoin", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of merge join plans."),
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index 69c3ebff00a..05ab4cae4a0 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -107,7 +107,9 @@ extern void ExecMarkPos(PlanState *node);
extern void ExecRestrPos(PlanState *node);
extern bool ExecSupportsMarkRestore(struct Path *pathnode);
extern bool ExecSupportsBackwardScan(Plan *node);
+extern bool ExecSupportsPrefetch(Plan *node);
extern bool ExecMaterializesOutput(NodeTag plantype);
+extern void ExecPrefetch(PlanState *node);
/*
* prototypes from functions in execCurrent.c
diff --git a/src/include/executor/nodeIndexscan.h b/src/include/executor/nodeIndexscan.h
index 3cddece67c8..114ca94352e 100644
--- a/src/include/executor/nodeIndexscan.h
+++ b/src/include/executor/nodeIndexscan.h
@@ -28,6 +28,7 @@ extern void ExecIndexScanInitializeDSM(IndexScanState *node, ParallelContext *pc
extern void ExecIndexScanReInitializeDSM(IndexScanState *node, ParallelContext *pcxt);
extern void ExecIndexScanInitializeWorker(IndexScanState *node,
ParallelWorkerContext *pwcxt);
+extern void ExecPrefetchIndexScan(IndexScanState *node);
/*
* These routines are exported to share code with nodeIndexonlyscan.c and
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index af7d8fd1e72..ddd9f1f3332 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -2131,6 +2131,13 @@ typedef struct NestLoopState
bool nl_NeedNewOuter;
bool nl_MatchedOuter;
TupleTableSlot *nl_NullInnerTupleSlot;
+
+ /* for prefetch of batch items */
+ int nl_PrefetchCount; /* maximum number of queued slots */
+ int nl_PrefetchUsed; /* current number of queued slots */
+ int nl_PrefetchNext; /* next slot to return from queue */
+ bool nl_PrefetchDone; /* no more outer slots */
+ TupleTableSlot **nl_PrefetchSlots; /* array of virtual slots */
} NestLoopState;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 854a782944a..3303677c05b 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -58,6 +58,7 @@ extern PGDLLIMPORT bool enable_sort;
extern PGDLLIMPORT bool enable_incremental_sort;
extern PGDLLIMPORT bool enable_hashagg;
extern PGDLLIMPORT bool enable_nestloop;
+extern PGDLLIMPORT bool enable_nestloop_prefetch;
extern PGDLLIMPORT bool enable_material;
extern PGDLLIMPORT bool enable_memoize;
extern PGDLLIMPORT bool enable_mergejoin;
--
2.46.0
nestloop.sql
Description: application/sql
1 0.0001 1 off 3257.5660000 1 0.0001 1 on 2128.7880000 1 0.0001 2 off 3305.4200000 1 0.0001 2 on 2124.3290000 1 0.0001 3 off 3264.2670000 1 0.0001 3 on 2151.3740000 1 0.0001 4 off 3238.8620000 1 0.0001 4 on 2140.2210000 1 0.0001 5 off 3215.6060000 1 0.0001 5 on 2139.3040000 1 0.0001 6 off 3266.4270000 1 0.0001 6 on 2138.7720000 1 0.0001 7 off 3248.1510000 1 0.0001 7 on 2151.1630000 1 0.0001 8 off 3230.9810000 1 0.0001 8 on 2150.2650000 1 0.0001 9 off 3271.5660000 1 0.0001 9 on 2184.4550000 1 0.0001 10 off 3337.8910000 1 0.0001 10 on 2203.6760000 2 0.0001 1 off 6198.0750000 2 0.0001 1 on 3927.5530000 2 0.0001 2 off 6400.4900000 2 0.0001 2 on 3943.0690000 2 0.0001 3 off 6236.7340000 2 0.0001 3 on 3915.9020000 2 0.0001 4 off 6027.9150000 2 0.0001 4 on 3879.8400000 2 0.0001 5 off 6044.3310000 2 0.0001 5 on 3844.5230000 2 0.0001 6 off 6109.2740000 2 0.0001 6 on 3781.4710000 2 0.0001 7 off 6144.0480000 2 0.0001 7 on 3822.5560000 2 0.0001 8 off 6189.7970000 2 0.0001 8 on 3867.5100000 2 0.0001 9 off 6234.9790000 2 0.0001 9 on 3971.2820000 2 0.0001 10 off 6397.8670000 2 0.0001 10 on 4005.5920000 3 0.0001 1 off 9373.5310000 3 0.0001 1 on 5803.6380000 3 0.0001 2 off 9237.4510000 3 0.0001 2 on 5657.4080000 3 0.0001 3 off 9396.8200000 3 0.0001 3 on 5604.4000000 3 0.0001 4 off 8949.4880000 3 0.0001 4 on 5477.6800000 3 0.0001 5 off 8987.0600000 3 0.0001 5 on 5487.3220000 3 0.0001 6 off 8972.9910000 3 0.0001 6 on 5474.6390000 3 0.0001 7 off 9081.0270000 3 0.0001 7 on 5410.8260000 3 0.0001 8 off 8872.6170000 3 0.0001 8 on 5529.1990000 3 0.0001 9 off 8864.4100000 3 0.0001 9 on 5533.6380000 3 0.0001 10 off 8998.9690000 3 0.0001 10 on 5627.5200000 4 0.0001 1 off 12030.4020000 4 0.0001 1 on 7220.1440000 4 0.0001 2 off 12187.9020000 4 0.0001 2 on 7373.5800000 4 0.0001 3 off 12718.2130000 4 0.0001 3 on 7669.3360000 4 0.0001 4 off 12866.3740000 4 0.0001 4 on 7532.7740000 4 0.0001 5 off 12057.8190000 4 0.0001 5 on 7247.7390000 4 0.0001 6 off 11771.5150000 4 0.0001 6 on 7347.3760000 4 0.0001 7 off 11797.9850000 4 0.0001 7 on 7248.3260000 4 0.0001 8 off 11608.2210000 4 0.0001 8 on 7199.5710000 4 0.0001 9 off 11887.9310000 4 0.0001 9 on 7357.6240000 4 0.0001 10 off 11930.9530000 4 0.0001 10 on 7358.8520000 1 0.0005 1 off 11635.7820000 1 0.0005 1 on 8562.1780000 1 0.0005 2 off 11656.3340000 1 0.0005 2 on 8637.4600000 1 0.0005 3 off 11595.2510000 1 0.0005 3 on 8588.5130000 1 0.0005 4 off 11653.7100000 1 0.0005 4 on 8642.9750000 1 0.0005 5 off 11556.1780000 1 0.0005 5 on 8636.1820000 1 0.0005 6 off 11570.2250000 1 0.0005 6 on 8557.1720000 1 0.0005 7 off 11863.4360000 1 0.0005 7 on 8945.3400000 1 0.0005 8 off 11539.4950000 1 0.0005 8 on 8643.9290000 1 0.0005 9 off 11401.7160000 1 0.0005 9 on 8526.7020000 1 0.0005 10 off 11549.9810000 1 0.0005 10 on 8681.8490000 2 0.0005 1 off 19573.6010000 2 0.0005 1 on 12659.3610000 2 0.0005 2 off 19332.7870000 2 0.0005 2 on 12280.8540000 2 0.0005 3 off 19394.4040000 2 0.0005 3 on 12166.4530000 2 0.0005 4 off 19377.9600000 2 0.0005 4 on 12285.8210000 2 0.0005 5 off 19407.6610000 2 0.0005 5 on 12153.4800000 2 0.0005 6 off 19462.8320000 2 0.0005 6 on 12654.6380000 2 0.0005 7 off 19512.5000000 2 0.0005 7 on 12488.0880000 2 0.0005 8 off 19978.7780000 2 0.0005 8 on 13112.6930000 2 0.0005 9 off 19574.6140000 2 0.0005 9 on 12166.6520000 2 0.0005 10 off 19348.0930000 2 0.0005 10 on 12338.8620000 3 0.0005 1 off 27786.8420000 3 0.0005 1 on 15563.2280000 3 0.0005 2 off 27897.2260000 3 0.0005 2 on 16121.1620000 3 0.0005 3 off 27840.1470000 3 0.0005 3 on 15711.9940000 3 0.0005 4 off 27847.4310000 3 0.0005 4 on 15747.4180000 3 0.0005 5 off 28047.8150000 3 0.0005 5 on 16173.0200000 3 0.0005 6 off 28177.0320000 3 0.0005 6 on 15935.8780000 3 0.0005 7 off 27896.9570000 3 0.0005 7 on 16359.6560000 3 0.0005 8 off 27805.7410000 3 0.0005 8 on 15786.2410000 3 0.0005 9 off 27695.4700000 3 0.0005 9 on 15646.8300000 3 0.0005 10 off 27801.9790000 3 0.0005 10 on 15966.7190000 4 0.0005 1 off 36973.8290000 4 0.0005 1 on 19886.7770000 4 0.0005 2 off 37411.1000000 4 0.0005 2 on 19545.4120000 4 0.0005 3 off 36537.8550000 4 0.0005 3 on 19592.2000000 4 0.0005 4 off 36527.4270000 4 0.0005 4 on 19395.5590000 4 0.0005 5 off 36716.1570000 4 0.0005 5 on 19210.2340000 4 0.0005 6 off 36528.7870000 4 0.0005 6 on 19497.1190000 4 0.0005 7 off 36996.3680000 4 0.0005 7 on 19445.4630000 4 0.0005 8 off 36580.4370000 4 0.0005 8 on 19765.4630000 4 0.0005 9 off 36809.4180000 4 0.0005 9 on 19297.4810000 4 0.0005 10 off 36137.9450000 4 0.0005 10 on 19302.0330000
nestloop.sh
Description: application/shellscript
