On 19.08.2020 09:25, michael michaud wrote:
> Indeed, from my tests, things are getting worst :
well. it shouldn't after all i'm trying to improve things :)
> I save my dataset to a shapefile after it has been freshly extracted from a
> database (it contains a geometry and a timestamp)
> 1.15 : 1st save : 55s
> 2nd save < 1s> r6382 : 1st save : 188s!
> 2nd save < 1s
let me check. actually r6382 should be faster now. in my tests it is.
what i do is.
load your data as JML (exported earlier), as it is text based and needs to be
parsed. then save it as SHP and compare times and FlexdateParser behaviour.
what's important is if the loader creates FlexibleFeatures or not. and even
then, if then lazy conversion is optional, as the loader may of course set the
attribute as an object of the Date class.
> Another thing about your explanation : I remember that you introduced the
> LazyConversion of attributes with the json parser. I think the reason was
> related to the type system of json (which has only strings, numbers and
> boolean).
wrt. GeoJSON that's true. but later on someone noticed that opening JML was
very slow and we found date parsing to be the root cause, so lazy conversion
was introduced.
>In the case of shapefile or database where data is precisely typed, I think
>the lazy conversion is finally very costly.
that's incorrect ;) brute forcing date formats is what is very_ costly here.
lazy conversion is merely distributing cost from time A to time B. if we do not
need to convert at some place we should remove conversion there.
>Data should be stored with the right type at the first place. It would avoid
>all these useless conversions of dates. What do you think ? Seems that lazy
>conversion made for the json should stay the exception, not the general way to
>handle data types.
the concept is sound. there is something else goin on.
here check out the commit
https://sourceforge.net/p/jump-pilot/code/5482/
for testing purposes you could simply replace FlexibleFeature with the
BasicFeature again in your local copy to disable lazy conversion and observe
the difference.
what i can't find right now where JML/GML parses/parsed the date which made it
slow. i'll have a look in the bug description again. i think it was Jukka via
mailing list.
so far.. ede
>
>
> ---
>
> ** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**
>
> **Status:** open
> **Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
> **Last Updated:** Tue Aug 18, 2020 09:42 PM UTC
> **Owner:** michael michaud
>
>
> Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb)
> with version 1.15 lasted more than 20 minutes.
> Most of the time is used by FlexibleDateParser.parse().
>
>
> ---
>
> Sent from sourceforge.net because you indicated interest in
> <https://sourceforge.net/p/jump-pilot/bugs/497/>
>
>
>
> To unsubscribe from further messages, please visit
> <https://sourceforge.net/auth/subscriptions/>
>
---
** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**
**Status:** open
**Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
**Last Updated:** Wed Aug 19, 2020 07:39 AM UTC
**Owner:** michael michaud
Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb)
with version 1.15 lasted more than 20 minutes.
Most of the time is used by FlexibleDateParser.parse().
---
Sent from sourceforge.net because jump-pilot-devel@lists.sourceforge.net is
subscribed to https://sourceforge.net/p/jump-pilot/bugs/
To unsubscribe from further messages, a project admin can change settings at
https://sourceforge.net/p/jump-pilot/admin/bugs/options. Or, if this is a
mailing list, you can unsubscribe from the mailing list.
_______________________________________________
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
