just committed r6381 which speeds up FlexibleDateParser by prefering successful
DateFormatters (see log below). please test.
other observations.
1. saving the SHP data immediately again is lightning speed fast. not sure why.
2. sometimes writing with the new "Save Dataset as (testing)" dialog fails with
the stack below
[INFO] 21:23:06.501 Reading 'test_date.jml' took 3.39s.
[INFO] 21:23:07.247 Done. Current committed
memory:plugin.AbstractPlugIn.executing = Executing271 MB
[INFO] 21:23:07.253 Activating Select Features Tool
[INFO] 21:23:11.837 Executing Save Dataset As (testing)
[INFO] 21:23:19.141 Activating Select Features Tool
[INFO] 21:23:26.271 Writing 'test_date8.shp' took 7.00s.
[INFO] 21:23:26.275 Done. Current committed
memory:plugin.AbstractPlugIn.executing = Executing321 MB
[INFO] 21:23:35.053 Activating Select Features Tool
[INFO] 21:23:38.203 Executing Save Dataset As (testing)
[INFO] 21:23:42.603 Activating Select Features Tool
[INFO] 21:23:43.037 Writing 'test_date9.shp' took 0.36s.
[ERROR] 21:29:32.398 java.io.EOFException
java.io.EOFException
at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:290)
at
com.vividsolutions.jump.io.EndianDataInputStream.readUnsignedByteLE(EndianDataInputStream.java:86)
at
org.geotools.dbffile.DbfFile$DbfFileHeader.getDbfFileHeader(DbfFile.java:342)
at org.geotools.dbffile.DbfFile$DbfFileHeader.<init>(DbfFile.java:336)
at org.geotools.dbffile.DbfFile.init(DbfFile.java:172)
at org.geotools.dbffile.DbfFile.<init>(DbfFile.java:80)
at org.geotools.dbffile.DbfFile.<init>(DbfFile.java:66)
at
com.vividsolutions.jump.io.ShapefileWriter.writeDbf(ShapefileWriter.java:420)
at
com.vividsolutions.jump.io.ShapefileWriter.write(ShapefileWriter.java:293)
at
com.vividsolutions.jump.io.datasource.ReaderWriterFileDataSource$1.executeUpdate(ReaderWriterFileDataSource.java:141)
at
org.openjump.core.ui.io.file.DataSourceFileLayerSaver.write(DataSourceFileLayerSaver.java:62)
at
org.openjump.core.ui.plugin.file.save.SaveToFileWizard.run(SaveToFileWizard.java:75)
at
org.openjump.core.ui.plugin.file.SaveWizardPlugIn.run(SaveWizardPlugIn.java:127)
at
com.vividsolutions.jump.workbench.ui.task.TaskMonitorManager$TaskWrapper.run(TaskMonitorManager.java:151)
On 18.08.2020 20:57, ede wrote:
> ok, got it. SHP reader does not seem to use lazy date parsing via
> FlexibleDateParser. converting your set to JML and tryinf to save that
> involves FlexibleDateParser and shows the symptoms you observed.
>
> ..ede
>
> On 18.08.2020 20:24, ede wrote:
>> sorry Mike, can't reproduce it.
>>
>> actually writing is magnitudes faster than reading on my laptop. see
>>
>> [INFO] 20:05:45.910 Reading 'test_date.shp' took 6.64s.
>> [INFO] 20:05:47.011 Done. Current committed
>> memory:plugin.AbstractPlugIn.executing = Executing199 MB
>> [INFO] 20:05:47.013 Activating Select Features Tool
>> ...
>> [INFO] 20:06:09.648 Executing Save Dataset As (testing)
>> [INFO] 20:06:26.760 Activating Select Features Tool
>> [INFO] 20:06:27.881 Writing 'test_date-out.shp' took 0.93s.
>> [INFO] 20:06:27.891 Done. Current committed
>> memory:plugin.AbstractPlugIn.executing = Executing227 MB
>>
>> i also added a debug breakpoint at
>> com.vividsolutions.jump.util.FlexibleDateParser.parse(String s, boolean
>> lenient) and it isn't even called during saving as SHP.
>>
>> ..ede
>>
>> On 17.08.2020 21:15, michael michaud wrote:
>>> Here is a dataset of 100 000 features including datetimes. It takes nearly
>>> 1 minute to save as shapefile (less than 2 seconds to read it).
>>>
>>>
>>> Attachments:
>>>
>>> -
>>> [test_date.zip](https://sourceforge.net/p/jump-pilot/bugs/_discuss/thread/62e6f051ce/89eb/attachment/test_date.zip)
>>> (1.6 MB; application/x-zip-compressed)
>>>
>>>
>>> ---
>>>
>>> ** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**
>>>
>>> **Status:** open
>>> **Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
>>> **Last Updated:** Mon Aug 17, 2020 04:38 PM UTC
>>> **Owner:** michael michaud
>>>
>>>
>>> Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb)
>>> with version 1.15 lasted more than 20 minutes.
>>> Most of the time is used by FlexibleDateParser.parse().
>>>
>>>
>>> ---
>>>
>>> Sent from sourceforge.net because you indicated interest in
>>> <https://sourceforge.net/p/jump-pilot/bugs/497/>
>>>
>>>
>>>
>>> To unsubscribe from further messages, please visit
>>> <https://sourceforge.net/auth/subscriptions/>
>>>
>>
>>
>>
>> ---
>>
>> ** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**
>>
>> **Status:** open
>> **Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
>> **Last Updated:** Mon Aug 17, 2020 07:15 PM UTC
>> **Owner:** michael michaud
>>
>>
>> Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb)
>> with version 1.15 lasted more than 20 minutes.
>> Most of the time is used by FlexibleDateParser.parse().
>>
>>
>> ---
>>
>> Sent from sourceforge.net because you indicated interest in
>> <https://sourceforge.net/p/jump-pilot/bugs/497/>
>>
>>
>>
>> To unsubscribe from further messages, please visit
>> <https://sourceforge.net/auth/subscriptions/>
>>
>
>
>
>
>
> ---
>
> ** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**
>
> **Status:** open
> **Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
> **Last Updated:** Mon Aug 17, 2020 07:15 PM UTC
> **Owner:** michael michaud
>
>
> Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb)
> with version 1.15 lasted more than 20 minutes.
> Most of the time is used by FlexibleDateParser.parse().
>
>
> ---
>
> Sent from sourceforge.net because you indicated interest in
> <https://sourceforge.net/p/jump-pilot/bugs/497/>
>
>
>
> To unsubscribe from further messages, please visit
> <https://sourceforge.net/auth/subscriptions/>
>
---
** [bugs:#497] Shapefile export slowed down because of FlexibleDateParser**
**Status:** open
**Created:** Mon Aug 17, 2020 11:47 AM UTC by michael michaud
**Last Updated:** Mon Aug 17, 2020 07:15 PM UTC
**Owner:** michael michaud
Exporting a big dataset to shapefile (1 300 000 objets - around 225+115 Mb)
with version 1.15 lasted more than 20 minutes.
Most of the time is used by FlexibleDateParser.parse().
---
Sent from sourceforge.net because jump-pilot-devel@lists.sourceforge.net is
subscribed to https://sourceforge.net/p/jump-pilot/bugs/
To unsubscribe from further messages, a project admin can change settings at
https://sourceforge.net/p/jump-pilot/admin/bugs/options. Or, if this is a
mailing list, you can unsubscribe from the mailing list._______________________________________________
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
