date:20200507




On Thu, 7 May 2020, Ryan Joseph via fpc-pascal wrote:


Working on the language server it's become clear that JSON is going to be a 
killer bottleneck, but that's what Microsoft decided to use. It's almost a deal 
breaker just because of the sheer size of the data due to JSON's verbose plain 
text formatting.

I've seen there is such thing as BSON (https://en.wikipedia.org/wiki/BSON)
which could be implemented over JSON without disrupting the underlying
architecture.  Is there any BSON formatters for the FPC JSON libraries? 
It's just serializing in a  different text format so it's probably pretty

easy to implement but I thought I would ask first.


There currently isn't but it has been on my todo list for ages. Patches welcome 
:)

Michael.
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Bugs in StrToHostAddr6 in sockets unit


On 6/05/20 11:21 pm, Christo Crause via fpc-pascal wrote:
> On Wed, May 6, 2020 at 12:19 PM Noel Duffy via fpc-pascal
>  >
> wrote:
>
> So I guess the question is, is it worth the effort to make
> StrToHostAddr6 RFC4291 compliant? Is that something the FPC team 
would

> want, or do they just not use the sockets unit?
>
>
> There have been 2 changes applied to sockets.inc in the last two
> months so it is not neglected. I am convinced a patch to ensure
> compliance will be considered for inclusion, especially if you
> provide test cases demonstrating shortcomings in the current
> implementation.

Great. My worry was that the sockets unit might fall into the same 
category as the libc one, available but deprecated and not recommended 
for use. Plus, when rewriting something that's been unchanged for years, 
there's a risk of breaking programs that depend on the bug.


___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Bugs in StrToHostAddr6 in sockets unit

On Thu, 7 May 2020, Noel Duffy via fpc-pascal wrote:

On 6/05/20 11:21 pm, Christo Crause via fpc-pascal wrote:
> On Wed, May 6, 2020 at 12:19 PM Noel Duffy via fpc-pascal
>  >
> wrote:
>
> So I guess the question is, is it worth the effort to make
> StrToHostAddr6 RFC4291 compliant? Is that something the FPC team 
would

> want, or do they just not use the sockets unit?
>
>
> There have been 2 changes applied to sockets.inc in the last two
> months so it is not neglected. I am convinced a patch to ensure
> compliance will be considered for inclusion, especially if you
> provide test cases demonstrating shortcomings in the current
> implementation.

Great. My worry was that the sockets unit might fall into the same 
category as the libc one, available but deprecated and not recommended 
for use. Plus, when rewriting something that's been unchanged for years, 
there's a risk of breaking programs that depend on the bug.

I think the processing of wrong IP addresses and missing IPv4-ending
addresses support is not really something one depends on :)
So fixing this is quite OK.

Michael.
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] BSON formatter

2020-05-07 Thread Andreas Schneider


Am 2020-05-07 08:40, schrieb Ryan Joseph via fpc-pascal:

Working on the language server it's become clear that JSON is going to
be a killer bottleneck, but that's what Microsoft decided to use. It's
almost a deal breaker just because of the sheer size of the data due
to JSON's verbose plain text formatting.

I've seen there is such thing as BSON
(https://en.wikipedia.org/wiki/BSON) which could be implemented over
JSON without disrupting the underlying architecture. Is there any BSON
formatters for the FPC JSON libraries? It's just serializing in a
different text format so it's probably pretty easy to implement but I
thought I would ask first.


I would go with ProtoBuf for something like this. If you really want to
be flexibel in the datastructure, look at msgpack/messagepack. Both
(protobuf and msgpack) are pretty good in benchmarks.

--
Andreas
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Bugs in StrToHostAddr6 in sockets unit


On 6/05/20 11:40 pm, Karoly Balogh (Charlie/SGR) wrote:


As much as we use the compiler and associated libraries ourselves, we
can't cover every use case, or find every bug. A lot of packages are there
for completeness, or because someone submitted it, or for compatibility
with Delphi, not because we necessarily use it ourselves. So if you find
weak spots like this where you feel you can do better, do it and submit
it!


Sure, I do understand this, but it can be difficult sometimes  to tell 
whether a unit is actively used or whether it's a relic of some long 
abandoned idea, like the libc unit for instance. Added to that, I am 
aware that rewriting functions in old units risks breaking things, even 
if the fix is correct, and so I feel it's prudent (and, hopefully, 
polite) to ask before writing the code.



___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

[fpc-pascal] Using include files in code tools

When using code tools I've found that it needs to start from a main unit 
(otherwise other relevant macros may not be set) and then insert includes into 
the source text. This is fine but when I inspect the node tree (TCodeTreeNode) 
how do I know which include a node came from?

I need this information so I can get navigate to the include file AND relative 
text offset into the include file text. 

Regards,
Ryan Joseph

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Bugs in StrToHostAddr6 in sockets unit


On 7/05/20 9:17 pm, Michael Van Canneyt wrote:


On Thu, 7 May 2020, Noel Duffy via fpc-pascal wrote:

Great. My worry was that the sockets unit might fall into the same 
category as the libc one, available but deprecated and not recommended 
for use. Plus, when rewriting something that's been unchanged for 
years, there's a risk of breaking programs that depend on the bug.


I think the processing of wrong IP addresses and missing IPv4-ending
addresses support is not really something one depends on :)
So fixing this is quite OK.


Heh! Guess I really am the only one who implements RFCs in Object Pascal 
in his spare time. Maybe I should have learned to play golf :D




___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Bugs in StrToHostAddr6 in sockets unit




On Thu, 7 May 2020, Noel Duffy via fpc-pascal wrote:


On 7/05/20 9:17 pm, Michael Van Canneyt wrote:


On Thu, 7 May 2020, Noel Duffy via fpc-pascal wrote:

Great. My worry was that the sockets unit might fall into the same 
category as the libc one, available but deprecated and not recommended 
for use. Plus, when rewriting something that's been unchanged for 
years, there's a risk of breaking programs that depend on the bug.


I think the processing of wrong IP addresses and missing IPv4-ending
addresses support is not really something one depends on :)
So fixing this is quite OK.


Heh! Guess I really am the only one who implements RFCs in Object Pascal 
in his spare time. Maybe I should have learned to play golf :D


Well, considering the amount of RFCs I implemented for FPC, I think I'll join 
you.
Let me know what golf course you picked. If your email address is any
indication, new zealand is a bit far off, but has always been on my bucket list 
;-)

Michael.
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] BSON formatter




On Thu, 7 May 2020, Andreas Schneider wrote:


Am 2020-05-07 08:40, schrieb Ryan Joseph via fpc-pascal:

Working on the language server it's become clear that JSON is going to
be a killer bottleneck, but that's what Microsoft decided to use. It's
almost a deal breaker just because of the sheer size of the data due
to JSON's verbose plain text formatting.

I've seen there is such thing as BSON
(https://en.wikipedia.org/wiki/BSON) which could be implemented over
JSON without disrupting the underlying architecture. Is there any BSON
formatters for the FPC JSON libraries? It's just serializing in a
different text format so it's probably pretty easy to implement but I
thought I would ask first.


I would go with ProtoBuf for something like this. If you really want to
be flexibel in the datastructure, look at msgpack/messagepack. Both
(protobuf and msgpack) are pretty good in benchmarks.


The LSP protocol requires JSON, so it's not an option to use something else.
I'm not even sure BSON will be accepted for LSP.

Michael.
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Using include files in code tools

2020-05-07 Thread Mattias Gaertner via fpc-pascal

On Thu, 7 May 2020 16:26:12 +0700
Ryan Joseph via fpc-pascal  wrote:

> When using code tools I've found that it needs to start from a main
> unit (otherwise other relevant macros may not be set) and then insert
> includes into the source text. This is fine but when I inspect the
> node tree (TCodeTreeNode) how do I know which include a node came
> from?

Nodes are always associated with a Tool. Tool.MainFilename gives the
file, where parsing started.

 
> I need this information so I can get navigate to the include file AND
> relative text offset into the include file text. 

There are plenty of functions for include files. Can you give an
example what you need?

Mattias

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Using include files in code tools

2020-05-07 Thread Mattias Gaertner via fpc-pascal

On Thu, 7 May 2020 12:26:53 +0200
Mattias Gaertner via fpc-pascal  wrote:

>[...]how do I know which include a
> > node came from?  
> 
> Nodes are always associated with a Tool. Tool.MainFilename gives the
> file, where parsing started.
>  
> > I need this information so I can get navigate to the include file
> > AND relative text offset into the include file text.   
> 
> There are plenty of functions for include files. Can you give an
> example what you need?

Tool.CleanPosToCaret(Node.StartPos,CXY)
Tool.CleanPosToCodePos(Node.StartPos,CP)

Mattias

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Using include files in code tools

> On May 7, 2020, at 5:26 PM, Mattias Gaertner via fpc-pascal 
>  wrote:
> 
> Nodes are always associated with a Tool. Tool.MainFilename gives the
> file, where parsing started.
> 

Just curious, can you explain the reasoning of "tools"? I haven't exactly 
figured out the delegation of tasks in your system.

> 
>> I need this information so I can get navigate to the include file AND
>> relative text offset into the include file text. 
> 
> There are plenty of functions for include files. Can you give an
> example what you need?

I start with CodeToolBoss.Explore after loading the unit and get the current 
tool from that. I need to start with an actual unit right? I don't think it 
will parse plain include files on its own, and that won't often work anyways 
because there are  macros in the unit which includes the include file.

I'm collecting the symbols so that I can return them to the client and it can 
navigate to a particular file at a line/column number. The file which I 
navigate to must be the include file but I only have those location relative to 
the unit, not the include. 

Regards,
Ryan Joseph

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] BSON formatter

> On May 7, 2020, at 4:44 PM, Michael Van Canneyt  
> wrote:
> 
> I'm not even sure BSON will be accepted for LSP.

I talked to them already and they're not keen to add extensions to the plugin 
so I'll probably give up on this route. The spec from Microsoft doesn't say 
anything about this either so maybe it's a bottle neck others have hit. Sending 
plain text is horribly stupid and Microsoft doesn't seem to be taking 
performance seriously. If they want to improve performance there is other low 
hanging fruit but it requires Microsoft to extend the protocol.

The amount of redundant information being sent around is ridiculous. Mainly, 
the spec gives no way for the client to cache results and refer to them later 
in various requests. We're literally sending around mega bytes of data for no 
reason whatsoever, except lazy programmers I guess.

Computers just keep getting slower and slower sometimes it feels...

Regards,
Ryan Joseph

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] BSON formatter

On Thu, 7 May 2020, Ryan Joseph via fpc-pascal wrote:

On May 7, 2020, at 4:44 PM, Michael Van Canneyt wrote:

I'm not even sure BSON will be accepted for LSP.

I talked to them already and they're not keen to add extensions to the
plugin so I'll probably give up on this route. The spec from Microsoft
doesn't say anything about this either so maybe it's a bottle neck others
have hit. Sending plain text is horribly stupid and Microsoft doesn't
seem to be taking performance seriously. If they want to improve
performance there is other low hanging fruit but it requires Microsoft to
extend the protocol.

At least partial updates to a file should be there.
Sending the whole file at essentially every few keystrokes is just braindead...

The amount of redundant information being sent around is ridiculous.
Mainly, the spec gives no way for the client to cache results and refer to

them later in various requests. We're literally sending around mega bytes
of data for no reason whatsoever, except lazy programmers I guess.

Indeed.

I was doing some tests in VS Code with omnipascal; The Omnipascal LSP server
process takes 100% cpu.
No surprises there

Computers just keep getting slower and slower sometimes it feels...

Welcome to IT in 2020:
waiting, staring at a wait cursor... And more waiting :-)

Like I said:
This is probably one of the reasons for the current trend towards huge amounts of really small files.

Michael.
___
fpc-pascal maillist - fpc-pascal@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] BSON formatter

2020-05-07 Thread Andreas Schneider


Am 2020-05-07 15:26, schrieb Ryan Joseph via fpc-pascal:

The amount of redundant information being sent around is ridiculous.
Mainly, the spec gives no way for the client to cache results and
refer to them later in various requests. We're literally sending
around mega bytes of data for no reason whatsoever, except lazy
programmers I guess.

Computers just keep getting slower and slower sometimes it feels...


That's the normal approach now. Why should you optimize anything if
you can simply buy better hardware? So I end up with a machine ten
times as fast as a few years ago and still cannot run much more in
parallel than back then, because even calendar apps are now written
with a full browser stack (looking at electron here).

As someone who still is happy when a GUI is a statically compiled
app that is < 10MB and doesn't take more than that as memory when
starting up, I am disgusted by the current "state of the art"
development. It's no longer an art.

(I am even more pissed off about this on my smartphone. Why does
a messenger app have to be 100 MB in size?! We had whole operating
systems in just 1% of that size. Even a full oracle database is
just around 30 MB if you strip away the java based management
console stuff ... and that thing is a feature monster for data
management.)



--
Andreas
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] BSON formatter

> On May 7, 2020, at 9:11 PM, Michael Van Canneyt  
> wrote:
> 
> At least partial updates to a file should be there.
> Sending the whole file at essentially every few keystrokes is just 
> braindead...
> 

That's still happening but it's Sublime Texts fault for a bad API. It is fixed 
in the current development branch though (ST4). 

The problem is workspace symbols. For my project there are 40k symbols (system 
SDKs I like to browse easily) which translates to 10MB of JSON text.

It's so stupid it makes my feel ill and I weep for my CPU. I'm going to submit 
an issue to Microsoft since they at least allow this on GitHub.

>> 
>> The amount of redundant information being sent around is ridiculous. Mainly, 
>> the spec gives no way for the client to cache results and refer to
>> them later in various requests.  We're literally sending around mega bytes
>> of data for no reason whatsoever, except lazy programmers I guess.
> 
> Indeed.
> 
> I was doing some tests in VS Code with omnipascal; The Omnipascal LSP server 
> process takes 100% cpu.
> No surprises there

I've been talking to the LSP people at ST and they tell me it's really common 
for servers to be written in JavaScript because it's easily supported within 
VSCode's architecture (which Electron I'm sure you know). They may have written 
the thing in JavaScript for all I know. That's a problem right there as far as 
performance is concerned.

Regards,
Ryan Joseph

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Bugs in StrToHostAddr6 in sockets unit