Thanks for the response. You are correct that there are issues with
the subtle maths in the "otherwise" portion of the code. I fixed that
issue, and it improved the situation somewhat; however, I was still
forced to insert a Thread.sleep inside of connection-read to get
predictable results. This problem seems to be subtle enough that even
a Thread.sleep(0) makes it perform in a sane fashion, but doing this
feels so radically incorrect, that I don't view it as a long-term
solution.
Regarding your suggestion to do a blocking read, I'm assuming you mean
to read up to specified length rather than EOF? InputStream::read() is
blocking as far as I can see. The -1 was unintentional and shouldn't
happen anyway because the connection should remain open in this
portion of the code.
I guess my question at this point is if there's a way to ensure that
all the data I want is ready to be properly read? I've found
DataInputStream::readFully(), which seems like it might be a better
solution? Apologies, as I'm a bit of a novice when it comes to (Java)
socket programming.
Thanks,
Travis
On Jun 1, 3:35 am, Timothy Pratley <timothyprat...@gmail.com> wrote:
> Hi Travis,
>
> connection-read appears to be subtlety wrong.
> If all the bytes are ready to be read then it is correct, but if only
> some are read it is incorrect. Now in the large data scenario the
> latter will appear if there are no delays. Inserting a sleep delays
> the program per packet such that there is always data to be read -
> causing it to work. This is a much more likely scenario...
>
> How is it wrong?
> the otherwise read more data part recurs until new-bytes-read == -1
> (connection closed) so its no longer reading the amount specified by
> the packet header. I think this was unintentional. A better solution
> would be to just do a blocking read instead which can only give you
> the right number of bytes or error. Otherwise the maths for remaining
> data needs to be fixed. (Blocking is simpler and has the same result
> and doesn't put you in a CPU spin).
>
> Also regarding the state transitions, it looks to me that the way it
> is set up now there has to be an eof to transition to :row-start
> state, which will lead to another recur to reading the (presumably
> finished) response which will be eof also. However I can't see how
> that would result in your described issue its probably me just
> misunderstanding. Your issue sounds highly related to a 'message'
> being split across multiple packets - as it is dependent on data
> size... so I believe the real problem is the way read is handled if
> all the bytes aren't ready to be read.
>
> I've had to make some assumptions about what you are doing so forgive
> me if no relevant.
>
> Regards,
> Tim.
>
> On Jun 1, 1:02 pm, tmountain <tinymount...@gmail.com> wrote:
>
> > Hi all - I'm in the process of writing a proxy for MySQL in Clojure,
> > and until now everything has been going smoothly. My project has
> > reached the point where it can shuffle data up and down the wire
> > between client and server with accurate results, but I'm running into
> > a strange issue. I've noticed that when a certain result size is
> > reached, (exactly 138 rows in this case), I get a malformed packet
> > response from the client.
>
> > Naturally, I assumed this was just an issue with mishandling the data
> > being sent to and fro somehow, but then I ran into something strange.
> > In trying to nail down the issue, I wrote a debug macro which inserts
> > prints a given statement so long as a *debug* variable is set to true.
> > As soon as I enabled the macro, the problem resolved itself, and the
> > client returned the appropriate result. After a brief WTF moment, I
> > decided it might be some kind of timing issue and changed the debug
> > macro to simply do a Thread.sleep for 100 milliseconds. Again, this
> > resolved the problem. Finally, I narrowed down the offending code to
> > the following:
>
> > (defn get-result-query [client server state]
> > "handle the states necessary to return a standard query result"
> > (let [server-response (connection-read server)]
> > (Thread/sleep 100) ; !!!!!!!!! remove this, and the function will
> > sporadically fail !!!!!!!!!
> > (connection-write client server-response)
> > (cond (and (= state :start) (eof-packet? server-response))
> > (recur client server :row-start)
> > (and (= state :row-start) (eof-packet? server-response))
> > {:state :send-command :server server}
> > (error-packet? server-response)
> > {:state :send-command :server server}
> > :else (recur client server state))))
>
> > It seems that reading from the socket without a subsequent pause
> > introduces problems for some reason. I have no idea why this would
> > happen unless the data isn't fully down the wire or something along
> > those lines. For good measure, here's the connection-read function as
> > well:
>
> > (defn connection-read [#^Socket conn]
> > "reads data up to the amount specified by the packet header"
> > (let [#^InputStream instream (. conn (getInputStream))
> > bytes (make-array (. Byte TYPE) 4)]
> > (. instream (read bytes 0 4))
> > ; make an array big enough for header + data
> > (let [data (make-array (. Byte TYPE)
> > (+ 4 (packet-length? bytes)))]
> > (System/arraycopy bytes 0 data 0 4)
> > ; read data into the buffer
> > (loop [bytes-read (. instream (read data 4 (- (alength data)
> > 4)))]
> > ; if we got all the desired data in one pass, return it
> > (if (>= bytes-read (- (alength data) 4))
> > data
> > ; otherwise, read more data
> > (let [new-bytes-read (. instream
> > (read data
> > bytes-read
> > (- (alength data) bytes-
> > read)))]
> > (if (== new-bytes-read -1)
> > data
> > (recur (+ bytes-read new-bytes-read)))))))))
>
> > If anybody has any suggestions to why this would be happening, I'd
> > really appreciate it. I intend to release this project OSS when it's
> > fully functional, and this is one of the few roadblocks in my way.
>
> > Thanks,
> > Travis
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
-~----------~----~----~----~------~----~------~--~---
