Re: CQL decimal encoding

2014-02-26 Thread Ben Hood
On Thu, Feb 27, 2014 at 12:05 AM, Ben Hood <0x6e6...@gmail.com> wrote: > BTW thanks and kudos go to Theo and Tyler (of the cql-rb and the > datastax python drivers respectively) for publishing encoding test > cases for the decimal type - that was quite helpful :-) Sorry, I forgot to mention the in

Re: CQL decimal encoding

2014-02-26 Thread Ben Hood
On Thu, Feb 27, 2014 at 12:01 AM, Ben Hood <0x6e6...@gmail.com> wrote: > Hopefully the gocql team can code review this soon and if that's good > to go, we'll have another CQL driver that can deal with decimals. BTW thanks and kudos go to Theo and Tyler (of the cql-rb and the datastax python driver

Re: CQL decimal encoding

2014-02-26 Thread Ben Hood
On Wed, Feb 26, 2014 at 12:10 PM, Laing, Michael wrote: > go uses 'zig-zag' encoding, perhaps that is the difference? > > > On Wed, Feb 26, 2014 at 6:52 AM, Peter Lin wrote: >> >> >> You may need to bit shift if that is the case Thanks for everybody's help, I've managed to solve the issue: the u

Re: CQL decimal encoding

2014-02-26 Thread Laing, Michael
go uses 'zig-zag' encoding, perhaps that is the difference? On Wed, Feb 26, 2014 at 6:52 AM, Peter Lin wrote: > > You may need to bit shift if that is the case > > Sent from my iPhone > > > On Feb 26, 2014, at 2:53 AM, Ben Hood <0x6e6...@gmail.com> wrote: > > > > Hey Colin, > > > >> On Tue, Feb

Re: CQL decimal encoding

2014-02-26 Thread Peter Lin
You may need to bit shift if that is the case Sent from my iPhone > On Feb 26, 2014, at 2:53 AM, Ben Hood <0x6e6...@gmail.com> wrote: > > Hey Colin, > >> On Tue, Feb 25, 2014 at 10:26 PM, Colin Blower wrote: >> It looks like you are trying to implement the Decimal type. You might want >> to s

Re: CQL decimal encoding

2014-02-25 Thread Ben Hood
Hey Colin, On Tue, Feb 25, 2014 at 10:26 PM, Colin Blower wrote: > It looks like you are trying to implement the Decimal type. You might want > to start with implementing the Integer type. The Decimal type follows pretty > easily from the Integer type. > > For example: > i = unmarchalInteger(data

Re: CQL decimal encoding

2014-02-25 Thread Colin Blower
Hey Ben, It looks like you are trying to implement the Decimal type. You might want to start with implementing the Integer type. The Decimal type follows pretty easily from the Integer type. For example: i = unmarchalInteger(data[4:]) s = decInt(data[0:4]) out = inf.newDec(i, s) On 02/24/2014 09

Re: CQL decimal encoding

2014-02-25 Thread Peter Lin
sounds like maybe the bytes aren't being converted to big endian before sending the bytes to to Cassandra. In my C# driver, this is what I do. 1. I get the bytes for the scale and unscaled value 2. I reverse both byte arrays 3. I copy the bytes to a new byte array here is the actual C# code.

Re: CQL decimal encoding

2014-02-25 Thread Ben Hood
On Tue, Feb 25, 2014 at 12:50 PM, Peter Lin wrote: > > if I have time this week, I'll try to make a patch for the spec. Can't > promise I can get to it this week, but having come across this issue with > FluentCassandra, I'd like to help others avoid it. So I may be running into an encoding bug w

Re: CQL decimal encoding

2014-02-25 Thread Peter Lin
if I have time this week, I'll try to make a patch for the spec. Can't promise I can get to it this week, but having come across this issue with FluentCassandra, I'd like to help others avoid it. On Tue, Feb 25, 2014 at 5:38 AM, Sylvain Lebresne wrote: > On Mon, Feb 24, 2014 at 8:50 PM, Theo Hul

Re: CQL decimal encoding

2014-02-25 Thread Ben Hood
Sylvain, On Tue, Feb 25, 2014 at 10:38 AM, Sylvain Lebresne wrote: > The honest answer is, no-one took the time to write that down properly and > include it in the spec. My small excuse for initially skipping it in the > spec is that the CQL data type encodings are really not different from what

Re: CQL decimal encoding

2014-02-25 Thread Sylvain Lebresne
On Mon, Feb 24, 2014 at 8:50 PM, Theo Hultberg wrote: > I don't know if it's by design or if it's by oversight that the data types > aren't part of the binary protocol specification. > The honest answer is, no-one took the time to write that down properly and include it in the spec. My small exc

Re: CQL decimal encoding

2014-02-24 Thread Ben Hood
On Mon, Feb 24, 2014 at 7:50 PM, Theo Hultberg wrote: > I don't know if it's by design or if it's by oversight that the data types > aren't part of the binary protocol specification. I had to reverse engineer > how to encode and decode all of them for the Ruby driver. There were > definitely a few

Re: CQL decimal encoding

2014-02-24 Thread Ben Hood
On Mon, Feb 24, 2014 at 7:43 PM, Paul "LeoNerd" Evans wrote: > On Mon, 24 Feb 2014 19:14:48 + > Ben Hood <0x6e6...@gmail.com> wrote: > >> So I have a question about the encoding of 0: \x00\x00\x00\x00\x00. > > The first four octets are the decimal shift (0), and the remaining ones > (one in th

Re: CQL decimal encoding

2014-02-24 Thread Theo Hultberg
I don't know if it's by design or if it's by oversight that the data types aren't part of the binary protocol specification. I had to reverse engineer how to encode and decode all of them for the Ruby driver. There were definitely a few bugs in the first few versions that could have been avoided if

Re: CQL decimal encoding

2014-02-24 Thread Paul "LeoNerd" Evans
On Mon, 24 Feb 2014 19:14:48 + Ben Hood <0x6e6...@gmail.com> wrote: > So I have a question about the encoding of 0: \x00\x00\x00\x00\x00. The first four octets are the decimal shift (0), and the remaining ones (one in this case) encode a varint - 0 in this case. So it's 0 * 10**0 literall

Re: CQL decimal encoding

2014-02-24 Thread Ben Hood
On Mon, Feb 24, 2014 at 7:02 PM, Paul "LeoNerd" Evans wrote: > On Mon, 24 Feb 2014 13:55:07 -0500 > Peter Lin wrote: > >> I did the same thing :) >> >> I inserted lots of bigDecimal in Cqlsh and read it from my C# client. >> Then I did the opposite, inserts BigDecimal from C# and query it from >>

Re: CQL decimal encoding

2014-02-24 Thread Paul "LeoNerd" Evans
On Mon, 24 Feb 2014 13:55:07 -0500 Peter Lin wrote: > I did the same thing :) > > I inserted lots of bigDecimal in Cqlsh and read it from my C# client. > Then I did the opposite, inserts BigDecimal from C# and query it from > cqlsh. Once both directions worked, I had unit tests to make sure > da

Re: CQL decimal encoding

2014-02-24 Thread Peter Lin
I did the same thing :) I inserted lots of bigDecimal in Cqlsh and read it from my C# client. Then I did the opposite, inserts BigDecimal from C# and query it from cqlsh. Once both directions worked, I had unit tests to make sure data is cross platform compatible. I know older versions of FluentC

Re: CQL decimal encoding

2014-02-24 Thread Ben Hood
Hey Paul, On Mon, Feb 24, 2014 at 6:40 PM, Paul "LeoNerd" Evans wrote: > And the unit tests live here: > > > https://metacpan.org/source/PEVANS/Protocol-CassandraCQL-0.11/t/02types.t#L111 Very cool - I'll port these examples to the gocql marshaling test suite - kudos to you for reverse engine

Re: CQL decimal encoding

2014-02-24 Thread Paul "LeoNerd" Evans
On Mon, 24 Feb 2014 17:51:54 + Ben Hood <0x6e6...@gmail.com> wrote: > Or in the absence of a spec, just a heads up from other language > driver implementors as to what approach they've taken. I reverse-engineered it by using cqlsh to insert lots of known numerical values, then seeing what the

Re: CQL decimal encoding

2014-02-24 Thread Ben Hood
On Mon, Feb 24, 2014 at 6:09 PM, Peter Lin wrote: > I took a look at the code. Java uses big endian encoding. I don't know if GO > defaults to big or little. In my port of Hector to C#, I reverse the bytes > due to the fact that .Net uses little endian. Cool - I'll take this as a spec - thanks fo

Re: CQL decimal encoding

2014-02-24 Thread Peter Lin
ok, I "think" I understand. I took a look at the code. Java uses big endian encoding. I don't know if GO defaults to big or little. In my port of Hector to C#, I reverse the bytes due to the fact that .Net uses little endian. hope that helps On Mon, Feb 24, 2014 at 12:51 PM, Ben Hood <0x6e6..

Re: CQL decimal encoding

2014-02-24 Thread Ben Hood
Hey Peter, On Mon, Feb 24, 2014 at 5:25 PM, Peter Lin wrote: > > Not sure what you mean by the question. > > Are you talking about the structure of BigDecimal in java? If that is your > question, the java's BigDecimal uses the first 4 bytes for scale and > remaining bytes for BigInteger I'm talk

Re: CQL decimal encoding

2014-02-24 Thread Peter Lin
Not sure what you mean by the question. Are you talking about the structure of BigDecimal in java? If that is your question, the java's BigDecimal uses the first 4 bytes for scale and remaining bytes for BigInteger On Mon, Feb 24, 2014 at 10:47 AM, Ben Hood <0x6e6...@gmail.com> wrote: > Hi, >

CQL decimal encoding

2014-02-24 Thread Ben Hood
Hi, I'd like to implement decimal encoding for gocql but I'm wondering what this should be compatible with. Is there some kind of wire format that arbitrary precision numbers should adhere to to ensure interoperability? Cheers, Ben