[
https://issues.apache.org/jira/browse/CASSANDRA-14982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16770201#comment-16770201
]
C. Scott Andreas commented on CASSANDRA-14982:
----------------------------------------------
Hi [~vaclavt], it looks like the size of the batch generated via your cqlsh
import is exceeding the max batch size configured in Cassandra. This is
controlled in cqlsh via MAXBATCHSIZE=x, and in cassandra.yaml via
batch_size_fail_threshold_in_kb.
To address this you can either specify a MAXBATCHSIZE in cqlsh by adding a low
limit such as "and MAXBATCHSIZE=10" to your COPY command, or via increasing the
max batch size server-side with batch_size_fail_threshold_in_kb to a value that
exceeds what would be imported by cqlsh on a per-batch basis.
Point taken re: the PicklingError and exit-0 status that's returned; thanks for
filing this.
> PicklingError: Can't pickle <class 'cqlshlib.copyutil.ImmutableDict'>:
> attribute lookup cqlshlib.copyutil.ImmutableDict failed
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-14982
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14982
> Project: Cassandra
> Issue Type: Bug
> Components: CQL/Interpreter
> Environment: cqlsh --version
> cqlsh 5.0.1
>
> Reporter: Vaclav
> Priority: Major
> Fix For: 3.11.3
>
>
> Hello
> I am trying to load records from file containing some very long lines (up to
> 180_000 characters). In some cases order of lines in file causes error
> 'Error from server: code=2200 [Invalid query] message="Batch too large"'
> catched and printed in copyutils.py, class SendingChanel in method feed(),
> but error is just printed, records are not loaded, no error file with
> unimported rows is created and import continues. cqlsh at the end returns
> code 0 even not all rows are imported. Even that number of imported rows is
> wrong, it shows number of records in file but in fact it loaded less records.
> So I cannot be sure, based on returned code, that copy command did load all
> rows. My problem is that in this case only way to find something went wrong
> and data are not loaded correctly is to watch for error message on screen,
> which is problem when this happens in very long import script (script loading
> many tables).
> I think when not all rows are loaded correctly return code should not be 0,
> or when records aren't loaded it should exit with error immediatelly.
> $ cqlsh 127.0.0.1 --request-timeout="3600" -e "copy woc.item_container from
> '/tmp/cexport/woc/item_container.csv' with escape='\"' and null=null and
> header=True"
> Reading options from the command line: \{'header': 'True', 'null': 'null',
> 'escape': '"'}
> Using 7 child processes
> Starting copy of woc.item_container with columns [container_id, capacity,
> classes, instances, owner_id, type].
> PicklingError: Can't pickle <class 'cqlshlib.copyutil.ImmutableDict'>:
> attribute lookup cqlshlib.copyutil.ImmutableDict failed
> PicklingError: Can't pickle <class 'cqlshlib.copyutil.ImmutableDict'>:
> attribute lookup cqlshlib.copyutil.ImmutableDict failed
> PicklingError: Can't pickle <class 'cqlshlib.copyutil.ImmutableDict'>:
> attribute lookup cqlshlib.copyutil.ImmutableDict failed
> PicklingError: Can't pickle <class 'cqlshlib.copyutil.ImmutableDict'>:
> attribute lookup cqlshlib.copyutil.ImmutableDict failed
> PicklingError: Can't pickle <class 'cqlshlib.copyutil.ImmutableDict'>:
> attribute lookup cqlshlib.copyutil.ImmutableDict failed
> PicklingError: Can't pickle <class 'cqlshlib.copyutil.ImmutableDict'>:
> attribute lookup cqlshlib.copyutil.ImmutableDict failed
> PicklingError: Can't pickle <class 'cqlshlib.copyutil.ImmutableDict'>:
> attribute lookup cqlshlib.copyutil.ImmutableDict failed
> PicklingError: Can't pickle <class 'cqlshlib.copyutil.ImmutableDict'>:
> attribute lookup cqlshlib.copyutil.ImmutableDict failed
> PicklingError: Can't pickle <class 'cqlshlib.copyutil.ImmutableDict'>:
> attribute lookup cqlshlib.copyutil.ImmutableDict failed
> PicklingError: Can't pickle <class 'cqlshlib.copyutil.ImmutableDict'>:
> attribute lookup cqlshlib.copyutil.ImmutableDict failed
> Processed: 38420 rows; Rate: 7455 rows/s; Avg. rate: 6881 rows/s
> 38420 rows imported from 1 files in 5.584 seconds (0 skipped).
> $ echo $?
> 0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
