[jira] [Commented] (HIVE-18942) ALTER TABLE may generate huge event (with all partitions)

[kalyan kumar kalvagadda (JIRA)]

    [ 
https://issues.apache.org/jira/browse/HIVE-18942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16396929#comment-16396929
 ] 

kalyan kumar kalvagadda commented on HIVE-18942:
------------------------------------------------

[~akolb] That may not be the case.

AddPartitionEvent that is generated here is different from the 
NotificationEvent that is inserted in the NOTIFICATION_LOG table.

{noformat}
  public void onAlterTable(AlterTableEvent tableEvent) throws MetaException {
    Table before = tableEvent.getOldTable();
    Table after = tableEvent.getNewTable();
    NotificationEvent event =
        new NotificationEvent(0, now(), EventType.ALTER_TABLE.toString(), 
msgFactory
            .buildAlterTableMessage(before, after, 
tableEvent.getIsTruncateOp()).toString());
    event.setDbName(after.getDbName());
    event.setTableName(after.getTableName());
    process(event, tableEvent);
  }
{noformat}
 
This is how later table JSON is constructed as sentry is interested in below 
things in the alter table notification
# Old Db Name
# New Db Name
# Old Table Name
# New Table Name
# Old Location
# New Location

{noformat}
  public JSONAlterTableMessage(String server, String servicePrincipal, Table 
tableObjBefore, Table tableObjAfter, Long timestamp) {
    this.server = server;
    this.servicePrincipal = servicePrincipal;
    this.db = tableObjBefore.getDbName();
    this.table = tableObjBefore.getTableName();
    this.timestamp = timestamp;

    try {
      this.tableObjBeforeJson = 
JSONMessageFactory.createTableObjJson(tableObjBefore);
      this.tableObjAfterJson = 
JSONMessageFactory.createTableObjJson(tableObjAfter);
    } catch (TException var7) {
      throw new IllegalArgumentException("Could not serialize: ", var7);
    }

    this.checkValid();
  }
{noformat}


I think the NotificationEvent inserted in NOTIFICATION_LOG table may not have 
details of the partitions. 

> ALTER TABLE may generate huge event (with all partitions)
> ---------------------------------------------------------
>
>                 Key: HIVE-18942
>                 URL: https://issues.apache.org/jira/browse/HIVE-18942
>             Project: Hive
>          Issue Type: Bug
>          Components: Metastore
>    Affects Versions: 3.0.0
>            Reporter: Alexander Kolbasov
>            Priority: Major
>
> ALTER TABLE handler in HiveAlterHandler has this code:
> {code:java}
> if (isPartitionedTable) {
>   parts = msdb.getPartitions(newt.getDbName(), newt.getTableName(), -1);
>   MetaStoreListenerNotifier.notifyEvent(transactionalListeners,
>           EventMessage.EventType.ADD_PARTITION,
>           new AddPartitionEvent(newt, parts, true, handler),
>           environmentContext);
> }{code}
> The problem is that table may contain huge number of partitions and the event 
> will contain all of them. Partition object itself isn't very small either, so 
> we may end up with huge events which would be stored and then transmitted 
> over the wire to consumers.
> [~spena] [~kkalyan] [~lina.li] [~vaidyand] FYI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)