Re: startTimestamp doesn't work when using rate-micro-batch format

2024-01-29 Thread Mich Talebzadeh
As I stated earlier on,, there are alternatives that you might explore socket sources for testing purposes. from pyspark.sql import SparkSession from pyspark.sql.functions import expr, when from pyspark.sql.types import StructType, StructField, LongType spark = SparkSession.builder \ .master(

Re: startTimestamp doesn't work when using rate-micro-batch format

2024-01-29 Thread Perfect Stranger
Yes, there's definitely an issue, can someone fix it? I'm not familiar with apache jira, do I need to make a bug report or what? On Mon, Jan 29, 2024 at 2:57 AM Mich Talebzadeh wrote: > OK > > This is the equivalent Python code > > from pyspark.sql import SparkSession > from pyspark.sql.functio

Re: startTimestamp doesn't work when using rate-micro-batch format

2024-01-28 Thread Mich Talebzadeh
OK This is the equivalent Python code from pyspark.sql import SparkSession from pyspark.sql.functions import expr, when from pyspark.sql.types import StructType, StructField, LongType from datetime import datetime spark = SparkSession.builder \ .master("local[*]") \ .appName("StreamingSp