Re: [FFmpeg-devel] [PATCH 06/11] avutil/half2float: adjust conversion of NaN

2022-08-10 Thread Mark Reid
On Wed, Aug 10, 2022 at 3:56 PM Timo Rothenpieler wrote: > On 11.08.2022 00:37, Mark Reid wrote: > > On Wed, Aug 10, 2022 at 3:28 PM Timo Rothenpieler > > > wrote: > > > >> On 11.08.2022 00:18, James Almer wrote: > >>> Then maybe the current implementation should be moved back to exr (it > >>> u

Re: [FFmpeg-devel] [PATCH 06/11] avutil/half2float: adjust conversion of NaN

2022-08-10 Thread Timo Rothenpieler
On 11.08.2022 00:37, Mark Reid wrote: On Wed, Aug 10, 2022 at 3:28 PM Timo Rothenpieler wrote: On 11.08.2022 00:18, James Almer wrote: Then maybe the current implementation should be moved back to exr (it used to be internal to exr until Paul made it standalone), so this lavu module can match

Re: [FFmpeg-devel] [PATCH 06/11] avutil/half2float: adjust conversion of NaN

2022-08-10 Thread Mark Reid
On Wed, Aug 10, 2022 at 3:28 PM Timo Rothenpieler wrote: > On 11.08.2022 00:18, James Almer wrote: > > Then maybe the current implementation should be moved back to exr (it > > used to be internal to exr until Paul made it standalone), so this lavu > > module can match the existing hardware imple

Re: [FFmpeg-devel] [PATCH 06/11] avutil/half2float: adjust conversion of NaN

2022-08-10 Thread Timo Rothenpieler
On 11.08.2022 00:18, James Almer wrote: Then maybe the current implementation should be moved back to exr (it used to be internal to exr until Paul made it standalone), so this lavu module can match the existing hardware implementations of IEEE-734 half floats for the purpose of relevant pixel

Re: [FFmpeg-devel] [PATCH 06/11] avutil/half2float: adjust conversion of NaN

2022-08-10 Thread James Almer
On 8/10/2022 7:14 PM, Mark Reid wrote: On Wed, Aug 10, 2022 at 2:53 PM Timo Rothenpieler wrote: On 10.08.2022 23:43, Andreas Rheinhardt wrote: Timo Rothenpieler: On 10.08.2022 23:24, Andreas Rheinhardt wrote: Timo Rothenpieler: IEEE-754 differentiates two different kind of NaNs. Quiet and

Re: [FFmpeg-devel] [PATCH 06/11] avutil/half2float: adjust conversion of NaN

2022-08-10 Thread Mark Reid
On Wed, Aug 10, 2022 at 2:53 PM Timo Rothenpieler wrote: > On 10.08.2022 23:43, Andreas Rheinhardt wrote: > > Timo Rothenpieler: > >> On 10.08.2022 23:24, Andreas Rheinhardt wrote: > >>> Timo Rothenpieler: > IEEE-754 differentiates two different kind of NaNs. > Quiet and Signaling ones.

Re: [FFmpeg-devel] [PATCH 06/11] avutil/half2float: adjust conversion of NaN

2022-08-10 Thread Timo Rothenpieler
On 10.08.2022 23:43, Andreas Rheinhardt wrote: Timo Rothenpieler: On 10.08.2022 23:24, Andreas Rheinhardt wrote: Timo Rothenpieler: IEEE-754 differentiates two different kind of NaNs. Quiet and Signaling ones. They are differentiated by the MSB of the mantissa. For whatever reason, actual har

Re: [FFmpeg-devel] [PATCH 06/11] avutil/half2float: adjust conversion of NaN

2022-08-10 Thread Andreas Rheinhardt
Timo Rothenpieler: > On 10.08.2022 23:24, Andreas Rheinhardt wrote: >> Timo Rothenpieler: >>> IEEE-754 differentiates two different kind of NaNs. >>> Quiet and Signaling ones. They are differentiated by the MSB of the >>> mantissa. >>> >>> For whatever reason, actual hardware conversion of half to

Re: [FFmpeg-devel] [PATCH 06/11] avutil/half2float: adjust conversion of NaN

2022-08-10 Thread Timo Rothenpieler
On 10.08.2022 23:24, Andreas Rheinhardt wrote: Timo Rothenpieler: IEEE-754 differentiates two different kind of NaNs. Quiet and Signaling ones. They are differentiated by the MSB of the mantissa. For whatever reason, actual hardware conversion of half to single always sets the signaling bit to

Re: [FFmpeg-devel] [PATCH 06/11] avutil/half2float: adjust conversion of NaN

2022-08-10 Thread Andreas Rheinhardt
Timo Rothenpieler: > IEEE-754 differentiates two different kind of NaNs. > Quiet and Signaling ones. They are differentiated by the MSB of the > mantissa. > > For whatever reason, actual hardware conversion of half to single always > sets the signaling bit to 1 if the mantissa is != 0, and to 0 if

[FFmpeg-devel] [PATCH 06/11] avutil/half2float: adjust conversion of NaN

2022-08-10 Thread Timo Rothenpieler
IEEE-754 differentiates two different kind of NaNs. Quiet and Signaling ones. They are differentiated by the MSB of the mantissa. For whatever reason, actual hardware conversion of half to single always sets the signaling bit to 1 if the mantissa is != 0, and to 0 if it's 0. So our code has to fol