Quantcast

Datatype double: missing digits

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Datatype double: missing digits

Andreas Linder
Hi,
Selecting data from one table with a double precise column delivers the following output:
42013.946988
42013.946984
42013.947016
42013.947016

The real data has a higher precision and digits 7,8,.. are missing.
Is this hard coded or could that be changed? Working with ms-values in double format is not possible this way.

BR,
Andreas

------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________
Flamerobin-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/flamerobin-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Datatype double: missing digits

Michael Hieke
Hi,

On 27.04.2015 12:08, Andreas Linder wrote:

> Selecting data from one table with a double precise column delivers
> the following output:
 >
> 42013.946988
> 42013.946984
> 42013.947016
> 42013.947016
>
> The real data has a higher precision and digits 7,8,.. are missing.
> Is this hard coded or could that be changed? Working with ms-values
> in double format is not possible this way.

you can force all single and double precision numbers to be formatted
with a specific number of decimals, see screen shot.  The setting is on
the "Data Grid" page of the dialog.

HTH

--
Michael Hieke


------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________
Flamerobin-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/flamerobin-devel

double_precision_format.png (63K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Datatype double: missing digits

Andreas Linder
Hi Michael,

Thanks!
For double this is working fine, the float-column (the 2nd) however is not affected by this setting:
42055.000073958334 69343.000000
42055.000073958334 11.972728
42055.000073958334 54399.000000

Am I missing something?

Cheers,
Andreas

-----Original Message-----
From: Michael Hieke [mailto:[hidden email]]
Sent: Montag, 27. April 2015 12:34
To: [hidden email]
Subject: Re: [Flamerobin-devel] Datatype double: missing digits

Hi,

On 27.04.2015 12:08, Andreas Linder wrote:

> Selecting data from one table with a double precise column delivers
> the following output:
 >
> 42013.946988
> 42013.946984
> 42013.947016
> 42013.947016
>
> The real data has a higher precision and digits 7,8,.. are missing.
> Is this hard coded or could that be changed? Working with ms-values in
> double format is not possible this way.

you can force all single and double precision numbers to be formatted with a specific number of decimals, see screen shot.  The setting is on the "Data Grid" page of the dialog.

HTH

--
Michael Hieke


------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________
Flamerobin-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/flamerobin-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Datatype double: missing digits

Michael Hieke
Hi,

On 27.04.2015 14:31, Andreas Linder wrote:

> For double this is working fine, the float-column (the 2nd) however
> isnot affected by this setting:
> 42055.000073958334 69343.000000
> 42055.000073958334 11.972728
> 42055.000073958334 54399.000000
>
> Am I missing something?

I don't think you are missing something, it most probably should just be
labeled/documented accordingly.

It shouldn't matter though, a single precision value has only 7
significant digits anyway, IIRC.


Regards

--
Michael Hieke


------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________
Flamerobin-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/flamerobin-devel
Loading...