Omgili - forum search, search forums  
  

Discussions about precision

Displaying 1 - 10 out of 272,694 discussions.  
RSS Feed Options
Time Frame: (Any time)   Minimum number of replies: (2)   Minimum number of discussing users: (0)
  |  

Page: 1   2   3   4   5   6   7   8   9   10  
Keep this page open to be updated with the newest discussions automatically.
Hi, (1) I have met several cases where epsilon is added to a non-negative variable to guarantee nonzero value. So I wonder why not add the minimum value that the data type can represent instead of epsilon? What are the difference problems that these two...
Started by on , 4 posts by 4 people.  
In the case of double precision, the difference between the maximum and the next, there is no strict answer, it depends....
It depends on the precision of the floating point number of course.
Is actually 1e-16 while the second is approx 1e+11.
Hi All, Just wondering what the difference between a signle precision floating point operation and double precision floating operation is. I'm especially interested in practical terms in relation to video game consoles, for example does the nintendo 6...
Started by on , 7 posts by 7 people.  
The number of bits in double precision increases the maximum value that can be stored as well....
Basically single precision floating point arithmetic deals with 32 bit floating point numbers whereas double precision deals with 64 bit.
I'm using long double in a C program to compute 2D images of the Mandelbrot Set but wish to have further precision to zoom deeper. Are there any performance gains to be had from an arbitrary precision maths library that can restrict the amount of precision...
Started by on , 3 posts by 3 people.  
If you need more precision, see qd at http.
This wiki article contains links to several libraries.
Ask your Facebook Friends
I want to get 1/7 with better precision, but it got truncated. How can I get better precision when I convert a rational number? Thanks. >>> str(1.0/7)[:50] '0.142857142857'
Started by on , 5 posts by 5 people.  
Python's floats are never arbitrary precision....
EDIT) 14285714285714285714285714285714285714285714285714L Python's integers are arbitrary precision.
You could multiply the numerator by a large 10^N and stick with arbitrary-precision integers.
Is there a way to set Perl script's floating point precision (to 3 digits), without having to change it specifically for every variable? Something similar to TCL's: global tcl_precision set tcl_precision 3
Started by on , 3 posts by 3 people.  
BigFloat or bignum ? use Math::BigFloat; Math::BigFloat->precision(-3); my $x = Math::BigFloat->new.
Is there a way to alter the precision of an existing decimal column in Sql Server?
Started by on , 4 posts by 4 people.  
I haven't done any testing ....
You will have to modify your decimal(precision, scale) replacing the precision and scale with your desired values.
In the properties at the bottom there is a precision property None that I'm aware of.
Hello, can you tell me how can I get a high precision sleep-function in Python2.6 on Win32 and on Linux?
Started by on , 3 posts by 3 people.  
However, it's unlikely that you will get much more than millisecond precision with sleep because operating.
Why is the precision of my location in Mapkit not as good as in the maps app? The accuracy is set to best. If i'm trying to get my location in maps.app i have an acc of 50m in my app i habe 500m. What am i doing wron :)? Any tipps to get a better result...
Started by on , 4 posts by 4 people.  
GPS is very sensitive to obstructions above you, and might even be disturbed by wires etc that are connected... .
The only thing I can think of, under the assumption that you have looked through all references, is that you test them under different conditions .
I'm interested in implementing an algorithm on the GPU using HLSL, but one of my main concerns is that I would like a variable level of precision. Are there techniques out there to emulate 64bit precision and higher that could be implemented on the GPU...
Started by on , 5 posts by 5 people.  
There are a wide variety of techniques that have composed of multiple floats in whatever... .
GPUs are just beginning to support double precision in hardware, though it will continue to be much slower than single precision in the near future.
Precision loss is one thing, but precision gain??? I have a text file w/ the following coordinates: 41.88694340165634 -87.60841369628906 When I paste this into SQL Server Mgmt Studio table view, it results in this: 41.886943401656339 -87.6084136962890...
Started by on , 4 posts by 4 people.  
It's not adding precision to decimal, it only LOOKS like it gained precision..
Remember that floats are all approximate anyway.
Be stored in the db.
Page: 1   2   3   4   5   6   7   8   9   10  
Related Searches
Mentor precision    Dell Precision    dell precision 360 slow    double precision cuda    intel 4300 and dell precision    double precision in php    boards ie dell precision    cuda quadruple precision    EVGA precision 8800gt    precision 490 fan control   
More Information


Forum Search About Omgili Help Plugins Forum/Board Owners Privacy

i
In Title
In Topic
In Reply
Exclude
Boost