|
I am writing a function to extract decimals from a number. Ignore the exception and its syntax, I am working on 2.5.2 (default Leopard version). My function does not yet handle 0's. My issue is, the function produces random errors with certain numbers...
Started by dbmikus on
, 7 posts
by 7 people.
Answer Snippets (Read the full thread at stackoverflow):
In [49]: num Out[49]: 1.988 In [50]: ....
See extract_d time: 0.753302766373
As Ned Batchelder said, not all decimals are exactly representable.
The problem is that (binary) floating point numbers aren't precisely representable as decimals.
|
|
When 2 decimal(30,10) numbers are divided in Sql Server 05, 2 last decimals seem to be getting lost (not even rounded off, simply truncated).
For example:
Declare @x decimal(30,10) Declare @y decimal(30,10) Declare @z decimal(30,10) select @x = 2.1277...
Started by WebMatrix on
, 3 posts
by 3 people.
Answer Snippets (Read the full thread at stackoverflow):
The....
This helps to explain why division by those values comes up short of your expectations.
When you assign @x and @y to literal values, they are probably adopting the precision of those literals .
In short, use casting to guarantee your results.
|
|
Is there an algorithm for figuring out the following things?
If the result of a division is a repeating decimal (in binary). If it repeats, at what digit (represented as a power of 2) does the repetition start? What digits repeat? Some examples:
1/2 =...
Started by Imagist on
, 6 posts
by 6 people.
Answer Snippets (Read the full thread at stackoverflow):
I can give a hint - repeating decimals in base ten are all fraction.
Of a fraction.
|
Ask your Facebook Friends
|
Select 2/3 how to result 0.67 via sql sever 2005
Started by monkey_boys on
, 6 posts
by 6 people.
Answer Snippets (Read the full thread at stackoverflow):
ALTER FUNCTION GetDecimalFromDivided(@interger1 INT, @interger....
Cheers
Try this
select cast(ROUND(2/3.0, 2) as decimal(2,2))
Hope that will help.
SELECT ROUND(2.0/3.0, 2)
Hi
Select cast(2.0/3.0 as decimal(3,2)) as result , should help.
|
|
I have some code to convert a time value returned from QueryPerformanceCounter to a double value in milliseconds, as this is more convenient to count with.
The function looks like this:
double timeGetExactTime() { LARGE_INTEGER timerPerformanceCounter...
Started by Adion on
, 3 posts
by 3 people.
Answer Snippets (Read the full thread at stackoverflow):
It throws....
You are correct about the size of the numbers.
Then cast the resulting number back to double.
Adion,
If you don't mind the performance hit, cast your QuadPart numbers to decimal instead of double before performing the division.
|
|
I need some division algorithm which can handle big integers (128-bit). I've already asked how to do it via bit shifting operators. However, my current implementation seems to ask for a better approach
Basically, I store numbers as two long long unsigned...
Started by Etan on
, 6 posts
by 6 people.
Answer Snippets (Read the full thread at stackoverflow):
Of decimals (1/24 is for instance 0.041 ...) and can therefore interfere with the fourth term in your.
|
|
Hi.
I'm trying to calculate a percentage "factor". That is, given a 20%, convert it into 0.2 (my intention is to later multiply values by that and get the 20% of the values).
Anyway, the question is related with this piece of code:
public static void ...
Started by gonso on
, 5 posts
by 5 people.
Answer Snippets (Read the full thread at stackoverflow):
That means that your percentage.divide(hundred, BigDecimal.ROUND_FLOOR) will produce zero (it's effectively... .
You probably want to use ROUND_UP as rounding mode
The scale of new BigDecimal("20") is zero because you've got no decimal point in there.
|
|
Here is the code I'm using in the example:
PRINT @set1 PRINT @set2 SET @weight= @set1 / @set2; PRINT @weight
Here is the result:
47 638 0
I would like to know why it's returning 0 instead of 0,073667712
Started by mnml on
, 6 posts
by 6 people.
Answer Snippets (Read the full thread at stackoverflow):
You need to declare them as floating point numbers or decimals, or cast.
Because it's an integer.
|
|
Hello everyone,
I recently came across a problem that sparked my curiosity. Try to solve this problem using long division w/ decimals instead of remainders, solving to 4 decimal places. After you have done that, use long division w/ remainder. Finally...
Started by asnyder171 on
, 3 posts
by 3 people.
Answer Snippets (Read the full thread at mymathforum):
After you.
Solving to 4 decimal places.
Using long division w/ decimals instead of remainders,
.
|
|
I have two unsigned long longs X and Y, where X < Y, but both may be very large. I want to compute the first digit past the decimal point of X / Y. For example, if X is 11 and Y is 14, then 11 / 14 is .785…, so the result should be 7.
(X * 10) / Y ...
Answer Snippets (Read the full thread at stackoverflow):
division:
(long)((long)10*R/Y)
This should round down the numbers and drop any extra decimals.
|
