[Logo] Jaikoz and SongKong Forums
  [Search] Search   [Recent Topics] Recent Topics   [Members]  Member Listing   [Groups] Back to home page 
[Register] Register / 
[Login] Login 
Memory usage  XML
Forum Index -> Jaikoz Questions
Author Message
rlyster

Pro
[Avatar]
Joined: 12/08/2010 11:16:45
Messages: 29
Offline

Jaikoz uses 400 MB of memory by default, and that is supposedly good for 10,000 songs. Is there a "rule of thumb" on how much more memory to allocate when the song library is bigger than that - i.e., so manymore MB of memory for each 10,000 additional songs?
[WWW]
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7361
Offline

No, not really - I generally recommend you work in batches though.

thanks Paul (Administrator)
lyster



Joined: 03/08/2010 16:04:56
Messages: 7
Offline

Say you've worked in batches and have finished Autocorrecting 70,000 + songs. Next you wish to remove duplicates. How does one do that? Do you need to load all 70,000 songs?
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7361
Offline

For this yes you would.

So in this instance just increase your memory allocation to whatever you can, load the files and run delete duplicates. But I wouldn't recommend running with this many files in normal circumstances.

thanks Paul (Administrator)
MAzE5h1p69wB

Pro

Joined: 21/04/2010 14:05:13
Messages: 83
Offline

Translation: No, remove duplicates do not work for 100% of users of this program due to the way it has been written.
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7361
Offline

Okay MAzE5h1p69wB we have gone though this, yes Jaikoz does not scale indefinitently to super huge collections, because when originally written a few years ago it wasn't envisaged that users would be trying to process these collections in one go, and Jaikoz did not even have a Delete Duplicates function, you'll find that most taggers have limits on how much they can cope with.

But I have a clear roadmap on how I will be solving this problem, and it will be solved by the end of 2010,

thanks Paul (Administrator)
rlyster

Pro
[Avatar]
Joined: 12/08/2010 11:16:45
Messages: 29
Offline

What is the worst thing that could happen using the delete duplicates function? Should I wait until the end of the year to take advantage of this feature?
[WWW]
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7361
Offline

Ooh no dont wait, just increase the memory as required and run it.

The only bad thing that could happen is Jaikoz runs out of memory before Delete Duplicates finishes, no harm will come to your files

thanks Paul (Administrator)
rlyster

Pro
[Avatar]
Joined: 12/08/2010 11:16:45
Messages: 29
Offline

I have 8GB Ram - and can let Jaikoz have as much as it needs. I've got Jaikoz set at 1GB currently. Let's see how that goes....
[WWW]
rlyster

Pro
[Avatar]
Joined: 12/08/2010 11:16:45
Messages: 29
Offline

Here's the line I have in the shortcut that starts Jaikoz -
"C:\Program Files (x86)\Jthink\Jaikoz\jaikoz.exe" -Jmaxheap=3072000000
Does that increase available memory to 3GB? If not, what am I doing wrong? If so, I'm still running out of memory when trying to load 74,000 songs....
[WWW]
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7361
Offline

You could be hitting the PermGen problem.

Java uses two different memory areas , heap and permgen. Heap is used for most things but permgen is used for storing text and processing images, unfortunately permgen cannot be changed for Jaikoz.exe, but it can be changed for Jaikoz.bat instead which runs exactly the same thing.

Open Jaikoz.bat in text editor an edit the following parameters

Code:
 Xmx=800m -XX:MaxPermSize=200m 
 

Modify heap (-Xmx) and then Max perm size by proportional amount , i.e
Code:
 Xmx=2400m -XX:MaxPermSize=600m 
 


Then make sure you start Jaikoz using Jaikoz.bat NOT Jaikoz.exe, you can always check how much memory has been allocated to Jaioz by checking the Console tab within Jaikoz and looking at the first line entered a default installation should say something like

Code:
 Aug 15, 2010 8:52:06 PM: INFO: Jaikoz 3.7.1 using Java 1.6.0_20 16.3-b01-279 64bit on Mac OS X 10.6.4 x86_64 initialized successfully 
 Aug 15, 2010 8:52:06 PM: INFO: Jaikoz has been configured with minimum heap memory of 150 mb, maximum heap memory of 795 mb and maximum permanent memory of 216 mb
 

thanks Paul (Administrator)
rlyster

Pro
[Avatar]
Joined: 12/08/2010 11:16:45
Messages: 29
Offline

Here's that .bat file:
@echo off
set CLASSPATH=
java -Xms150m Xmx=2400m -XX:MaxPermSize=600m -jar lib\jaikoz.jar -l2 -m2 -f

How do you run it?
[WWW]
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7361
Offline

Just doubleclick on it

thanks Paul (Administrator)
rlyster

Pro
[Avatar]
Joined: 12/08/2010 11:16:45
Messages: 29
Offline

Guess I'll need to do some troubleshooting then - it doesn't run.
[WWW]
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7361
Offline

Okay, open a command window and run it from there instead that should give an error message indicating the problem. I guess you are actually using 32bit java rather than 64bit Java in which case there is a maximum limit of how much memory can be allocated of about 2GB, so switch to 64bit if you can, if not reduce the total memory to less than 2GB.

thanks Paul (Administrator)
rlyster

Pro
[Avatar]
Joined: 12/08/2010 11:16:45
Messages: 29
Offline

You were right.... I hadn't realized it, but I was running 32-bit Java. I uninstalled that, ran the 64-bit IE, installed 64-bit Java, and.... We'll see. Not working as of this writing....
[Thumb - java.jpg]
 Filename java.jpg [Disk] Download
 Description Here's the error message
 Filesize 88 Kbytes
 Downloaded:  385 time(s)

[WWW]
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7361
Offline

Looks like you've made a mistake editing Jaikoz.bat, can you paste the contents onto the forum

thanks Paul (Administrator)
lyster



Joined: 03/08/2010 16:04:56
Messages: 7
Offline

Sure... Jump back a few messages in this thread and you'll see it... Here it is again:
@echo off
set CLASSPATH=
java -Xms150m Xmx=2400m -XX:MaxPermSize=600m -jar lib\jaikoz.jar -l2 -m2 -f
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7361
Offline

lyster wrote:
Sure... Jump back a few messages in this thread and you'll see it... Here it is again:
@echo off
set CLASSPATH=
java -Xms150m Xmx=2400m -XX:MaxPermSize=600m -jar lib\jaikoz.jar -l2 -m2 -f  


ahh, there is a spurious equals sign it should be

Code:
  java -Xms150m Xmx2400m -XX:MaxPermSize=600m -jar lib\jaikoz.jar -l2 -m2 -f 
 

thanks Paul (Administrator)
lyster



Joined: 03/08/2010 16:04:56
Messages: 7
Offline

I'll try that when I get home tonight. (I'm in Pacific Daylight Time.) Thank you! (Note I got that "=" by pasting the suggested revision from the message just prior to the one I mentioned above... the one where the idea of using the batch file was first raised....)
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7361
Offline

Sorry about that, I was also the road when I posted that reply from my phone without access to a PC to check.

thanks Paul (Administrator)
rlyster

Pro
[Avatar]
Joined: 12/08/2010 11:16:45
Messages: 29
Offline

Here's my latest try....
@echo off
set CLASSPATH=
java -Xms150m Xmx2400m -XX:MaxPermSize=600m -jar lib\jaikoz.jar -l2 -m2 -f

And the result... And one more issue. Jaikoz is not recognizing 64-bit Java. It tells me I have no Java, and offers to take me to the Java website to install it.
[Thumb - batch.jpg]
 Filename batch.jpg [Disk] Download
 Description
 Filesize 43 Kbytes
 Downloaded:  401 time(s)

[WWW]
rlyster

Pro
[Avatar]
Joined: 12/08/2010 11:16:45
Messages: 29
Offline

I am praying that this resolves the problem... But I put a hyphen before "Xmx2400m" and the program runs.
[WWW]
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7361
Offline

Ah yes the hyphen is required as well.

thanks Paul (Administrator)
rlyster

Pro
[Avatar]
Joined: 12/08/2010 11:16:45
Messages: 29
Offline

Loading has seemingly stopped at this....
[Thumb - stuck.jpg]
 Filename stuck.jpg [Disk] Download
 Description
 Filesize 15 Kbytes
 Downloaded:  407 time(s)

[WWW]
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7361
Offline

How many could you do before changing memory setting ?

I'm suprised its just hanging rather than giving an error message which is what should happen if it has run out of heap memory, can you cancel the task ?

Could you change the debugging levels for jaikoz.bat by modifying

Code:
-m2

to
Code:
-m3


and run again, would be interesting to see if it hangs at about the same location indicating a problem with a specific song, and then send me your support file using Advanced/Create Support Files.

thanks Paul (Administrator)
lyster



Joined: 03/08/2010 16:04:56
Messages: 7
Offline

I'll make that change and let you know. It got to 29K songs before, so this is significantly better. I was able to cancel out, but could do nothing with the files that had been loaded. My entire machine was slowed down. I exited the command window, and my machine was responsive once again.
rlyster

Pro
[Avatar]
Joined: 12/08/2010 11:16:45
Messages: 29
Offline

I increased the debugging level to -m3, as you suggested, but it turns out I didn't need it. I also incresed the other parametersto allocate 4 GB of ram and this time it worked! Thanks for all your help.
[WWW]
 
Forum Index -> Jaikoz Questions
Go to:   
Powered by JForum 2.1.6 © JForum Team