Jaikoz- Discussion Forum

There actually two type of memory used by Jaikoz, heap memory and permanent memory. The instructions only explain how to increase heap memory but it is possible to run outof permanent memory (which doesnt normally happen) . If you are running out of memory after loading a mosdest amount ofg files please try increasing your permananent memory from the default of 64mb to 150 mb as follows.

Add -XX:MaxPermSize=150m as an additional option

OSX:Info.plist
<string>-Xms150m -Xmx300m</string>
becomes
<string>-Xms150m -Xmx300m -XX:MaxPermSize=150m</string>

linux:jaikoz.sh
java -Xms150m -Xmx300m -jar lib/jaikoz.jar -l2 -m2 -f
becomes
java -Xms150m -Xmx300m -XX:MaxPermSize=150m -jar lib/jaikoz.jar -l2 -m2 -f

windows:jaikoz.bat
java -Xms150m -Xmx300m lib\jaikoz.jar -l2 -m2 -f
becomes
java -Xms150m -Xmx300m -XX:MaxPermSize=150m -jar lib\jaikoz.jar -l2 -m2 -f

windows:jaikoz.exe
here the bad news, its not possible to change this value for .exe file so if you are running into this problem you will have to use jaikoz.bat instead for the moment.

cool, thanks paul. helps somewhat. i've just run a test with:

-d64 -Xms1250m -Xmx2300m -XX:MaxPermSize=2300m

on 12,000 tracks (about 7% of my library) through autocorrect. (i have 32GB ram in my desktop, so this is not unreasonable). I'll report on higher numbers when i let this test run out in its entirety, but this is the first time i've been able to get it to be stable much over 4k.

of course, there seem to be major opportunities for optimization and parallellization, and you start to really feel those when running over a very large library. retrieve acoustic ids runs extremely slowly (~1 track per 30 seconds) with all cpus at a max of 8% utilized, seems to be only using one thread most of the time. loading songs initially is also another opportunity, its not iobound, something else is making it just crawl through the initial reads. i'll be more than happy to help you run some diagnostics offline if it will help.

chroma wrote:

retrieve acoustic ids runs extremely slowly (~1 track per 30 seconds) with all cpus at a max of 8% utilized, seems to be only using one thread most of the time.

Hmm, shoudld take about 5 seconds per track, and Jaikoz utilizes a seperate thread for each cpu when retrieving acoustic ids so if youve 4 cpus it should process 4 in parallel. If the file are on a network try moving them to local fs that should improve it.

chroma wrote:

loading songs initially is also another opportunity, its not iobound, something else is making it just crawl through the initial reads. i'll be more than happy to help you run some diagnostics offline if it will help.

This cant easily be parallelised, are you reading the files over a network that slows down stuff alot.

no, the filesystem is local, and very performant.

it 'feels' like a concurrence locking problem.

also, i have eight very fast cores.

occasionally, but rarely, it will suddenly move fast for a few tracks and will utilize all cores. but this is by far the exception. this is why i say it 'feels' like some sort of threadlocking issue.

How large can the permanent memory be set to?
My library is over 100K songs. Jaikoz only seems to be able to load about a third of that at a time. Is it possible to get everything loaded if I bump up the memory a whole lot?

Have you increased Heap memory this is the most important memory to increase, but Jaikoz will probably run a bit slow if you do load all 100,000 tracks , I would recommend working with smaller sets.

What do I do add

java -Xms150m -Xmx300m -XX:MaxPermSize=150m -jar lib\jaikoz.jar -l2 -m2 -f

jaikoz.exe

to the bat file from there? Its telling me

'java' is not recognized as an internal or external command,
operable program or batch file.

do I now have java installed correctly?

I'm recovering from a hard drive crash and my backups weren't tagged ironically I thin the drive failed because I was tagging so I could back it up. I'm doing about 12k in files. I have 4 gig of ram. How do I give it like a gig of ram?

Oh yeah a copy of my path

%CommonProgramFiles%\Microsoft Shared\Windows Live%c:\Program Files (x86)\Java\jre6\bin%

You either edit jaikoz.bat and run Jaikoz.bar or you edit the properties on Jaikoz.exe and run Jaikoz.exe

There should not be a % at the end of your path. The error you were getting is because your Windows system cannot find Java.

What is the status of this information with regards to the new release as of a couple days ago - and especially with regards to both the java and .exe versions for Windows?

chroma wrote:

cool, thanks paul. helps somewhat. i've just run a test with:

-d64 -Xms1250m -Xmx2300m -XX:MaxPermSize=2300m

on 12,000 tracks (about 7% of my library) through autocorrect. (i have 32GB ram in my desktop, so this is not unreasonable). I'll report on higher numbers when i let this test run out in its entirety, but this is the first time i've been able to get it to be stable much over 4k.

of course, there seem to be major opportunities for optimization and parallellization, and you start to really feel those when running over a very large library. retrieve acoustic ids runs extremely slowly (~1 track per 30 seconds) with all cpus at a max of 8% utilized, seems to be only using one thread most of the time. loading songs initially is also another opportunity, its not iobound, something else is making it just crawl through the initial reads. i'll be more than happy to help you run some diagnostics offline if it will help.

I believe you still have to use the bat file on windows.

As for the rest of the information, still seems valid, although I note the beta defaults the MaxPerm to 200 now instead of 150.

I just opened 100K+ files on my 6GB machine today with these settings

<string>-Xms300m -Xmx4200m -XX:MaxPermSize=300m</string>

and was running a few other apps and still had over a GB of RAM left. It was still fairly reactive (a little delay but that is expected) and I had it chugging on a couple of thousand files. (autocorrecting)

(This is OSX)

chroma wrote:

no, the filesystem is local, and very performant.

it 'feels' like a concurrence locking problem.

Yep, it's locking.

Whenever I run Jaikoz over more than the recommended number of files*, my harddisks die.

Absolutely no access to anything on them, everything stalls etc..

* The recommended number of files, according to Paul, is "less" than the number of files which reveal the programming errors in Jaikoz, being ONE file at a time as nothing is less than that and two is more.

Thanks Paul!

Sounds like you are jumping to invalid conclusions here have you actually tried increasing memory allocated ?

To summarise, with Java you cannot automatically use available memory on your system unless you have configured Java application accordingly, and this cannot be done by the installer because only the user can know what is an acceptable upper memory limit to use.

Jaikoz memory usage is proportional to how many files are opened in Jaikoz, because the metadata is stored in memory, and this is the bulk of the memory usage. It was not really envisaged that users would load their whole large collection into Jaikoz in one go, in which case there is no problem. This is a problem which affects many taggers not just Jaikoz.

With hindsight it may have been better to store details in a database and only have a few screens worth of the songs metadata held in memory at one time, but this is not without its drawbacks and its very difficult to retrofit this into the existing application, but Im still hoping to do that.