[Logo] Jaikoz and SongKong Forums
  [Search] Search   [Recent Topics] Recent Topics   [Members]  Member Listing   [Groups] Back to home page 
[Register] Register / 
[Login] Login 
Running extremely slow and maxing out all CPUs  XML
Forum Index -> SongKong Beta
Author Message
greengeek

Pro
[Avatar]

Joined: 18/09/2007 02:50:48
Messages: 435
Offline

I find SK does great on single albums of only 20 or so songs. I then tried pointing it to my junk folder that has about 1500 mp3 files that are not tagged. SK started out nice and quick but then really slowed down to a near stand still. Jaikoz usually will fix a folder of this size in about 30 minutes. I am now going onto about 4 hours with SK barely moving and my computer at a near unusable point. The memory is hardly being touched at only 25% total system usage,but it is wrecking my CPUs. All 4 cores stay maxed out at 100% usage.

The system I am testing it on is a slightly older mid level system. Stats are:
win 7 64 pro
amd phenom ii x4 940 processor 3.00ghz
8gb of ddr2

KevinBluemel.com
Contemporary Instrumental Musician and Composer. Stop by to listen, watch videos, and get free downloads of songs and sheet music.
[WWW]
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7215
Online

Hmm, this is very much like the problem some users report on Jaikoz, and most do not encounter.

What would be useful is if you could get a stack trace of the system when it slows down so we can see what is happening

http://docs.oracle.com/javase/6/docs/technotes/tools/share/jstack.html

thanks Paul (Administrator)
greengeek

Pro
[Avatar]

Joined: 18/09/2007 02:50:48
Messages: 435
Offline

I did a long jstack dump. I had to set it to force as it would not take otherwise. I have mailed the dump file to you at the support@jthink.net address. I noticed when I used it with the -F switch that it would free up the CPUs for a few seconds, before going right back into the max state. Without the -F switch jstack would just hang.

This is the first time I have used this tool, so please let me know if that is the information you need.

Thank you.

KevinBluemel.com
Contemporary Instrumental Musician and Composer. Stop by to listen, watch videos, and get free downloads of songs and sheet music.
[WWW]
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7215
Online

Thanks, I'm not very familiar with jstack myself.

But it does seem that if JStack without -F didnt work and then using -F did work
and caused SongKong to appear to work for a few seconds that jstack is having an unintended effect of freeing a thread

http://stackoverflow.com/questions/12061226/how-does-jstack-f-affect-a-running-java-process

I've looked at the jstack and it has some useful informstion but Im unclear if it is showing anything wrong.

I can see:

Four threads waiting to send request to Musicbrainz but having to obey mandatory 1 second delay between requests.
Two threads scoring potential matches

Then there are two threads where jstack reported an error trying to determine what they were doing.

no other threads are doing anything interesting.


Have you tried the run again , would be interesting to know if this was failng conistently for you

thanks Paul (Administrator)
greengeek

Pro
[Avatar]

Joined: 18/09/2007 02:50:48
Messages: 435
Offline

Yes, I get the same behavior every time I run it. I have tried 4 attempts now and all the CPUs start maxing out about an hour into it, shortly after all the tracks have been loaded and it is working on matching.

KevinBluemel.com
Contemporary Instrumental Musician and Composer. Stop by to listen, watch videos, and get free downloads of songs and sheet music.
[WWW]
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7215
Online

Just to clarify, its stats working on matching as soon as it has loaded the first folder, not after loading all songs. Could you try increasing debug log level in preferences and rerunning

Ive added this problem to issue tracker
http://www.jthink.net:8081/browse/SONGKONG-54

thanks Paul (Administrator)
greengeek

Pro
[Avatar]

Joined: 18/09/2007 02:50:48
Messages: 435
Offline

True, but for me it doesn't seem to max out until after it has already completely loaded the first bar which might just be coincidental. It is shortly after it completely loads all the songs that it starts to max out.

What would you like me to set the debug level to, and how do I get the logs to you after?

KevinBluemel.com
Contemporary Instrumental Musician and Composer. Stop by to listen, watch videos, and get free downloads of songs and sheet music.
[WWW]
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7215
Online

Hi, Ive just released SongKong beta 2 please install that first. There is a new action added to the Help Menu called Create Support FIles that works in the same way as it does in Jaikoz.

In preferences set the Debug Log Level to CONFIG and leave the IO Debug Level at WARNING.

thanks Paul (Administrator)
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7215
Online

I'm not sure why it appears to be doing nothing, but I can see a problem with my logic for cpu stuff, and can see how we could have more threads trying to run then we should have later on in the process.

Perhaps we have so many threads trying to run that we waste alot of time with cpu contention, swapping between processes.

thanks Paul (Administrator)
greengeek

Pro
[Avatar]

Joined: 18/09/2007 02:50:48
Messages: 435
Offline

Was running low on disc space so had to clear out all my untagged loose files. I copied a bunch of already tagged files I did a long time ago in jaikoz to see how SK would handle them. Was hoping to at least start up jaikoz to clear out all the MBIDs info so they would appear somewhat untagged before running them across SK, but I can't even get jaikoz to run now.

I installed the latest version of SK and will try it against these well tagged files. Not sure if it will see all the existing mbids and just skip over them or not, or if I need files that don't contain all that info. If that is the case I will need to find a way to modify those files first.

I will give it a try and let you know what happens.

KevinBluemel.com
Contemporary Instrumental Musician and Composer. Stop by to listen, watch videos, and get free downloads of songs and sheet music.
[WWW]
greengeek

Pro
[Avatar]

Joined: 18/09/2007 02:50:48
Messages: 435
Offline

I am running a test now. I noticed that the CPUs are still maxing out. However instead of it hitting all 4 with 100%, it is occasionally dropping down to 99% every few seconds. Also the songs fingerprinted is still moving and increasing so it is not fully locking up this time.

It appears that fpcalc is running multiple instances. The resource monitor is showing anywhere from a few to 20+ processes. However only a few are running at a time. Most would show the status of being terminated, but yet still retaining 20% to 22% of cpu usage each.

Once the songs fingerprinted got done, which was around 90%, the fpcalc processes cleared out totally. At that point the total cpu usage dropped way down to around 2% to 30%, typically averaging around 5%. I have about 3k files loaded currently that it is working on. I will email the log files once it completes or stops.

KevinBluemel.com
Contemporary Instrumental Musician and Composer. Stop by to listen, watch videos, and get free downloads of songs and sheet music.
[WWW]
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7215
Online

greengeek wrote:
Not sure if it will see all the existing mbids and just skip over them or not, or if I need files that don't contain all that info. If that is the case I will need to find a way to modify those files first.
 

When you start b]Fix Songs[/b] on the first tab is the option Search for a Musicbrainz match even if Song already has a Musicbrainz Id lets you decide whether SongKong should ignore existing id or not.

thanks Paul (Administrator)
greengeek

Pro
[Avatar]

Joined: 18/09/2007 02:50:48
Messages: 435
Offline

I emailed you the support files from beta 2. I will download beta 3 and try that and let you know what happens.

KevinBluemel.com
Contemporary Instrumental Musician and Composer. Stop by to listen, watch videos, and get free downloads of songs and sheet music.
[WWW]
greengeek

Pro
[Avatar]

Joined: 18/09/2007 02:50:48
Messages: 435
Offline

Just noticed beta 3 that is mentioned in jira is not posted yet.

KevinBluemel.com
Contemporary Instrumental Musician and Composer. Stop by to listen, watch videos, and get free downloads of songs and sheet music.
[WWW]
greengeek

Pro
[Avatar]

Joined: 18/09/2007 02:50:48
Messages: 435
Offline

Just downloaded beta 3. I will update you with what I find out if this issue is still happening.

KevinBluemel.com
Contemporary Instrumental Musician and Composer. Stop by to listen, watch videos, and get free downloads of songs and sheet music.
[WWW]
greengeek

Pro
[Avatar]

Joined: 18/09/2007 02:50:48
Messages: 435
Offline

So tried beta 3 today. CPUs never maxed out. Program never froze. The song fix completed. Looking good!

KevinBluemel.com
Contemporary Instrumental Musician and Composer. Stop by to listen, watch videos, and get free downloads of songs and sheet music.
[WWW]
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7215
Online

Great could you send me your support files as well please, because there is debugging that should help me see why you were encountering the issue. Ive added some code to bail out if get stuck in a loop, but don't have a real world example whereby it could get stuck in a loop.

thanks Paul (Administrator)
greengeek

Pro
[Avatar]

Joined: 18/09/2007 02:50:48
Messages: 435
Offline

Sure. The files have been emailed. As mentioned above, this run did not get stuck using beta 3, nor did it even come close to maxing out any of the cpus.

KevinBluemel.com
Contemporary Instrumental Musician and Composer. Stop by to listen, watch videos, and get free downloads of songs and sheet music.
[WWW]
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7215
Online

Thanks, I can see from the logs that the problem did reoccur and the fix resolved the problem, so that is good news.

thanks Paul (Administrator)
 
Forum Index -> SongKong Beta
Go to:   
Powered by JForum 2.1.6 © JForum Team