[Logo] Jaikoz and SongKong Forums
  [Search] Search   [Recent Topics] Recent Topics   [Members]  Member Listing   [Groups] Back to home page 
[Register] Register / 
[Login] Login 
moving unmatched files  XML
Forum Index -> SongKong Issues Go to Page: Previous  1, 2
Author Message
greengeek

Pro
[Avatar]

Joined: 18/09/2007 02:50:48
Messages: 435
Offline

Thanks for the update. Going to try a run of about 4000 songs and see how it does

KevinBluemel.com
Contemporary Instrumental Musician and Composer. Stop by to listen, watch videos, and get free downloads of songs and sheet music.
[WWW]
greengeek

Pro
[Avatar]

Joined: 18/09/2007 02:50:48
Messages: 435
Offline

Just tried it on about 300 albums and it is looking really good! Thanks!!!

KevinBluemel.com
Contemporary Instrumental Musician and Composer. Stop by to listen, watch videos, and get free downloads of songs and sheet music.
[WWW]
greengeek

Pro
[Avatar]

Joined: 18/09/2007 02:50:48
Messages: 435
Offline

Paul,

So I bumped it up to a run of 25,000 mp3s using a local musicbrainz server. It moved multiple albums that jaikoz came back as missing. There was also an album that couldnt be matched that was not moved to the unmatched folder as well as a song it left behind after moving the rest of the album. This took about 24 hours to run.

I will send you the support files for sk and the reports for jaikoz.

KevinBluemel.com
Contemporary Instrumental Musician and Composer. Stop by to listen, watch videos, and get free downloads of songs and sheet music.
[WWW]
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7361
Offline

So I think you are using your (out of date) local musicbrainz server for SongKong and the real server for Jaikoz.
SongKong has actually matched the whole album ( as can be seen from looking at the releaseid), however the recordings that Jaikoz lists as missing are ones that have been merged

ie. 'Call the Shots' by Girls Aloud was originally

https://musicbrainz.org/recording/4cda4bd0-21f6-46ba-ae45-80ab072bb062

but this was merged into:

https://musicbrainz.org/recording/fb064be9-565c-4f3e-985f-e2a19b33c248

in December 2013

https://musicbrainz.org/recording/fb064be9-565c-4f3e-985f-e2a19b33c248/edits

Both ids now refer to same recording, but the current id that is returned for the track in the release is fb064be9-565c-4f3e-985f-e2a19b33c248

So if Im right about the Musicbrainz servers SongKong has worked correctly in this respect, but Jaikoz missing Track report could be improved to recognise old ids

There is the bug in SongKong that you have noticed with artist field having a trailing ', and space'

I also tried Update Metadata from Jaikoz and this failed to update all the recording with the old ids, so this is another bug in Jaikoz

thanks Paul (Administrator)
greengeek

Pro
[Avatar]

Joined: 18/09/2007 02:50:48
Messages: 435
Offline

Interesting. I keep mine updated, or I think i do, as I run the replication now command before I use it. Hard to say as the page auto clears before it finishes, so I can't tell if it throws any errors or not. I noticed that the statistics page on musicbrainz says it was Last updated: 2014-05-15 with ARtists: 846,105, yet the same page on the vm shows Last updated: 2013-11-23 with 799,525 artists. So if that page is suppose to update in real time, I would assume there is something wrong with the VM or at least with mine.

KevinBluemel.com
Contemporary Instrumental Musician and Composer. Stop by to listen, watch videos, and get free downloads of songs and sheet music.
[WWW]
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7361
Offline

Hi, you can run

Code:
 bin/replicate now
 

from the command line to bring the database up to date, you should be able to see if that has worked. There was an issue with a replication packets though - that might be what you are hitting. There should be a new VM available within the next few days now that they have just done their new schema release, that will then bring everything up to date.

http://blog.jthink.net/2014/05/musicbrainz-adds-series-support_15.html

(Also you need to rebuild search indexes as well so they correlate with the database using bin/reindex)

thanks Paul (Administrator)
greengeek

Pro
[Avatar]

Joined: 18/09/2007 02:50:48
Messages: 435
Offline

So I ended up installing a UI on my vm. I was then able to run the terminal inside the ui which allowed me to scroll through all the output. The vm without the ui wouldnt let me scroll more than 2 lines up even using the | less or | more commands. I was then able to see that replication was running into some issues. A little research showed that it was related to a known bug and I needed to follow this, http://blog.musicbrainz.org/2013/12/05/important-update-on-replication/ , to fix it.

I did that update and now am running another replicate which is pulling far more info. Will try another run of SK soon as this replication finishes.

KevinBluemel.com
Contemporary Instrumental Musician and Composer. Stop by to listen, watch videos, and get free downloads of songs and sheet music.
[WWW]
greengeek

Pro
[Avatar]

Joined: 18/09/2007 02:50:48
Messages: 435
Offline

So I finished running a batch of 250,000 files. It took little under 7 days to run and songkong did not crash at all. It moved all the files extra for 2 albums, about 25 files. 78% files were moved as matched. 22% were moved to non matched.

I did notice a few interesting things. Sk reported it was using on avg 700 of 2200 mb. However windows task manager was showing it was around 3000 mb being used. So the numbers in the program ui and windows task manager did not match up.

I am also currently verifying the matched files first. I am still fairly early on in the process, but i am noticing about 2% of the albums were moved as complete even though they did not contain any musicbrainz IDs. i left Search for Discogs match checked when I ran sk. Do files that were not matched in musicbrainz but maybe matched in discogs get moved to the matched folder?


KevinBluemel.com
Contemporary Instrumental Musician and Composer. Stop by to listen, watch videos, and get free downloads of songs and sheet music.
[WWW]
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7361
Offline

greengeek wrote:
So I finished running a batch of 250,000 files. It took little under 7 days to run and songkong did not crash at all. It moved all the files extra for 2 albums, about 25 files. 78% files were moved as matched. 22% were moved to non matched.
 

Great, have you got the report working now. Of the remaining 22% how many did SongKong manage to match the recording only for ?

greengeek wrote:

I did notice a few interesting things. Sk reported it was using on avg 700 of 2200 mb. However windows task manager was showing it was around 3000 mb being used. So the numbers in the program ui and windows task manager did not match up.
 

So you allocated a maximum of 2200mb heap memory to SongKong, the 700mb is the amount of that that SongKong is using at any time. Whereas WIndows task manager shows the amount allocated (2200mb) plus non-heap memory used such as storing the the running program itself, running java runtime and permanent memory usage.

greengeek wrote:

I am also currently verifying the matched files first. I am still fairly early on in the process, but i am noticing about 2% of the albums were moved as complete even though they did not contain any musicbrainz IDs. i left Search for Discogs match checked when I ran sk. Do files that were not matched in musicbrainz but maybe matched in discogs get moved to the matched folder?
 

Correct, the majority of users are just interested in matched folders, whether it is matched to Musicbrainz or Discogs if of little interest so either match will move to matched folder. If your primary concern is matching to Musicbrainz and just using Discogs to update matches with any extra info then I would disable Search Discogs and then Matched folder would only contain matches to MusicBrainz.

thanks Paul (Administrator)
greengeek

Pro
[Avatar]

Joined: 18/09/2007 02:50:48
Messages: 435
Offline


Great, have you got the report working now. Of the remaining 22% how many did SongKong manage to match the recording only for ?
 


Was fairly low, around 7% or so. The initial report that comes up in songkong's UI showed the numbers. The web based report was still giving me problems viewing it. I hadn't made any changes though to fix the issue when sk was running the large batch of files as I didn't want to take any chances of interrupting the batch. I will try another run on a smaller batch of files with the style folder copied over to see if it has been fixed.

KevinBluemel.com
Contemporary Instrumental Musician and Composer. Stop by to listen, watch videos, and get free downloads of songs and sheet music.
[WWW]
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7361
Offline

If you just copy over the style folder then reopen the report from the reports menu it should now work. The 7% is 7% of the original number of songs not of the not matched 22%, so 78% were matched to song and album, another 7% to song only and the remaining 15 % to nothing at all.

thanks Paul (Administrator)
greengeek

Pro
[Avatar]

Joined: 18/09/2007 02:50:48
Messages: 435
Offline

I only have it set to move complete albums, not individual recordings. The 78% would have been complete albums that were moved. The 7% that it said were matched to a single recording I am guessing would have fallen into that 22%. Not exactly sure as for my purposes I treat those as unmatched as I only have complete albums in my collection.

With some of the larger 100 song compilations getting tossed into the 22%, I bet that number would even be lower if those were allowed to be matched and got moved out. So maybe even a higher matched percentage.

KevinBluemel.com
Contemporary Instrumental Musician and Composer. Stop by to listen, watch videos, and get free downloads of songs and sheet music.
[WWW]
greengeek

Pro
[Avatar]

Joined: 18/09/2007 02:50:48
Messages: 435
Offline

Tried a small batch in songkong. Reports are still not working on some links. Summary works fine, but the next 4 links don't work. They are all pointing to:

file:///C:/Users/myuser/AppData/Roaming/SongKong/Reports/FixSongsReport00001/FixSongsReport00001_index.html#

The last 3 links all work after those.

KevinBluemel.com
Contemporary Instrumental Musician and Composer. Stop by to listen, watch videos, and get free downloads of songs and sheet music.
[WWW]
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7361
Offline

greengeek wrote:
Tried a small batch in songkong. Reports are still not working on some links. Summary works fine, but the next 4 links don't work. They are all pointing to:

file:///C:/Users/myuser/AppData/Roaming/SongKong/Reports/FixSongsReport00001/FixSongsReport00001_index.html#

The last 3 links all work after those. 


When you click on those 4 links they are meant to expand the lefthandside, please look at this example report to see what i mean http://jthink.net/songkong/reports/FixSongsReport00028/FixSongsReport00028.html

I'm fairly convinced you haven't yet copied the style folder to the reports folder yet, please check.

thanks Paul (Administrator)
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7361
Offline

greengeek wrote:
I only have it set to move complete albums, not individual recordings. The 78% would have been complete albums that were moved. The 7% that it said were matched to a single recording I am guessing would have fallen into that 22%. Not exactly sure as for my purposes I treat those as unmatched as I only have complete albums in my collection.

With some of the larger 100 song compilations getting tossed into the 22%, I bet that number would even be lower if those were allowed to be matched and got moved out. So maybe even a higher matched percentage.  


Yes I think we are saying the same thing to clarify, that remaining 22% can be broken down into 7% that matched recording, and 15% unmatched at all. And SongKong also treats match to recording as essentially unmatched because these songs are not moved to your matched folder as it could break up songs in a folder. Of the 15% not matched I expect few of these are actually in MusicBrainz, but many would be in Discogs - once I have reworked Discogs matching (which I am doing now) then Discogs matching should go up considerably.

As you say we could match some further albums once I increase the 100 song limit.

thanks Paul (Administrator)
greengeek

Pro
[Avatar]

Joined: 18/09/2007 02:50:48
Messages: 435
Offline

Correct.

I think a lot of the unmatched to MB ones are either releases MB does not have or ones that it might have the duration off on. I have a fairly broad collection and there are a lot of smaller artists in there that may have not been added yet to MB. I will know for sure once I get to the stage that I am verifying the non matched files. Still working on verifying the matched files, but getting close to finishing that part up. When I get to that point, I may have to verify each album online or from the physical media itself and then submit it to musicbrainz.

I also noticed that some albums contain one off oddities. Such as cd ripping software may shorten a mp3 length by cutting off any extreme amounts of silence if it is at the start or at the end of the song which throws off the duration matching. Or there might be a box set of older albums that contain both the mono and stereo version of the album which throws off matching and duplications. The outside of the rules that really throw off any software.

Oh, on a side note, the last 4 posts you have made in this topic have sent me emails to let me know that there was an update to the topic yet the links in the email have all pointed to broken links that come back as "The topic you are trying to see does not exist." Not sure if it is topic specific or something that got broken recently on the forums.

KevinBluemel.com
Contemporary Instrumental Musician and Composer. Stop by to listen, watch videos, and get free downloads of songs and sheet music.
[WWW]
greengeek

Pro
[Avatar]

Joined: 18/09/2007 02:50:48
Messages: 435
Offline

Out of roughly 250,000 files that run, all but 2 albums got moved to either the matched or non matched folders. SongKong performs wonderfully on very large batches of music. I cleared everything out and ran SK again on just those 2 albums which failed again. I have sent you the support files as well as the files themselves if you want to investigate why they are not being moved.

The reports showed these errors:

Code:
 Errors
 
 Unable to match group to MusicBrainz:M:\temp\b\Barbra Streisand\(1966) Color Me Barbra (US):org.hibernate.exception.DataException: Value too long for column "TITLE VARCHAR(255)": "'Medley: Animal Crackers in My Soup / Funny Face / That Face / They Didn''t Believe Me / Were Thine That Special Face / I''ve Gr... (321)"; SQL statement: update AcoustidRecordingIdPair set acoustId=?, artist=?, recordingId=?, sources=?, title=?, version=? where id=? and version=? [22001-172]
 
 Unable to match group to MusicBrainz:M:\temp\s\Sufjan Stevens\(2005) Illinois (GB):org.hibernate.exception.DataException: Value too long for column "TITLE VARCHAR(255)": "'The Black Hawk War, or, How to Demolish an Entire Civilization and Still Feel Good About Yourself in the Morning, or, We Apolog... (288)"; SQL statement: update AcoustidRecordingIdPair set acoustId=?, artist=?, recordingId=?, sources=?, title=?, version=? where id=? and version=? [22001-172]
 

KevinBluemel.com
Contemporary Instrumental Musician and Composer. Stop by to listen, watch videos, and get free downloads of songs and sheet music.
[WWW]
paultaylor

Pro
[Avatar]

Joined: 21/08/2006 09:21:27
Messages: 7361
Offline

Hi, thanks raised and now fixed in http://jthink.net:8081/browse/SONGKONG-621

thanks Paul (Administrator)
 
Forum Index -> SongKong Issues Go to Page: Previous  1, 2
Go to:   
Powered by JForum 2.1.6 © JForum Team