Sorry, a few days offline...

Quoting mediadogg:

Quote:
Man I hope you guys are not totally sick of me, but I am determined to get this damned thing right. If I were smarter, it would be faster. Sorry.

Anyways, is there anybody that can confirm how many profiles in this XML file have credits for "ziyi zhang" spelled exactly that way? (I know there are 366 profiles in the file. But do I really have two profiles with no valid credits???? If so, which two?)

I appreciate the help in advance. I am so dizzy with code variations and watching progress bars ...

As far as I can see, the two profiles are the German version of 4011976827085 and 4011976829584. Both Swiss profiles are complete crap:
- the EAN 401x belongs to Germany
- there are no swiss cover justifying a swiss locality profile for this titles
- the credits are either from the cover or from IMDb
- in one case the credited as is entered as role name

... but they are valid "Ziyi Zhang" profiles by the means of the CLT, both have FN="Ziyi", LN="Zhang" and CreditedAs=""

The German profiles, which are not listed by the CLT have a credited as entry CreditedAs="Zhang Ziyi", which makes them to profiles for "Zhang Ziyi" but they are NOT countable for "Ziyi Zhang"!

From this example 364 is the correct number of profiles.

Quote:
There are multiple cases:
(1) - credited only in the creditedAs field
(2) - credited in both creditedAs and F/M/L
(3) - credited only in F/M/L
(4) - neither

F/M/L is only valid, if there is no CreditedAs entry...

Let me use a (syntactical wrong) meta program code:
if ( CreditedAs != "") then
  SearchString := CreditedAs
else
  SearchString := FirstName + " " + MiddleName + " " + LastName
endIf
// Remove unneeded spaces
SearchString := Regex.Replace( SearchString, "\s+", " ")
SearchString := Trim(SearchString)

I have found a way to analyze the results using CookTop XPath tool. Not sure what your point is the about meta code. We have had the discussion before, unless you think somehow I didn't understand. Aside from the fact that your code will often crash when run with the plugin API, it would not return the same results as the CLT tool, which is what I am trying to match. The CLT results DO NOT ignore multiple spaces. That is part of the problem.

99% of the problems I am having, have nothing to do with the search code - it is mostly the tricks I am inventing to getting around the timing dependencies of screen scraping sequential web pages with no notification of when the pages are completely downloaded.

Pardon me for butting in here, but it seems to me that since there are two programs involved, it is important that we make it clear which program is responsible for what.

My understanding is that CLTBoss should extract all profiles that contain the searched name, regardless of where the name was found, and it would be up to CLTinfo to count the relevant credits.

If that is the case, then AiAustria’s code snippet would be directed to me, not to mediadogg. If we’re not 100% in agreement on who is responsible for what, then things are going to be very confusing.

The (only) purpose of the CLT was (is) to find the name an artist most often uses, the name we consider as his common name.

This is and will allways be the main issue to be solved by any CLT tool. And, therefore 364 is the correct count for Ziyi Zhang.

That said, I'm open to any information provided additionally... But the 366 example does not seem to be of any value, because it lists 364 Ziyi Zang profiles with two Zhang Ziyi profiles, which are rather randomly selected out 317 existing Zhang Ziyi profiles.

And yes, it was an idea popped up a while ago, to scan for all known name variants at once and leave the counting to the presentation part of a solution.
But this would mean a lot of extra work:
- there would be an array neede to collect all valid name variants
  - some help for the user entering them would be nice (eg. automatic flipping, removing middle initials, ...)
  - name variants that come up while scanning should be added automatically
  - ...
- then there would have to be single CLT call and scraping process for each and every name variant
- then the presentation tool would be requested to separate the whole bunch into the different variants and count them

I did not follow this idea, since, from my point of view it is more iportant to get a stable and futureproof CLT tool than getting high sophisticated new features. But it would save a lot of time, before setting up common name threads, if all the information for one person could be gathered with one single step

Quoting GSyren:

Quote:
Ok, that's not how I initially understood that it should work, but if that's what you guys agree on, then that's fine with me.

We are actually all on the same page. Remember I considered the 366 a bug and asked for help figuring it out. I did finally discover a bug in my duplicate detection code, which allowed the extras to slip through and somehow get allocated as having a match when they didn't. And I also stated up front that my goal was to match the CLT exactly.

So, guess what, we are all saying the same thing. The biggest limitations for me are my poor programming skills and frustration with the plague that has been cast upon us all.

Still waiting for Tom Cruise to finish, so I will post a bit of a preview what I am trying to perfect:

I have given up trying to choose a set of "delays" that always work with an unpredictable network.

So, here is what I will provide, three ways to scrape:

(1) Click-scrape-next: this is a very fast and very accurate if you have less than 10 to 20 pages. Takes 10 min.

(2) An auto-scrape that uses AutoIt to press a "Scrape Displayed Page" button combination with javascript to click on the page, in a loop. The start and end pages of the loop, I scrape off the CLT screen. After each page, if I get less than 25 profiles, then I scrape again, and if still not, I add to an error list. At the end of the all pages scraped, the error report is presented and the user can manually click on the few pages that had errors, before running the XML scan.

(3) A pop-up scrape, that interrupts the user after each page that somehow triggers the complete download of the data. This is annoying, but you can be "sure" to get all the profiles scraped before running the XML scan.

Quoting GSyren:

Quote:
Quoting AiAustria:

Quote:
And yes, it was an idea popped up a while ago, to scan for all known name variants at once

Sorry if there is something I have missed, but does this mean that CLTBoss will now only scan one single name at a time?

Regardless, it would still be advantageous to have the actual search argument as part of the output.

The current design and implementation of CLTBoss is to allow for scanning as many variants as it finds in the variants table. The resulting set of profile IDs collected and the resulting XML scan will be a collection that includes the results for all variants. The user has the option to use this feature or not.

That being said, given the issues I am having stabilizing the scraping operation, I obviously am focusing on getting 1 right. Then I will worry about more than 1. But the design remains the same. As we all are (or have been) programmers, I am sure you understand that it is very difficult to rip out features and change a program design after the fact. I thought the idea of multiple variants was a good one, so I took a shot at it.

In hindsight, it might have been better to go with the idea of an automatic generation of variants from a single search field. If I live long enough, and if people continue to care, I have a list of things to try in "son / daughter of CLTBoss" which includes:

- separate plugin
- use of Chrome instead of IE
- single search field with auto variants

And BTW, I am not going to spend time proving anything about the Profiler database. If you have a point to make, YOU prove it with a profile example. You show me, and I will write code to handle it.

Sorry, one more. But you will love it.

Something you might not have noticed is that CLTBoss will load the CLTPlus XML. If one trusts CLTPlus scraping (when it doesn't crash on a corrupt profile), then go ahead and load it into CLTBoss. You will have a set of credited profiles that you trust. Simply ask CLTBoss to download the Invelos XML into a collection and then search / sort / dance to your heart's content. Don't worry about how many spaces I squeeze out!

Oh yes, it might not be your Birthday, but you can go ahead and celebrate anyway. 

(Edit: If you want the current CLTBoss just for the purpose of using the XML download, I can post you a link in your PM. But it will still be unreleased for testing of the scraping and XML scan)